JP7764123B2 - Image capture device failure determination system - Google Patents

Description

The present invention relates to technology for determining whether an imaging device is faulty.

There is technology that recognizes objects from images captured by an imaging device and controls the device based on the results of that recognition. For example, technology has been developed that recognizes pedestrians and other objects from images captured by an imaging device mounted on a vehicle, and then controls the vehicle to stop so as not to collide with the recognized pedestrians and other objects.

When an imaging device is used for the above purposes, if the imaging device malfunctions, errors will occur in the captured images, and objects will not be correctly recognized from those images, which could result in the vehicle failing to avoid pedestrians, etc.

Therefore, it is necessary to determine whether or not there is a malfunction in the imaging device, and if a malfunction is determined to exist, to take measures such as making an emergency stop of the vehicle.

One example of a document disclosing technology for determining whether an imaging device is faulty is Patent Document 1. Patent Document 1 describes a mechanism for determining that a digital camera is faulty if the digital images captured by the digital camera remain uniform over a predetermined period of time.

Japanese Patent Application Laid-Open No. 2001-36927

When using the mechanism described in Patent Document 1, after a malfunction occurs in the imaging device, a predetermined number of uniform images are captured before the imaging device is determined to have malfunctioned. Therefore, the mechanism described in Patent Document 1 takes too long to determine whether or not an imaging device used for control that requires real-time performance, such as autonomous vehicle driving, has malfunctioned.

In consideration of the above circumstances, the present invention provides technology that makes it possible to virtually instantaneously determine whether or not an imaging device is malfunctioning.

The present invention proposes, as a first aspect, a failure determination system for an imaging device , which relates to each of a group of frames captured per second by an imaging device that captures video at a predetermined frame rate, projects a different test pattern within the imaging range when capturing each of a plurality of frames at different predetermined positions in the frame group, determines whether or not the imaging device is malfunctioning from the frame image of the imaging device in which the test pattern is captured, and performs object detection processing using other frame images.

The failure determination system according to the first aspect determines whether or not an imaging device is malfunctioning virtually instantaneously.

The present invention proposes, as a second aspect, a failure determination system for an imaging device that captures moving images, comprising a reflective member that intermittently projects a test pattern within the imaging range of the imaging device and reflects the test pattern toward the imaging device, the reflective member being in a reflective state when the test pattern is projected and in a transparent state when the test pattern is not projected , and that determines whether or not the imaging device is faulty from an image of the imaging device in which the test pattern is reflected .

According to the failure determination system of the second aspect, it is possible to determine substantially instantaneously whether or not the imaging device has a failure.
Furthermore, according to the failure determination system of the second aspect, the test pattern can be projected onto the reflecting member, thereby causing the imaging device to capture an image of the test pattern.

Furthermore, according to the failure determination system of the second aspect, there is no need to move the reflecting member.

The present invention provides a method for determining whether or not the imaging device is faulty from an image of the imaging device that captures moving images, the method comprising: providing a mirror that reflects a test pattern intermittently projected into the imaging range of the imaging device and reflects the test pattern toward the imaging device; the mirror being within the imaging range of the imaging device when the test pattern is projected and being outside the imaging range of the imaging device when the test pattern is not projected ; and determining whether or not the imaging device is faulty from an image of the imaging device that captures the test pattern.
A failure determination system for an imaging device is proposed as a third aspect.

According to the failure determination system of the third aspect, a clearer image is captured when the test pattern is not projected, compared to when a dimming mirror is used as the reflecting member, for example.

In the failure determination system according to any one of the first to third aspects, a fourth aspect may be adopted in which the imaging device takes images at shorter time intervals when a test pattern is irradiated than when the test pattern is not irradiated.

According to the failure determination system of the fourth aspect, the time during which normal photography cannot be performed due to photographing the test pattern is shortened.

1 is a diagram schematically illustrating the overall configuration of a failure determination system according to an embodiment; FIG. 1 is a diagram schematically illustrating a configuration of an imaging unit according to an embodiment. FIG. 1 is a diagram showing the configuration of a data processing device according to an embodiment. FIG. 4 is a diagram showing a processing flow of an image processing unit according to an embodiment. FIG. 4 is a diagram showing a processing flow of an irradiation instruction signal output unit according to an embodiment. 1 is a diagram showing a correspondence relationship between an image captured by an imaging device and processing performed by a data processing device according to an embodiment; FIG. 10 is a diagram schematically illustrating the configuration of an imaging unit according to a modified example. FIG. 10 is a diagram schematically illustrating the configuration of an imaging unit according to a modified example. FIG. 10 is a diagram illustrating an example of a test pattern used in a failure determination system according to a modified example.

[Embodiment]
A fault determination system 1 according to one embodiment of the present invention will be described below. Fig. 1 is a diagram schematically illustrating the overall configuration of the fault determination system 1. The fault determination system 1 in Fig. 1 is an object detection system that recognizes objects captured in an image of the area ahead of a train 9, and notifies a control device 8 that controls the running of the train 9 of the detection of the obstacle when an object that may obstruct the running of the train 9 is detected. However, in addition to the functions of a typical object detection system, the fault determination system 1 also has a function of determining whether or not an imaging device has malfunctioned in substantially real time, and, when it is determined that the imaging device has malfunctioned, notifying the control device 8 of the malfunction of the imaging device.

The fault determination system 1 includes an imaging unit 11 and a data processing device 12. The imaging unit 11 is placed in front of the lead car of the train 9 and captures images of the area ahead of the train 9.

The data processing device 12 is placed inside the train 9, acquires images captured by the imaging unit 11, recognizes objects from the images, and if it recognizes an obstacle, notifies the control device 8 of the detection of the obstacle. For this reason, the data processing device 12 is connected to communicate with the imaging unit 11 and the control device 8.

Figure 2 is a diagram showing a schematic configuration of the imaging unit 11. The imaging unit 11 includes an imaging device 111 that captures an image, an irradiation device 112 that irradiates a test pattern, and a dimming mirror 113 (an example of a reflective member). Although not shown in Figure 2, the imaging device 111, irradiation device 112, and dimming mirror 113 are each connected to the data processing device 12 for communication.

The imaging device 111 is a visible light camera that captures video. In this embodiment, for example, the time interval at which the imaging device 111 captures images, i.e., the frame rate, is assumed to be 60 fps (frames per second).

The imaging device 111 promptly outputs images captured every 1/60 seconds to the data processing device 12. The imaging device 111 also outputs a synchronization signal to the data processing device 12 every 1/60 seconds at the timing of capturing images.

The projection device 112 is a color projector. The projection device 112 projects a test pattern in response to a projection instruction signal output from the data processing device 12. In this embodiment, for example, when the frames captured by the imaging device 111 per second are the first to sixtieth frames, the projection device 112 projects a red test pattern over the entire surface when the fifty-eighth frame is captured, a green test pattern over the entire surface when the fifty-ninth frame is captured, and a blue test pattern over the entire surface when the sixtieth frame is captured.

The dimming mirror 113 is a device that switches, via electrical control, between a mirror state in which it reflects most light and a transmissive state in which it transmits most light. The dimming mirror 113 switches to the mirror state in response to an irradiation instruction signal output from the data processing device 12, and switches to the transmissive state when the output of the irradiation instruction signal from the data processing device 12 stops.

In Figure 2, the dashed line indicates the imaging range of the imaging device 111, and the dashed line indicates the illumination range of the illumination device 112. As shown in Figure 2, the positions of the imaging device 111, illumination device 112, and dimming mirror 113 are adjusted so that the test pattern illuminated by the illumination device 112 is reflected by the switchable mirror 113 in the mirror state and the reflected light is directed toward the imaging device 111.

The hardware of the data processing device 12 is a general-purpose computer with communication capabilities. The processor of the computer that is the hardware of the data processing device 12 processes data in accordance with a program stored in memory, thereby realizing the data processing device 12 having the configuration shown in Figure 3. Note that the data processing device 12 may also be configured as a so-called dedicated device.

The following describes each of the components of the data processing device 12 shown in Figure 3. The image acquisition unit 121 acquires the image output by the imaging device 111.

The image processing unit 122 receives images acquired by the image acquisition unit 121 from the imaging device 111, and performs either object detection processing or fault determination processing depending on the frame number of the received image. In this embodiment, the image processing unit 122 performs object detection processing for images with frame numbers 1 to 57 (images of the first frame to the 57th frame), and performs fault detection processing for images with frame numbers 58 to 60 (images of the 58th frame to the 60th frame).

The object detection process recognizes objects in an image using known image recognition techniques, and determines whether the object will pose an obstacle to the movement of the train 9 based on the type of object recognized and its position in the image. If the object is determined to be an obstacle, an obstacle detection signal is output to the control device 8 to notify it that an obstacle has been detected.

The fault detection process determines whether the image of the 58th frame is substantially entirely red, whether the image of the 59th frame is substantially entirely green, and whether the image of the 60th frame is substantially entirely blue. If it is determined that the image of any of these frames is not substantially entirely the specified color, a fault detection signal is output to the control device 8 to notify that a fault has been detected in the imaging device 111.

Note that "substantially the entire surface is XX (XX is the name of a specified color)" means that not only are all pixels of the image the specified color, but also that the image contains only noise that inevitably occurs even when the imaging device 111 is operating normally. The image processing unit 122 determines that the entire surface of the image is substantially the specified color when certain conditions are met, such as the ratio of pixels that are not the specified color being below a threshold and the number of consecutive pixels that are not the specified color being below a threshold.

Figure 4 shows the processing flow of the image processing unit 122. When the image processing unit 122 acquires an image from the imaging device 111 via the image acquisition unit 121 (step S101), it determines whether the frame number assigned to the image is 58, 59, 60, or some other number (step S102).

If the frame number is any one of 58 to 60 (step S102: "1", "2", or "3"), the image processing unit 122 performs a process to determine whether or not an error exists (step S103). Specifically, in step S103, if the frame number is 58 (step S102: "1"), the image processing unit 122 determines that there is no error if the image acquired in step S101 is an image that is substantially entirely red, and that there is an error if the image is not an image that is substantially entirely red.

Similarly, in step S103, if the frame number is 59 (step S102: "2"), the image processing unit 122 determines that there is no error if the image acquired in step S101 is an image in which the entire image is substantially green, and that there is an error if the image is not an image in which the entire image is substantially green. Also, in step S103, if the frame number is 60 (step S102: "3"), the image processing unit 122 determines that there is no error if the image acquired in step S101 is an image in which the entire image is substantially blue, and that there is an error if the image is not an image in which the entire image is substantially blue.

If it is determined in step S103 that an error has occurred (step S103: "Yes"), the image processing unit 122 outputs a failure detection signal to the data processing device 12 notifying that a failure in the imaging device 111 has been detected (step S104).

When the control device 8 receives a failure detection signal output from the image processing unit 122 of the data processing device 12, it notifies, for example, the driver of the train 9, of a failure in the imaging device 111 and performs predetermined control processing, such as deceleration, on the train 9.

If it is determined in step S103 that there is no error (step S103: "No"), or if the processing of step S104 is completed, the image processing unit 122 returns to step S101 and repeats the processing from step S101 onwards for the next image acquired from the imaging device 111 via the image acquisition unit 121.

The processing of steps S103 and S104 is the fault determination processing described above.

If it is determined in step S102 that the frame number is not between 58 and 60 (step S102: "4"), the image processing unit 122 recognizes the object appearing in the image acquired in step S101 using a known image recognition method (step S105).

Next, the image processing unit 122 determines whether the object recognized in step S105 is an obstacle to the movement of the train 9 based on the type of object recognized in the image and the position of the object recognized in step S105 (step S106).

If it is determined in step S106 that an object that may be an obstacle to the travel of the train 9 has been recognized (step S106: "Yes"), the image processing unit 122 outputs an obstacle detection signal to the data processing device 12 notifying that an obstacle has been detected (step S107).

When the control device 8 receives an obstacle detection signal output from the image processing unit 122 of the data processing device 12, it notifies, for example, the driver of the train 9 that an obstacle has been detected, and performs predetermined control processing on the train 9, such as stopping the train.

If it is determined in step S106 that no object that would obstruct the running of the train 9 has been recognized (step S106: "No"), or if the processing of step S107 has been completed, the image processing unit 122 returns to step S101 and repeats the processing from step S101 onwards for the next image acquired from the imaging device 111 via the image acquisition unit 121.

The processing of steps S105 to S107 is the object detection processing described above.

Referring to Figure 3, we will continue to explain the components of the data processing device 12. The synchronization signal acquisition unit 123 acquires the synchronization signal output by the imaging device 111. As mentioned above, the synchronization signal is a signal output by the imaging device 111 at the timing when it captures an image every 1/60 seconds.

The irradiation instruction signal output unit 124 outputs an irradiation instruction signal to the irradiation device 112 and the dimming mirror 113. Based on the synchronization signal acquired by the synchronization signal acquisition unit 123, the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a red test pattern after the imaging device 111 captures the 57th frame image and before the imaging device 111 captures the 58th frame image. Subsequently, the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a green test pattern, replacing the irradiation instruction signal previously output, after the imaging device 111 captures the 58th frame image and before the imaging device 111 captures the 59th frame image. Subsequently, the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a blue test pattern, replacing the irradiation instruction signal previously output, after the imaging device 111 captures the 59th frame image and before the imaging device 111 captures the 60th frame image. Next, the irradiation instruction signal output unit 124 stops outputting the irradiation instruction signal after the imaging device 111 captures the 60th frame image but before capturing the first frame image.

The illumination device 112 illuminates a red, green, or blue test pattern in response to an illumination instruction signal output from the illumination instruction signal output unit 124 of the data processing device 12. The dimming mirror 113 is in a reflective state while an illumination instruction signal for one of the colors is being output from the illumination instruction signal output unit 124 of the data processing device 12, and is in a transparent state while no illumination instruction signal is being output.

Figure 5 shows the processing flow of the irradiation instruction signal output unit 124. For example, when the data processing device 12 is started, the irradiation instruction signal output unit 124 first assigns an initial value of 0 to the counter C (step S201).

Next, when the irradiation instruction signal output unit 124 acquires a synchronization signal from the imaging device 111 via the synchronization signal acquisition unit 123 (step S202), it increments counter C by 1 (step S203) and determines whether counter C is 57, 58, 59, 60, or some other value (step S204).

If counter C is any of 57 to 59 (step S204: "1", "2", or "3"), the irradiation instruction signal output unit 124 begins outputting an irradiation instruction signal to the irradiation device 112 and the dimming mirror 113 (step S205).

Specifically, if counter C is 57 (step S204: "1"), the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a red test pattern in step S205. If counter C is 58 (step S204: "2"), the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a green test pattern instead of red in step S205. If counter C is 59 (step S204: "3"), the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the irradiation of a blue test pattern instead of green in step S205.

If counter C is 60 (step S204: "4"), the irradiation instruction signal output unit 124 stops outputting the irradiation instruction signal to the irradiation device 112 and the dimming mirror 113 (step S206). Note that the irradiation instruction signal whose output is stopped in step S206 is an irradiation instruction signal that instructs irradiation of a blue test pattern.

If counter C is not between 57 and 60 (step S204: "5"), or if the processing of step S205 is completed, the irradiation instruction signal output unit 124 returns the processing to step S202 and repeats the processing from step S202 onwards in accordance with the next synchronization signal acquired from the imaging device 111 via the synchronization signal acquisition unit 123.

On the other hand, if the processing of step S206 is completed, the irradiation instruction signal output unit 124 returns the processing to step S201, assigns the initial value 0 to counter C (step S201), and then repeats the processing from step S202 onwards in response to the next synchronization signal acquired from the imaging device 111 via the synchronization signal acquisition unit 123.

Figure 6 is a diagram showing the correspondence between the images captured by the imaging device 111 and the processing performed by the data processing device 12. Of the 60 images captured per second by the imaging device 111, frames 1 to 57 are images of the area ahead of the train 9. Frames 58 to 60 are images of a red, green, or blue test pattern.

The data processing device 12 performs object detection processing using images from frame 1 to frame 57 of the 60 images captured per second by the imaging device 111. The data processing device 12 then performs fault detection processing using images from frame 58 to frame 60.

With the above-described failure determination system 1, if a failure occurs in the imaging device 111, the failure is detected approximately one second later. In other words, the presence or absence of a failure in the imaging device 111 is determined virtually instantaneously. As a result, accidents such as the train 9 colliding with an obstacle that should have been detected due to a failure in the imaging device 111 can be avoided.

[Modification]
The above-described embodiment may be modified in various ways within the scope of the technical concept of the present invention. These modifications are shown below. Note that two or more of the modifications shown below may be combined as appropriate.

[First Modification]
Instead of the dimming mirror 113 employed in the above-described embodiment, a movable mirror may be employed so as to be within the imaging range of the imaging device 111 when the test pattern is being irradiated by the irradiation device 112, and to be outside the imaging range of the imaging device 111 when the test pattern is not being irradiated by the irradiation device 112.

Figure 7 is a diagram showing a schematic configuration of an imaging unit 21 that the failure determination system 1 according to this first modified example has in place of the imaging unit 11. Note that components that the imaging unit 21 has in common with the imaging unit 11 are designated by the same reference numerals as those used in the imaging unit 11.

The imaging unit 21 includes a movable mirror 213 (an example of a reflective member) instead of the dimming mirror 113. Figure 7(a) shows a state in which the movable mirror 213 is outside the imaging range of the imaging device 111, and Figure 7(b) shows a state in which the movable mirror 213 is within the imaging range of the imaging device 111. The movable mirror 213 is moved by a drive mechanism (not shown) to the position shown in Figure 7(a) when the irradiation device 112 is not irradiating a test pattern, and to the position shown in Figure 7(b) when the irradiation device 112 is irradiating a test pattern.

The first variant also allows for virtually instantaneous determination of whether or not the imaging device 111 is malfunctioning.

[Second Modification]
A half mirror (an example of a reflective member) that transmits approximately half of the light and reflects the remaining light may be used instead of the dimming mirror 113 used in the above-described embodiment. The configuration of the imaging unit provided in the failure determination system 1 according to this second modification is such that a half mirror is disposed in place of the dimming mirror 113 (see FIG. 2 ) provided in the imaging unit 11 according to the above-described embodiment.

When the illumination device 112 is not illuminating the test pattern, the image captured by the imaging device 111 is an image created by light that passes through the half mirror from the front of the train 9 and reaches the imaging device 111, and light that reaches the half mirror from various directions and is reflected by the half mirror and heads toward the imaging device 111. In this case, the amount of light that reaches the half mirror from various directions and is reflected by the half mirror and heads toward the imaging device 111 is sufficiently small compared to the amount of light that passes through the half mirror from the front of the train 9 and reaches the imaging device 111, and can be ignored. Therefore, the image captured by the imaging device 111 is essentially an image of the area ahead of the train 9.

When the illumination device 112 is illuminating the test pattern, the image captured by the imaging device 111 is an image drawn by light that passes through the half mirror from the front of the train 9 and reaches the imaging device 111, light that reaches the half mirror from various directions and is reflected by the half mirror toward the imaging device 111, and test pattern light that is illuminated from the illumination device 112, reflected by the half mirror, and toward the imaging device 111. The amount of test pattern light that is illuminated from the illumination device 112, reflected by the half mirror, and toward the imaging device 111 increases the greater the output of the illumination device 112. Therefore, when the output of the illumination device 112 is sufficiently high, the proportion of test pattern light in the light toward the imaging device 111 becomes sufficiently large, and the image captured by the imaging device 111 is essentially an image of the test pattern.

The second variant also allows for virtually instantaneous determination of whether or not the imaging device 111 is malfunctioning.

[Third Modification]
In the case of the failure determination system 1 according to the embodiment described above, during the period of one second during which three images from the 58th frame to the 60th frame are captured, i.e., during a period of 3/60 seconds = 0.05 seconds, no images are captured in front of the train 9, and no object detection processing is performed during that time. In order to shorten this period during which no object detection processing is performed, the imaging device 111 may capture images at shorter time intervals when the test pattern is irradiated than when the test pattern is not irradiated.

For example, in this third variant, the imaging device 111 captures images from the first frame to the 59th frame at intervals of 1/60 seconds, and from the 60th frame to the 62nd frame at intervals of 1/180 seconds.

Then, the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a red test pattern after capturing the 59th frame image and before capturing the 60th frame image; starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a green test pattern instead of red after capturing the 60th frame image and before capturing the 61st frame image; starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a blue test pattern instead of green after capturing the 61st frame image and before capturing the 62nd frame image; and stops outputting the irradiation instruction signal after capturing the 62nd frame image and before capturing the 1st frame image.

The image processing unit 122 then uses the images from frame 1 to frame 59 for object detection processing, and the images from frame 60 to frame 62 for fault detection processing.

The fault determination system 1 according to the third modification example is preferable because it shortens the time during which object detection processing is not performed compared to the fault determination system 1 according to the embodiment described above.

[Fourth Modification]
The time during which the object detection process is not performed may be shortened by a method different from the third modified example described above. Specifically, a configuration may be adopted in which the imaging device 111 constantly captures images at a frame rate (e.g., 1/180 fps) higher than the frame rate (e.g., 1/60 fps) required to avoid the train 9 colliding with an obstacle, and thinned images of images captured ahead of the train 9 are used for the object detection process.

For example, in this fourth modified example, the imaging device 111 always captures images at 180 fps. The illumination instruction signal output unit 124 starts outputting an illumination instruction signal instructing the imaging device 111 to illuminate a red test pattern after capturing the 177th frame image and before capturing the 178th frame image; starts outputting an illumination instruction signal instructing the imaging device 111 to illuminate a green test pattern instead of red after capturing the 178th frame image and before capturing the 179th frame image; starts outputting an illumination instruction signal instructing the imaging device 111 to illuminate a blue test pattern instead of green after capturing the 179th frame image and before capturing the 180th frame image; and stops outputting the illumination instruction signal after capturing the 180th frame image and before capturing the first frame image.

Then, the image processing unit 122 uses only the images from the first frame to the 177th frame (images of the area ahead of the train 9) with frame numbers, for example, (3n-2) (where n is a natural number satisfying 1≦n≦59) for the object detection process.

The failure determination system 1 according to the fourth modification also advantageously shortens the time during which object detection processing is not performed compared to the failure determination system 1 according to the embodiment described above.

[Fifth Modification]
Instead of using the switchable mirror 113 employed in the above-described embodiment, the irradiation device 112 may directly irradiate the imaging device 111 with a test pattern. Fig. 8 is a diagram schematically showing the configuration of an imaging unit 31 that is provided in the failure determination system 1 according to this fifth modification in place of the imaging unit 11. Note that the same reference numerals as those used in the imaging unit 11 are used for components that the imaging unit 31 has in common with the imaging unit 11.

As shown in Figure 8, in the imaging unit 31, the imaging device 111 and the illumination device 112 are positioned so that the test pattern illuminated by the illumination device 112 is directed directly toward the objective lens of the imaging device 111.

The fifth variant also makes it possible to virtually instantaneously determine whether or not the imaging device 111 is malfunctioning.

[Sixth Modification]
In the above-described embodiment, the three test patterns for red, green, and blue are captured in three consecutive frames. Alternatively, the three test patterns may be captured in three non-consecutive frames.

In this sixth modified example, for example, the irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a red test pattern after capturing the 19th frame image and before capturing the 20th frame image, and stops outputting the irradiation instruction signal after capturing the 20th frame image and before capturing the 21st frame image. The irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a green test pattern after capturing the 39th frame image and before capturing the 40th frame image, and stops outputting the irradiation instruction signal after capturing the 40th frame image and before capturing the 41st frame image. The irradiation instruction signal output unit 124 starts outputting an irradiation instruction signal instructing the imaging device 111 to irradiate a blue test pattern after capturing the 59th frame image and before capturing the 60th frame image, and stops outputting the irradiation instruction signal after capturing the 60th frame image and before capturing the 1st frame image.

As a result of the operation of the illumination device 112 and the dimming mirror 113 in accordance with the processing of the illumination instruction signal output unit 124 described above, of the 60 images captured by the imaging device 111 every second, the 20th frame is an image of the red test pattern, the 40th frame is an image of the green test pattern, the 60th frame is an image of the blue test pattern, and the remaining frames are images of the area ahead of the train 9.

The image processing unit 122 performs fault detection processing using images from frames 20, 40, and 60, and performs object detection processing using images from the other frames.

According to the sixth modification, the frequency of periods in which object detection processing is not performed increases compared to the case of the failure determination system 1 according to the embodiment described above, but the length of these periods is shortened, which is desirable.

[Seventh Modification]
The switchable mirror 113 used in the fault determination system 1 according to the above-described embodiment ideally reflects all light in the reflective state and transmits all light in the transmissive state. However, due to technical limitations, the switchable mirror 113 in the reflective state may transmit a certain proportion of light. Therefore, in a frame capturing an image of a test pattern, an image in which an image of the front of the train 9 is overlaid on the image of the test pattern (hereinafter referred to as a "mixed image") may be captured. This phenomenon may also occur in the above-described second modified example (in which a half mirror is used instead of the switchable mirror 113) and fifth modified example (in which the test pattern is directly irradiated onto the imaging device 111).

Therefore, a configuration may be adopted in which the image processing unit 122 removes at least some of the components of the image ahead of the train 9 from the mixed image, generates an image similar to an image capturing only the test pattern (hereinafter referred to as a "test pattern separated image"), and performs fault detection processing using the test pattern separated image.

In this seventh variant, for example, the irradiation instruction signal output unit 124 outputs an irradiation instruction signal so that test patterns are captured in the 20th, 40th, and 60th frames, as in the sixth variant. The image processing unit 122 then generates a red test pattern separation image by removing the 19th frame image component from the 20th frame image (combined image), generates a green test pattern separation image by removing the 39th frame image component from the 40th frame image (combined image), and generates a blue test pattern separation image by removing the 59th frame image component from the 60th frame image (combined image).

The procedure for generating a red test pattern separated image is explained below using the example. For each red, green, and blue pixel in the 20th frame image (combined image), the image processing unit 122 calculates a value by subtracting from that value the color value of the corresponding pixel in the 19th frame multiplied by a predetermined multiplier. This multiplier is determined appropriately based on the characteristics of the imaging device 111, the intensity of the light of the test pattern emitted by the illumination device 112, the intensity of the ambient light around the imaging device 111, and other factors, so as to remove as much of the forward image component as possible from the image obtained by combining the forward image of the train 9 and the test pattern image.

The image processing unit 122 generates an image as a test pattern separation image, with the pixel values calculated by the above subtraction.

According to the seventh variant, even in situations where an image of the front of the train 9 is mixed in with the image of a frame capturing a test pattern, the presence or absence of a malfunction in the imaging device 111 can be determined with high accuracy.

[Other Modifications]
(1) In the above-described embodiment, except when capturing an image of a test pattern, the chromatic aberration mirror 113 does not reflect light emitted from the irradiation device 112 toward the image capturing device 111. Similarly, in the above-described first modified example, except when capturing an image of a test pattern, the movable mirror 213 does not reflect light emitted from the irradiation device 112 toward the image capturing device 111. Therefore, in the above-described embodiment or the first modified example, the irradiation device 112 may always emit a test pattern.

(2) In the above-described embodiment, an image whose entire surface is a predetermined color is used as the test pattern. The form of the test pattern is not limited to this. For example, the vertical stripes shown in FIG. 9(a), the horizontal stripes shown in FIG. 9(b), or the houndstooth test pattern shown in FIG. 9(c) may be used. A single test pattern may also include multiple different colors. The colors used in the test pattern are not limited to the three primary colors. For example, a white test pattern obtained by mixing the three primary colors of light may be projected, and the presence or absence of a malfunction in the imaging device 111 may be determined by checking the red, green, and blue components of the pixel values of the image captured by the imaging device 111.

(3) In the above-described embodiment, for the sake of convenience, it is assumed that the test pattern is imaged every second, but the frequency of image capture of the test pattern is not limited to every second.

(4) In the above-described embodiment, the imaging device 111 is a visible light camera, but other types of imaging devices, such as an infrared camera, may also be used.

1...Fault determination system, 8...Control device, 9...Train, 11...Imaging unit, 12...Data processing device, 21...Imaging unit, 31...Imaging unit, 111...Imaging device, 112...Illumination device, 113...Dimming mirror, 121...Image acquisition unit, 122...Image processing unit, 123...Synchronization signal acquisition unit, 124...Illumination instruction signal output unit, 213...Movable mirror.

Claims (4)

A system for determining a malfunction of an imaging device , which, for each of a group of frames captured per second by an imaging device that captures video at a predetermined frame rate, projects a different test pattern within the imaging range when capturing each of a plurality of frames at a different predetermined number among the group of frames, determines whether or not the imaging device is malfunctioning from the frame image of the imaging device in which the test pattern is captured, and performs object detection processing using other frame images. A fault determination system for an imaging device that captures video includes a reflective member that intermittently projects a test pattern into an imaging range of the imaging device and reflects the test pattern toward the imaging device, the reflective member being in a reflective state when the test pattern is projected and in a transparent state when the test pattern is not projected, and the system determines whether or not the imaging device is faulty from an image of the imaging device that reflects the test pattern. A test pattern is intermittently projected into an imaging range of an imaging device that captures moving images, and the reflection member reflects the test pattern toward the imaging device, the reflection member being within the imaging range of the imaging device when the test pattern is projected, and being outside the imaging range of the imaging device when the test pattern is not projected , and the presence or absence of a malfunction of the imaging device is determined from an image of the imaging device in which the test pattern is projected.
A system for determining failures in an imaging device. The system for determining a failure of an imaging device according to claim 1 , wherein the imaging device captures images at shorter time intervals when the test pattern is irradiated than when the test pattern is not irradiated.