JP7823451B2 - Image generation device, image generation method, and image generation program - Google Patents

Description

This disclosure relates to an image generation device, an image generation method, and an image generation program that generate an image from a camera image and information from an object sensor.

In recent years, in surveillance systems that use camera images to monitor people, vehicles, etc., the accuracy of recognizing monitored objects decreases when visibility is poor due to rain or snow. To compensate for this, sensor fusion technology is being used to obtain advanced recognition results by combining data obtained from other sensors with camera image data.

For example, Patent Document 1 discloses technology that improves the accuracy of object recognition by using a camera with excellent horizontal angular resolution and millimeter-wave radar with excellent recognition accuracy in bad weather.

WO2020-116194

However, the technology described in Patent Document 1 identifies the position of the object being monitored by surrounding it with a frame in the camera image. Therefore, when the quality of the camera image is poor, for example, there is extremely little visual information available, making it difficult to determine the type and condition of the object being monitored, making effective monitoring difficult.

This disclosure has been made to solve the above problem, and aims to obtain an image that realistically represents an object even when the quality of the camera image is poor.

In order to solve the above-described problems, an image generation device according to the present disclosure includes a camera image acquisition unit, a sensor information acquisition unit, a trained model storage unit, and an image generation unit. The camera image acquisition unit acquires a camera image, which is visual information of a subject to be photographed. The sensor information acquisition unit acquires sensor information obtained by detecting the subject to be photographed using an object sensor. The trained model storage unit stores a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and the sensor image, which is visual information that simulates the subject to be photographed, from the camera image and the sensor information. The image generation unit outputs a pseudo camera image using the trained model from the camera image acquired by the camera information acquisition unit and the sensor information acquired by the sensor information acquisition unit. The sharpness determination unit determines the sharpness of the camera image acquired by the camera image acquisition unit. The output switching unit outputs either the camera image or the pseudo camera image based on the sharpness.

This disclosure has the advantage that it is possible to obtain an image that simulates an object even when the quality of the camera image is poor.

1 is an overall diagram showing the configuration of a monitoring system according to a first embodiment; 1 is a diagram illustrating a configuration of an image generating device according to a first embodiment. 10A and 10B are diagrams illustrating the generation of a pseudo camera image according to the first embodiment; 10A and 10B are diagrams illustrating the generation of a pseudo camera image according to the first embodiment in a case where learning is insufficient; FIG. 2 is a diagram illustrating an example of the configuration of a learning unit according to the first embodiment; FIG. 4 is a diagram illustrating an example of a data flow in a learning unit according to the first embodiment. 10 is a diagram illustrating an example of a data flow when a learning unit according to the first embodiment trains a discrimination unit. FIG. 10 is a diagram illustrating an example of a data flow when a learning generation unit is trained in a learning unit according to the first embodiment. FIG. 4 is a flowchart showing a processing operation of the image generating device according to the first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the image generating device according to the first embodiment. FIG. 10 is a diagram illustrating another example of the configuration of the image generating device according to the first embodiment. FIG. 10 is an overall view showing another example of the configuration of the monitoring system according to the first embodiment. FIG. 10 is a diagram illustrating a configuration of an image generating device according to a second embodiment. FIG. 10 is a diagram illustrating an example of a data flow in a learning unit according to the second embodiment. 10 is a flowchart showing a processing operation of the image generating device according to the second embodiment. FIG. 10 is a diagram illustrating another example of the configuration of an image generating device according to the second embodiment.

Embodiment 1.
An image generating device 1 according to the first embodiment will be described below. Fig. 1 shows an example of a monitoring system 100 including the image generating device 1 according to the first embodiment.

The monitoring system 100 consists of an image generation device 1, a monitoring unit 2, and a display unit 3.

The monitoring unit 2 includes a camera 4 and an object sensor 5, and outputs camera images 6 and sensor information 7 obtained by detecting a monitoring area including subjects such as people and vehicles using the camera 4 and object sensor 5 to the image generation device 1. The camera 4 and object sensor 5 are positioned so that at least a portion of their detection ranges overlap, and the overlapping detection ranges form the monitoring area of the monitoring system 100.

The camera 4 may be, for example, a digital camera or a video camera. The object sensor 5 is a non-contact sensor that uses electromagnetic waves, ultrasound, or the like. In the following description, a millimeter-wave sensor is used as the object sensor 5. Millimeter-wave sensors have excellent environmental resistance, such as the ability to detect objects regardless of poor visibility due to weather or backlighting due to lighting. Therefore, by combining this with camera image information and performing sensor fusion, it is possible to obtain information about the object being photographed even when the environment when the camera image is captured is poor. Another example of the object sensor 5 is LiDAR (Laser Imaging Detection and Ranging). The image generation device 1 receives the camera image 6 and sensor information 7 from the monitoring unit 2, generates a pseudo camera image 14 (described below), and outputs it to the display unit 3.

The display unit 3 displays the pseudo camera image 14 generated by the image generation device 1. The display unit 3 is, for example, a liquid crystal display or an organic electroluminescence (EL) display. A user of the surveillance system 100 can grasp the situation in the surveillance area by, for example, visually viewing the pseudo camera image 14 displayed on the screen of the display unit 3.

The configuration of the image generation device 1 is described below. Figure 2 is a diagram showing the configuration of the image generation device 1 according to embodiment 1. The image generation device 1 is composed of an image generation unit 10, a camera image acquisition unit 11, a sensor information acquisition unit 12, and a trained model storage unit 13.

The camera image acquisition unit 11 acquires camera images 6, which are visual information obtained by capturing images of subjects such as people and cars with a camera. Visual information is information presented in a form that can be visually recognized by the user of the surveillance system 100, and in particular refers to information presented as an image. By visually viewing the visual information displayed on the display unit 3, the user can identify the shape, color, etc. of the subject.

The sensor information acquisition unit 12 acquires sensor information 7 obtained by detecting objects to be photographed, such as people or vehicles, using the object sensor 5. The sensor information 7 is, for example, information indicating the position of the detected object, and specifically may be information on direction, distance, and coordinates within the detection range, or this information may be processed into visual information and displayed as an image. Note that while the photography by the camera 4 and the detection by the object sensor 5 are performed simultaneously, a time or spatial delay is acceptable as long as it does not affect the processing of the image generation device 1. The camera image 6 and sensor information 7 are time-synchronized and positioned based on previously acquired time or spatial delay information.

The image generation unit 10 outputs a pseudo camera image 14, which simulates the object being photographed, as a composite image from the camera image 6 acquired by the camera image acquisition unit 11 and the sensor information 7 acquired by the sensor information acquisition unit 12 using the trained model 13a described below. The image generation unit 10 can output a pseudo camera image 14 even if the camera image 6 is an image that lacks some or all of the visual information about the object being photographed. An image that lacks some or all of the visual information about the object being photographed refers to an image that was taken under adverse conditions, such as insufficient ambient illumination, backlight, rain, or fog, making the object unclear and difficult to distinguish from the background, and making it difficult to identify the shape, color, etc. of the object from the image alone.

The trained model storage unit 13 stores a trained model 13a that has been trained in advance to output a pseudo camera image 14, which is a composite image that simulates the object being photographed, from the camera image 6 and the sensor information 7. The trained model 13a is a trained model that generates a sensor image, which is visual information that simulates the object being photographed within the shooting range of the camera image 6, from the camera image 6 and the sensor information 7. The sensor image is, for example, an image that represents the characteristics of the object, such as its outer shape, inferred from the visual information of the object remaining in the camera image 6 and the information indicating the position of the object contained in the sensor information 7. Note that although the sensor information 7 does not provide recognizable visual information such as the outer shape of the object, by training it in combination with a camera image 6 that captures the same object relatively clearly, it is possible to obtain a trained model 13a that generates a sensor image that simulates the visual information of the object. The pseudo camera image 14 is an image in which the camera image 6 and the above-mentioned sensor image are superimposed.

Figure 3 is a diagram illustrating the generation of a pseudo camera image 14 when the subject captured in the camera image 6 is unclear due to lighting or other factors. When the camera image 6 and sensor information 7 are input into the image generation device 1, a pseudo camera image 14 is generated. Object A in the camera image 6 represents the person being photographed. Multiple points B in the sensor information 7 are information detected by the object sensor 5 and represent the positions where the person being photographed was detected. Using these two pieces of information and the trained model 13a, the image generation device 1 outputs a pseudo camera image 14 that simulates the appearance of the person. Object C in the pseudo camera image 14 is a sensor image created from the sensor information 7 using the trained model 13a, and the pseudo camera image 14 is generated by superimposing this on the camera image 6.

Figure 3 shows an example in which part of the subject in the camera image 6 cannot be recognized, but it is also possible for the subject to be invisible or to appear faint in the camera image 6 due to dark surroundings or other reasons. In this case, the trained model 13a must be trained using the camera image 6 and sensor information 7 according to each situation. Also, while the pseudo camera image 14 in Figure 3 shows a clear human figure, if the trained model 13a has not trained sufficiently or if the learning content and the situation in the camera image 6 differ, the image may have only the contours emphasized, as shown in Figure 4.

The trained model 13a can be trained using Generative Adversarial Networks (GAN), a well-known technology, to output a pseudo camera image 14 that allows the user to more clearly recognize the subject. The following describes the case where the trained model 13a is obtained using GAN.

Figure 5 shows the configuration of the learning unit 20, which obtains the trained model 13a using a GAN. The learning unit 20 is composed of a discrimination unit 21, a training generation unit 22, a discriminative model storage unit 23, a generative model storage unit 24, a training data acquisition unit 25, and a degradation processing unit 26. In this disclosure, the discrimination unit 21 trains as a "Discreminator" neural network in the GAN, and the training generation unit 22 trains as a "Generator" neural network in the GAN.

The learning data acquisition unit 25 acquires learning image data from an external storage unit (not shown), including learning camera images 27, which are camera images in which the subject is clearly captured, and learning sensor information 28, which is sensor information that detects the subject simultaneously with capturing the learning camera images 27. The learning image data is used for learning in the learning unit 20.

The degradation processing unit 26 performs a process to degrade the image quality of the training camera image 27 included in the training image data, generating a degraded training camera image 27a. This process degrades an image in which the subject is photographed well, thereby generating an image that simulates an image photographed under adverse conditions. The process to degrade the image quality may be, for example, a process to reduce brightness, a process to reduce resolution, a process to reduce contrast, or a process to add random noise. The process to degrade the image quality is not particularly limited as long as it removes at least a portion of the visual information related to the subject in the training camera image 27, making the subject less recognizable to the user. Furthermore, the process to degrade the image quality may be performed on the entire training camera image 27, or on the area of the subject. The degraded training camera image 27a and training sensor information 28 generated by the degradation processing unit 26 are output to the training generation unit 22, and the training camera image 27 is output to the identification unit 21.

The identification unit 21 classifies training camera images 27, which are relatively good camera images, as "true," and training pseudo camera images 29 output by the training generation unit 22 as "false." The training generation unit 22 generates training pseudo camera images 29 from degraded training camera images 27a, which are obtained by degrading the training camera images 27, and training sensor information 28.

The discriminative model storage unit 23 stores a discriminative model 23a that the discrimination unit 21 uses to discriminate between the training camera image 27 and the pseudo training camera image 29 generated by the training generation unit 22. The generative model storage unit 24 also stores a generative model 24a that generates a training sensor image, which is pseudo visual information that simulates the visual information of the subject that was lost in the process of degrading the image quality of the training camera image 27, from the degraded training camera image 27a and the training sensor information 28, and outputs a synthetic image in which the training sensor image is superimposed on the degraded training camera image 27a.

Next, we will explain the learning process in which the learning unit 20 generates a trained model 13a using a GAN. In the GAN learning process, the identification unit 21, which is the "Discriminator," and the learning generation unit 22, which is the "Generator," are trained competitively.

When the identification unit 21 is trained, the identification model 23a is trained to improve the accuracy of distinguishing between the training camera image 27 and the pseudo training camera image 29. In the following description, the training camera image 27 is referred to as "true" and the pseudo training camera image 29 as "false," and the identification model 23a is trained to distinguish whether the input image is "true" or "false." Note that the identification model 23a may also be configured to receive the training camera image 27 and the pseudo training camera image 29 simultaneously and distinguish which is "true" and which is "false."

When the learning generation unit 22 is trained, the generative model 24a is trained so as to improve the accuracy of generating training pseudo camera images 29 that would otherwise be mistakenly identified as "true" by the identification unit 21.

Figure 6 is a diagram showing a simulation of the data flow when the learning unit 20 learns a GAN. The learning unit 20 improves the accuracy with which the identification unit 21 distinguishes between training camera images 27, which are images in good condition, and training pseudo camera images 29, and improves the accuracy with which the training pseudo camera images 29 output by the training generation unit 22 misidentify the identification unit 21 as "correct." The degradation processing unit 26 is used to degrade images. By having the identification unit 21 and the training generation unit 22 learn competitively in this way, the training generation unit 22 can generate a generative model 24a that outputs training pseudo camera images 29 that are closer to good camera images.

Competitive learning between the identification unit 21 and the learning generation unit 22 is performed, for example, by training either the identification model 23a or the generation model 24a based on the judgment results of the identification unit 21 on the learning pseudo camera image 29.

Figure 7 shows the data flow when the identification unit 21 determines that the training pseudo camera image 29 is "correct." If the identification unit 21 determines that the training pseudo camera image 29 generated by the training generation unit 22 from the degraded training camera image 27a and the training sensor information 28 is "correct," this means that the identification unit 21 has made an incorrect judgment, while the training generation unit 22 has successfully misidentified the identification unit 21. Therefore, in this case, the identification model 23a of the identification unit 21 that made the incorrect judgment is trained. For example, the training is performed by labeling the training camera image 27, which is a camera image in good condition, as "correct" and labeling the training pseudo camera image 29 output by the training generation unit 22 as "false," thereby reflecting this in the identification model 23a of the identification unit 21.

Figure 8 shows the data flow when the identification unit 21 determines that the training pseudo camera image 29 is "false." If the identification unit 21 determines that the training pseudo camera image 29 generated by the training generation unit 22 from the degraded training camera image 27a and the training sensor information 28 is "false," this means that the identification unit 21 made a correct determination, while the training generation unit 22 failed to make a misidentification. Therefore, in this case, the generative model 24a of the training generation unit 22 that failed to make a misidentification is trained. For example, the training pseudo camera image 29 output by the training generation unit 22 is input to the identification unit 21 with fixed weights, and the identification unit 21 learns to aim for a "correct" determination, i.e., to generate a training pseudo camera image 29 that is closer to the training camera image 27. The learning results are then reflected in the generative model 24a of the training generation unit 22.

As described above, either the discriminative model 23a or the generative model 24a is trained based on the discrimination unit 21's judgment of the training pseudo camera image 29. By repeating this process, the discriminative unit 21 and the training generation unit 22 are trained competitively. As a result, the training generation unit 22 creates a generative model 24a from the degraded training camera image 27a and the training sensor information 28 so as to output a training pseudo camera image 29 that is closer to the training camera image 27. By applying this generative model 24a to the trained model 13a, the image generation device 1 is constructed. In other words, the trained generative model 24a is stored in the trained model storage unit 13 as the trained model 13a and is used as the trained model 13a in the image generation device 1.

In the above explanation, either the discriminative model 23a or the generative model 24a is trained based on the judgment of the training pseudo camera image 29. However, it is also possible to input both the training camera image 27 and the training pseudo camera image 29 into the discrimination unit 21, have it determine which image is "correct," and train either the discriminative model 23a or the generative model 24a based on the result.

In the above description, the learning generation unit 22 generates training pseudo camera images 29 from the degraded training camera images 27a and training sensor information 28 to train the generative model 24a, which is a learning model. However, the training camera images 27 may also be used as training data. For example, the training camera images 27 are stored in the generative model storage unit 24 as part of the generative model 24a. Hereinafter, the training camera images 27 stored as training data will be referred to as stored training camera images 27b. The learning generation unit 22 generates training pseudo camera images 29 using the degraded training camera images 27a, the training sensor information 28, and the stored training camera images 27b stored in the generative model storage unit 24. As a result, if a stored training camera image 27b similar to the photographed subject exists in the generative model storage unit 24, the generative model 24a can be trained to generate clearer training pseudo camera images 29.

When training camera images 27 are used as training data, the generative model 24a is trained to generate training pseudo camera images 29 using degraded training camera images 27a, training sensor information 28, and stored training camera images 27b. The generative model memory unit 24, which stores the stored training camera images 27b and stores a generative model 24a that uses the stored training camera images 27b for image generation, is reflected in the trained model memory unit 13 and is used as a trained model 13a that uses the stored training camera images 27b to generate pseudo camera images 14 in the image generation device 1.

Next, the processing operations in the image generation device 1 according to embodiment 1, from when the camera image acquisition unit 11 acquires the camera image 6 and the sensor information acquisition unit 12 acquires the sensor information 7, until when the image generation unit 10 outputs the pseudo camera image 14, will be described using the flowchart in Figure 9.

Since the camera 4 and object sensor 5 acquire camera images 6 and sensor information 7 at predetermined intervals, the camera image acquisition unit 11 and sensor information acquisition unit 12 repeatedly check data acquisition until the camera images 6 and sensor information 7 acquired by the camera 4 and object sensor 5 are input (S11: No). When the camera image acquisition unit 11 receives the camera image 6 and the sensor information acquisition unit 12 receives the sensor information 7 (S10: Yes), the camera image 6 and sensor information 7 are input to the image generation unit 10 (S11). The image generation unit 10 then uses the trained model 13a to create a pseudo camera image 14 from the camera image 6 and sensor information 7 (S12). The pseudo camera image 14 is then output to the display unit 3 (S13).

Next, an example of the hardware configuration of the image generation device 1 according to embodiment 1 will be described. Figure 10 is a diagram showing an example of the hardware configuration of the image generation device 1. The image generation device 1 includes a computer having a processor 101 and a memory 102.

The processor 101 and memory 102 can send and receive information to and from each other via, for example, a bus 103. The processor 101 reads and executes programs stored in the memory 102 to perform functions such as the image generation unit 10, learning unit 20, learning generation unit 22, and degradation processing unit 26. The processor 101 is, for example, an example of a processing circuit, and includes one or more of a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an LSI (Large Scale Integration), and a GPU (Graphics Processing Unit).

Memory 102 includes one or more of RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), and EEPROM (Electrically Erasable Programmable Read Only Memory). Memory 102 also includes a recording medium on which a computer-readable program is recorded. Such recording media include non-volatile or volatile semiconductor memory, magnetic disks, optical disks, etc. The image generation device 1 may also include integrated circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field Programmable Gate Array).

As described above, the image generation device 1 according to embodiment 1 includes a camera image acquisition unit 11, a sensor information acquisition unit 12, a trained model storage unit 13, and an image generation unit 10. The camera image acquisition unit 11 acquires a camera image 6, which is visual information of the subject being photographed. The sensor information acquisition unit 12 acquires sensor information 7 obtained by detecting the subject using an object sensor. The trained model storage unit 13 stores a trained model 13a for generating a pseudo camera image 14, which is a composite image obtained by superimposing the camera image 6 and the sensor image, which is visual information that simulates the visual information of the subject, from the camera image 6 and the sensor information 7. The image generation unit 10 outputs the pseudo camera image 14 using the trained model 13a from the camera image 6 acquired by the camera image acquisition unit 11 and the sensor information 7 acquired by the sensor information acquisition unit 12. This makes it possible to obtain an image that simulates an object even when the camera image 6 is missing. Additionally, by displaying a pseudo camera image 14, which simulates the visual information of the subject, on the display unit 3, the user can determine the type and condition of the subject, enabling effective monitoring.

In the first embodiment, the camera image 6 contains one photographed object, but the camera image 6 may contain multiple photographed objects. For example, the image generation unit 10 may use the sensor information 7 to determine the number of photographed objects contained in the camera image 6, generate pseudo camera images using camera images 6 divided so that each image contains one photographed object, and combine the generated multiple pseudo camera images 14 into a single image for output. In the first embodiment, the degradation processing unit 26 generates the degradation learning camera image 27a, but the learning data acquisition unit 25 may acquire the previously prepared learning camera image 27 and degradation learning camera image 27a.

Below, a modified example of the image generating device 1 according to embodiment 1 is described. The image generating device 1a according to this modified example differs from the image generating device 1 in that it includes a sharpness determination unit 15 that determines the sharpness of the subject captured in the camera image 6, and an output switching unit 16 that outputs either the camera image 6 or the pseudo camera image 14 to the display unit 3 based on the sharpness. The following description will focus on the differences from embodiment 1.

Figure 11 is a diagram showing the configuration of image generation device 1a. Image generation device 1a is composed of image generation unit 10, camera image acquisition unit 11, sensor information acquisition unit 12, trained model storage unit 13, sharpness determination unit 15, and output switching unit 16. The sharpness determination unit 15 calculates the sharpness of the camera image 6 acquired by the camera image acquisition unit 11. The sharpness may be, for example, the high-frequency components or edge components of the entire camera image 6. Alternatively, the sharpness may be the high-frequency components of an area of the camera image 6 corresponding to the position where an object was detected in the sensor information 7. Furthermore, whether or not the photographed subject can be recognized as the relevant subject, i.e., whether or not object detection was successful, may also be added to the determination of sharpness.

The output switching unit 16 switches the image to be output to the display unit 3 based on the clarity. If the clarity meets a predetermined condition, it determines that the subject captured in the camera image 6 is clear, and outputs the camera image 6 sent from the camera image acquisition unit 11 to the display unit 3. On the other hand, if the clarity does not meet the predetermined condition, it determines that the subject captured in the camera image 6 is unclear, and outputs the pseudo camera image 14 generated by the image generation unit 10 to the display unit 3. As a result, in the surveillance system 100, if the subject is not clearly captured in the camera image 6, a pseudo camera image 14 that is a synthetic image of the subject is displayed on the display unit 3, and if the subject is clearly captured, the camera image 6 can be displayed as is on the display unit 3. As a result, it is possible to display either the camera image 6 or the pseudo camera image 14, whichever image makes the subject more easily recognizable.

Alternatively, the clarity determination unit 15 and output switching unit 16 may be provided outside the image generation device 1 as shown in FIG. 12, and if the clarity determination unit 15 determines that the subject captured in the camera image 6 is clear, it may output the camera image 6 and sensor information 7 to the image generation unit 10, but if it determines that the subject is not clear, it may output the camera image 6 directly to the display unit 3. This makes it possible to determine whether or not to output the pseudo camera image 14 depending on the clarity of the subject, without having to change the design of the image generation device 1.

Embodiment 2.
The image generation device according to the second embodiment differs from the image generation device 1 according to the first embodiment in that the image generation device includes a learning unit that learns a trained model. In the following, components having the same functions as those in the first embodiment are denoted by the same reference numerals and their description is omitted, and the description will focus on the differences from the image generation device 1 according to the first embodiment.

The configuration of image generation device 1b according to embodiment 2 will be described. FIG. 13 is a diagram showing the configuration of image generation device 1b according to embodiment 2. Image generation device 1b includes a learning unit 20a that trains a trained model 13a used by image generation unit 10 to create pseudo camera images 14. As a result, when the state of the camera image 6 is poor, image generation device 1b can use the visual information immediately before the image state deteriorates as a training model, thereby achieving the effect of being able to generate pseudo camera images 14 that are closer to the actual subject. Furthermore, image generation device 1b outputs pseudo camera images 14 while training trained model 13a in real time, thereby achieving the effect of being able to perform efficient learning suited to the environment.

The learning unit 20a included in the image generation device 1b in embodiment 2 differs from the learning unit 20 that generates the trained model 13a in embodiment 1 in that the learning data acquisition unit 25 acquires the camera images 6 acquired by the camera image acquisition unit 11 and the sensor information 7 acquired by the sensor information acquisition unit 12 in the image generation device 1b and uses these for learning. Furthermore, the learning unit 20a may be configured to apply a model that has been trained in advance by the learning unit 20 in embodiment 1 or the like, and additionally train part or all of the model.

As described above, the acquired camera images 6 and sensor information 7 are used to output the pseudo camera images 14 in the image generation unit 10, and at the same time, the learning unit 20a cumulatively learns the trained model 13a. However, because the camera images 6 used for learning need to be clear, not all acquired camera images 6 and sensor information 7 are trained; rather, for example, the learning unit 20a trains the trained model 13a only when camera images 6 that are deemed to be sufficiently clear are acquired.

Whether a camera image 6 that is deemed sufficiently clear and can be used for learning can be determined, for example, by setting thresholds for parameters such as image brightness, high frequency components, or edge components, or by switching based on the time of day. Alternatively, the operator can decide whether to switch between learning and non-learning states. Furthermore, whether the subject in the image can be recognized as the relevant object, i.e., whether object detection was successful, can also be added to the determination of clarity.

Learning using a GAN is also effective in embodiment 2. Figure 14 is a diagram showing the flow of data during learning by the learning unit 20a when a GAN is used. The identification unit 21 determines that the camera image 6 acquired by the camera image acquisition unit 11 is "true" and that the training pseudo camera image 29 output by the training generation unit 22 is "false." The training generation unit 22 also receives as input a degraded camera image 6a obtained by degrading the camera image 6 acquired by the camera image acquisition unit 11 and sensor information 7 acquired by the sensor information acquisition unit 12, and generates the training pseudo camera image 29 using the generative model 24a. The learning method is the same as that of the learning unit 20 in embodiment 1.

Furthermore, the learning unit 20a is more effective than in the first embodiment in using the camera image 6 as learning data in addition to the degraded camera image 6a and sensor information 7. This is because, if the condition of the camera image 6 of the subject deteriorates, the camera image 6 taken immediately before that, in which the subject is clearly visible, can be used to generate the pseudo camera image 14.

The processing of the learning unit 20a when using a camera image 6 as training data will be described. A clear camera image 6 and sensor information 7 are acquired by the training data acquisition unit 25, and a degraded camera image 6a is generated by the degradation processing unit 26. The degraded camera image 6a and sensor information 7 are then input to the training data generation unit 22, which then stores the camera image 6 as training data in the generative model storage unit 24 of the training data generation unit 22. Hereinafter, the camera image 6 stored as training data will be referred to as a stored camera image 6b. The training data generation unit 22 generates a training pseudo camera image 29 using the degraded camera image 6a, the sensor information 7, and the stored camera image 6b and generative model 24a stored in the generative model storage unit 24. This makes it possible to generate a clearer training pseudo camera image 29 based on a past camera image 6.

Using camera images 6 as learning data is particularly effective when stored camera images 6b of the same subject as the subject of the camera image 6 acquired by the camera image acquisition unit 11 are present in the generative model storage unit 24. This is because the generative model storage unit 24 storing camera images 6 of the same subject is reflected in the trained model storage unit 13, and camera images 6 of the same subject can be used to generate pseudo camera images 14. As a result, if the condition of the camera image 6 suddenly deteriorates, information on the camera image 6 immediately before the image condition deteriorates is stored in the trained model storage unit 13 and can be used to generate the pseudo camera image 14, making it possible to generate pseudo camera images 14 that are closer to the actual subject.

Also, the pseudo camera image 14 may be generated using the corresponding stored camera image 6b only if the learned model storage unit 13 contains a stored camera image 6b that captures the same subject as the subject in the shooting range of the camera image 6 acquired by the camera image acquisition unit 11. In this case, for example, during learning by the learning generation unit 22, the camera image 6 and the sensor information 7 acquired at the same time are associated and stored in the generative model storage unit 24. Hereinafter, the sensor information 7 stored in the generative model storage unit 24 will be referred to as stored sensor information 7b. The image generation unit 10 takes into consideration the positional continuity and shooting time between the stored sensor information 7b and the sensor information 7 acquired by the sensor information acquisition unit 12, and generates the pseudo camera image 14 using the stored camera image 6b if a stored camera image 6b capturing the same subject is present in the learned model storage unit 13.

However, if the learned model storage unit 13 contains a stored camera image 6b that captures the same subject as the subject in the shooting range of the camera image 6 acquired by the camera image acquisition unit 11, and the corresponding stored camera image 6b is used to generate the pseudo camera image 14, then during training the learning generation unit 22, it is necessary to train the learning generation unit 22 to both generate a training pseudo camera image 29 using the stored camera image 6b and generate a training pseudo camera image 29 without using the stored camera image 6b.

Next, the processing operations in the image generation device 1b according to embodiment 2 will be described using the flowchart in Figure 15, from when the camera image acquisition unit 11 acquires the camera image 6 and the sensor information acquisition unit 12 acquires the sensor information 7, until when the learning unit 20a reflects the learning in the trained model 13a, and until when the image generation unit 10 outputs the pseudo camera image 14.

The camera 4 and object sensor 5 acquire camera images 6 and sensor information 7 at predetermined intervals, so the camera image acquisition unit 11 and sensor information acquisition unit 12 repeatedly check data acquisition until the camera images 6 and sensor information 7 acquired by the camera 4 and object sensor 5 are input (S20: NO). When the camera image acquisition unit 11 receives the camera image 6 and the sensor information acquisition unit 12 receives the sensor information 7 (S20: YES), the camera image 6 and sensor information 7 are input to the image generation unit 10 (S21). After receiving the camera image 6 and sensor information 7, the image generation unit 10 creates a pseudo camera image 14 using the trained model 13a (S22). The pseudo camera image 14 is then output to the display unit 3 (S23).

The learning unit 20a also determines whether the acquired camera image 6 is clear (S30). If it is determined that the camera image 6 is clear (S30: YES), the learning unit 20a trains a model using the camera image 6 and sensor information 7 (S31). If a GAN is used for training, the identification unit 21 and the training generation unit 22 are trained here. Next, the training results are reflected in the trained model 13a (S32), and training ends. If a GAN is used for training, the trained generation model 24a of the training generation unit 22 is reflected in the trained model 13a. Note that if it is determined that the camera image 6 is not clear, the learning unit 20a simply terminates training without training (S30: NO).

Note that, although the image generation unit 10 has created the pseudo camera image 14 and then the learning unit 20a has performed the learning process, the learning unit 20a may perform the learning first and then the image generation unit 10 may create the pseudo camera image 14, or learning and image generation may be performed simultaneously.

An example of the hardware configuration of image generation device 1b according to embodiment 2 is the same as the hardware configuration of image generation device 1 shown in the figure. Processor 101 can execute the functions of image generation unit 10 and learning unit 20a by reading and executing programs stored in memory 102.

As described above, the image generation device 1b according to embodiment 2 further includes a learning unit 20a that trains the trained model 13a in addition to the configuration of the image generation device 1 according to embodiment 1. As a result, if the condition of the camera image 6 suddenly deteriorates due to a malfunction of the surrounding lighting, for example, it is possible to use training data that has been trained from the camera image 6 taken of the subject in good condition just before the deterioration, thereby achieving the effect of outputting a pseudo camera image 14 that is closer to the real thing. Furthermore, the image generation device 1a can train the trained model 13a in real time while operating the image generation unit 10. As a result, it is possible to learn information about people and vehicles in the environment of the monitored location, thereby achieving the effect of outputting a pseudo camera image 14 that has information that is closer to that of a camera image 6 taken in good condition.

Below, we will show a first variation of the image generation device 1b according to the second embodiment. The image generation device 1c according to the first variation differs from the image generation device 1b in that it includes a sharpness determination unit 15, an output switching unit 16, and an input switching unit 17 that inputs the camera image 6 and sensor information 7 to either the learning unit 20a or the image generation unit 10 based on the sharpness. The following will focus on the differences from the second embodiment. Figure 11 is a diagram showing the configuration of the image generation device 1a. The image generation device 1a is composed of the image generation unit 10, the camera image acquisition unit 11, the sensor information acquisition unit 12, the trained model storage unit 13, the sharpness determination unit 15, and the output switching unit 16.

Figure 16 is a diagram showing the configuration of image generation device 1c. Image generation device 1c is composed of image generation unit 10, camera image acquisition unit 11, sensor information acquisition unit 12, trained model storage unit 13, sharpness determination unit 15, output switching unit 16, input switching unit 17, and learning unit 20a. Camera image acquisition unit 11 outputs camera image 6 to the sharpness determination unit, input switching unit 17, and output switching unit 16, and sensor information acquisition unit 12 inputs sensor information 7 to input switching unit 17.

The input switching unit 17 switches the object to which the camera image 6 and sensor information 7 are input based on the clarity. If the clarity meets a predetermined condition, it determines that the object captured in the camera image 6 is clear, and inputs the camera image 6 and sensor information 7 to the learning unit 20a. At this time, the learning unit 20a trains the trained model 13a. Furthermore, since the camera image 6 and sensor information 7 are not input to the image generation unit 10, a pseudo camera image 14 is not generated. On the other hand, if the clarity does not meet the predetermined condition, it determines that the object captured in the camera image 6 is unclear, and inputs the camera image 6 and sensor information 7 to the image generation unit 10. At this time, the image generation unit 10 generates a pseudo camera image 14. Furthermore, since the camera image 6 and sensor information 7 are not input to the learning unit 20a, a trained model 13a is not trained.

The output switching unit 16 is the same as in the modified example of embodiment 1. As a result, in the surveillance system 100, if the subject is not clearly visible in the camera image 6, a pseudo camera image 14 that is a composite image of the subject is displayed on the display unit 3, and if the subject is clearly visible, the camera image 6 is displayed as is on the display unit 3, while the clear camera image 6 is used to train the trained model 13a. As a result, it is possible to display the image that makes it easier to recognize the subject, out of the camera image 6 and the pseudo camera image 14, and simultaneously train the trained model 13a efficiently.

The configurations shown in the above embodiments are merely examples, and may be combined with other known technologies, or different embodiments may be combined with each other. Parts of the configuration may also be omitted or modified without departing from the spirit of the invention.

1, 1a, 1b, 1c Image generation device, 2 Monitoring unit, 3 Display unit, 4 Camera, 5 Object sensor, 6 Camera image, 6a Degraded camera image, 6b Stored camera image, 7 Sensor information, 7b Stored sensor information, 10 Image generation unit, 11 Camera image acquisition unit, 12 Sensor information acquisition unit, 13 Learned model storage unit, 13a Learned model, 14 Pseudo camera image, 15 Sharpness determination unit, 16 Output switching unit, 17 Input switching unit, 20, 20a Learning unit, 21 Identification unit, 22 Learning generation unit, 23 Identification model storage unit, 23a Identification model, 24 Generative model storage unit, 24a Generative model, 25 Learning data acquisition unit, 26 Degradation processing unit, 27 Learning camera image, 27a Degraded learning camera image, 27b Stored learning camera image, 28 Learning sensor information, 29 Pseudo camera image for training, 100, 100a surveillance system, 101 processor, 102 memory, 103 bus.

Claims (11)

a camera image acquisition unit that acquires a camera image, which is visual information of a subject;
a sensor information acquisition unit that acquires sensor information obtained by detecting the object to be photographed using an object sensor;
a trained model storage unit that stores a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image that is visual information that simulates visual information of the subject, from the camera image and the sensor information; and
an image generation unit that outputs the pseudo camera image using the trained model from the camera image acquired by the camera image acquisition unit and the sensor information acquired by the sensor information acquisition unit;
a sharpness determination unit that determines the sharpness of the camera image acquired by the camera image acquisition unit;
an output switching unit that outputs either the camera image or the pseudo camera image based on the sharpness;
An image generating device comprising:

The image generation device according to claim 1, wherein the trained model storage unit trains an identification model, which is a learning model of the identification unit, so that the identification unit, which identifies training camera images, which are images in which the subject is clearly captured, as positive and the pseudo camera images as false, improves the accuracy of the identification, and the training generation unit, which is a learning generator that outputs the pseudo camera images from the camera images and the sensor information, trains a generation model, which is a learning model of the training generation unit, so that the pseudo camera images cause the identification unit to misjudge the identification, and stores the trained generation model as the trained model in a generative adversarial network. The image generation device according to claim 2, characterized in that the learning generation unit generates the pseudo camera image from a degraded learning camera image obtained by processing the learning camera image so that at least a portion of the visual information of the subject contained in the learning camera image is missing, and the sensor information. 2. The image generation device according to claim 1, further comprising a learning unit that acquires the camera image acquired by the camera image acquisition unit and the sensor information acquired by the sensor information acquisition unit as learning data, and uses the learning data to train the trained model for generating the pseudo camera image. The learning unit
an identification unit that identifies the camera images acquired as the learning data as genuine and the pseudo camera images as fake;
a learning generation unit that is a learning generator that outputs the pseudo camera image from the camera image acquired as the learning data and the sensor information;
A learning device using a generative adversarial network comprising:
the identification unit trains an identification model, which is a learning model of the identification unit, so as to improve the accuracy of the identification;
the generation unit for learning trains a generation model that is a learning model of the generation unit for learning so as to generate the pseudo camera image that causes the identification unit to misjudge the identification;
The image generating device according to claim 4 , wherein the trained generative model is stored in the trained model storage unit as the trained model. The image generation device according to claim 5, characterized in that the learning generation unit generates the pseudo camera image from the sensor information and a degraded camera image obtained by processing the camera image acquired as the learning data so that at least a portion of the visual information of the subject contained in the camera image is missing. the learning unit stores the camera images acquired as the learning data in a generative model storage unit that stores the generative model as camera images for image generation;
The image generating device according to claim 5 or 6, wherein the learning generation unit trains the generative model to generate the pseudo camera image from the learning data and the camera image for image generation. an input switching unit that inputs the camera image and the sensor information to either the image generation unit or the learning unit based on the sharpness;
the image generation unit generates the pseudo camera image when the input switching unit inputs the camera image and the sensor information to the image generation unit;
the learning unit causes the trained model to learn when the input switching unit inputs the camera image and the sensor information to the learning unit;
The image generation device according to any one of claims 4 to 7, characterized in that the output switching unit outputs a pseudo camera image when the input switching unit inputs the camera image and the sensor information to the image generation unit, and outputs the camera image when the input switching unit inputs the camera image and the sensor information to a learning unit. The image generating device according to claim 1 , wherein the sensor information acquiring unit acquires millimeter wave sensor information obtained by detecting the object to be photographed using a millimeter wave sensor. 1. A computer-implemented image generation method comprising:
a camera image acquisition step of acquiring a camera image which is visual information of the subject;
a sensor information acquisition step of acquiring sensor information obtained by detecting the object to be photographed using an object sensor;
a step of outputting the pseudo camera image from the camera image acquired in the camera image acquiring step and the sensor information acquired in the sensor information acquiring step, using a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image, which is data that simulates visual information of the subject, from the camera image and the sensor information;
a clarity determination step of determining clarity of the camera image acquired in the camera image acquisition step;
an output switching step of outputting either the camera image or the pseudo camera image based on the sharpness;
An image generating method comprising: a camera image acquisition step of acquiring a camera image which is visual information of the subject;
a sensor information acquisition step of acquiring sensor information obtained by detecting the object to be photographed using an object sensor;
a step of outputting the pseudo camera image from the camera image acquired in the camera image acquiring step and the sensor information acquired in the sensor information acquiring step, using a trained model for generating a pseudo camera image, which is a composite image obtained by superimposing the camera image and a sensor image, which is data that simulates visual information of the subject, from the camera image and the sensor information;
a clarity determination step of determining clarity of the camera image acquired in the camera image acquisition step;
an output switching step of outputting either the camera image or the pseudo camera image based on the sharpness;
An image generating program that causes a computer to execute the above.