JP7679900B2 - Image recording device and image recording method - Google Patents

Description

The present invention relates to an image recording device and an image recording method.

When driving a vehicle and passing an old car, it may be desirable to photograph the old car and record it as an image. In such a case, it is necessary to photograph and record the old car manually, which is troublesome and causes safety problems. Here, Patent Document 1 discloses a technology related to a driving support device.
The driving assistance device in Patent Document 1 recognizes the stop lamps, etc. of the preceding vehicle from external video captured while driving, and starts recording the video when it recognizes that the lights of the preceding vehicle are on.

JP 2012-221134 A

However, in the technology disclosed in Patent Document 1, since the image continues to be recorded each time image recognition is performed, the same vehicle is photographed and recorded multiple times, which causes a problem of insufficient storage capacity.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image recording device and an image recording method for recording useful images while reducing the use of memory capacity.

The first aspect of the present invention is an image recording device that includes an image recognition unit that uses a recognition model for recognizing a predetermined recognition target to calculate a recognition accuracy, which is the likelihood that the recognition target has been recognized from a captured image of the vehicle's periphery, and determines that the recognition target has been recognized if the recognition accuracy is equal to or greater than a predetermined threshold value; a recording control unit that records the captured image in a recording device when the recognition target is recognized; and a model update unit that updates the recognition model to increase the recognition accuracy threshold when the recognition target is recognized.

A second aspect of the present invention is an image recording method that includes an image recognition step of calculating a recognition accuracy, which is the likelihood that a recognition target has been recognized from a captured image of the vehicle's periphery, using a recognition model for recognizing a predetermined recognition target, and determining that the recognition target has been recognized if the recognition accuracy is equal to or greater than a predetermined threshold value; a recording control step of recording the captured image in a recording device if the recognition target has been recognized; and a model update step of updating the recognition model to increase the recognition accuracy threshold if the recognition target has been recognized.

The present invention provides an image recording device and an image recording method for recording useful images while minimizing the use of memory capacity.

1 is a block diagram showing a configuration of an image recording apparatus according to a first embodiment of the present invention. 5 is a flowchart showing the flow of an image recording process according to the first embodiment. FIG. 11 is a block diagram showing a configuration of an image recording apparatus according to a second embodiment of the present invention. 10 is a flowchart showing the flow of an image recording process according to the second embodiment.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings. In each drawing, the same elements are given the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

<Embodiment 1>
1 is a block diagram showing the configuration of an image recording device 100 according to the first embodiment. The image recording device 100 is, for example, a part of a so-called drive recorder mounted on a moving body such as an automobile. The image recording device 100 includes an image acquisition unit 110, an image recognition unit 120, a dictionary 130, a recording control unit 140, a recording device 150, and a model update unit 160.

The image recording device 100 is connected to an on-board camera that captures images of the surroundings of the vehicle.
The image acquisition unit 110 periodically acquires images captured by the vehicle-mounted camera and outputs them to the image recognition unit 120 .

The dictionary 130 is an example of a recognition model for recognizing a predetermined recognition target, and is also called a trained model. Here, the recognition target is information that defines the feature amount of a two-dimensional shape of an object that is to be recorded and saved as a collection, and examples of the recognition target include, but are not limited to, popular popular car models, vintage cars, famous places, historical sites, buildings such as stores, tourist spots such as tourist spots and landscapes, and people's clothing. The recognition model can be realized by a neural network, a support vector machine, etc.

The dictionary 130 may also output a recognition result indicating whether or not the recognition target has been recognized from the input photographed image. In particular, when the recognition accuracy exceeds a threshold value, the dictionary 130 may output a result indicating whether or not the recognition target has been recognized from the photographed image.
It may be determined that the recognition target has been recognized. In other words, the dictionary 130 may determine that the recognition target has been recognized when the recognition target is included in the captured image with a certainty equal to or higher than a predetermined value. The recognition result may include a label for the recognition target included in the captured image. The dictionary 130 may be a model capable of recognizing two or more recognition targets. The dictionary 130 may be present outside the image recording device 100 and connected to the image recording device 100 via a communication line.

The image recognition unit 120 uses the dictionary 130 to recognize a recognition target from a captured image of the periphery of the vehicle. The image recognition unit 120 performs calculations on the feature amounts of the input image data using a predetermined logic, calculates the likelihood (recognition accuracy) of the defined recognition target, and outputs it. The recognition accuracy is calculated, for example, as a value from 0 to 1, where 0 indicates that the target is definitely not a recognition target and 1 indicates that the target is definitely a recognition target, and the higher the value, the more likely (the higher the recognition accuracy). The recognition accuracy may be calculated as a percentage, for example, as a value from 0 to 100. For example, the dictionary 130 is a program module or a model formula in which a process is implemented to input a captured image, calculate feature amounts from the captured image, perform a predetermined calculation on the feature amounts using set parameters, and output the calculation result. Here, the parameters are also called weighting coefficients. The image recognition unit 120 records the recognition result in the recording control unit 1
40 and the model update unit 160.

When a recognition target is recognized in a captured image, the recording control unit 140 records the captured image in the recording device 150. Here, when a recognition target is recognized, it refers to a case where it is determined that the recognition target is included in the captured image in the recognition result by the dictionary 130. Alternatively, when a recognition target is recognized, it may be a case where the recognition accuracy exceeds a threshold value.

The recording device 150 is a non-volatile storage device such as a hard disk or a flash memory. The recording device 150 may be a storage medium such as an SD card (registered trademark), a slot for inserting the storage medium, and a controller for reading and writing the storage medium.
The photographed image 151 is stored. The photographed image 151 is associated with a photographing time or a recording time, etc. The photographed image 151 may also be labeled with a recognized object (recognition target).

When a recognition target is recognized and the recognition result satisfies a predetermined condition, the model update unit 160 updates the recognition model, i.e., the dictionary 130, so as to suppress recognition of the recognition target after the captured image is recorded in the recording device 150. As a result, when an equivalent recognition target is subsequently captured, recognition is suppressed, and it becomes easier to suppress additional recording of the captured image 151 in the recording device 150. In other words, when a recognition target is recognized and a captured image is recorded in the recording device 150, recording of the same recognition target in the recording device 150 is suppressed thereafter. Therefore, it is possible to record useful images while suppressing the use of memory capacity.

Here, the predetermined condition may be when the object to be recognized is photographed in a captured image at a size equal to or larger than a predetermined value, or when the object to be recognized is photographed continuously for a predetermined period of time or more. For example, if the object to be recognized is photographed in a captured image at a certain size or larger, it can be said that it is sufficient to record the object to be recognized. Alternatively, if a group of images (video) is captured in which the object to be recognized can be grasped in motion rather than as a still image, it can be said that it is sufficient to record the object to be recognized. Therefore,
By setting any of these as the predetermined condition, it is possible to hold a high-quality image group while suppressing further additional recording. For example, a size equal to or greater than a predetermined value means that the image is captured with a pixel count of 10% or 5% or more of the total pixel count of the captured image, or that the pixel count of the recognition target is a predetermined value, for example, 100,000 pixels or more. Furthermore, a period of time equal to or greater than a predetermined time means that the image is captured for a period of 30 seconds or more. The predetermined conditions are not limited to those described above.

Furthermore, if the recognition target included in the captured image meets the predetermined condition and is captured at a size equal to or larger than a predetermined value, the recording control unit 140 may delete the previously recorded captured image 151 that is smaller than the predetermined value. This allows useful images to be kept and unnecessary images to be deleted, improving the quality of the recording device 1.
50 memory capacity can be effectively utilized.

Furthermore, when the recognition target is recognized and the recognition result satisfies a predetermined condition, the model update unit 160 may update the dictionary 130 so as to raise the threshold for the recognition accuracy thereafter. As a result, the threshold for the recognition accuracy is raised in the next image recognition, so that when an equivalent recognition target is photographed thereafter, it will not be recognized with the same or lower recognition accuracy, and recording of unnecessary images is suppressed.

The model update unit 160 may also set the predetermined condition to be a case where the ratio of the recognition target in the captured image is equal to or greater than a predetermined value. In this case, when the recognition target is recognized and the recognition result satisfies the predetermined condition, the model update unit 160 updates the dictionary 130 so as to increase the predetermined value of the ratio of the recognition target thereafter. For example, when the recognition target is included in the captured image but is overlapped and blocked by another object, it becomes difficult to recognize. In addition, when the recognition target is included in a larger ratio in the captured image, it can be prioritized as a recording target. Furthermore, from then on, the image with a higher ratio of the recognition target in the captured image becomes the recording target, so the quality of the captured image 151 is improved. At this time, the recording control unit 140 may delete the captured images 151 recorded in the past that have a ratio less than the predetermined value.

Alternatively, when a recognition target is recognized and the recognition result satisfies a predetermined condition, the model update section 160 may update the recognition model so that the predetermined recognition target is not recognized thereafter.
For example, the model update unit 160 may delete the recognition target from the dictionary 130. That is, the model update unit 160 may exclude the recognition target from the dictionary 130.

The dictionary 130 may include a plurality of attributes and priorities for a specific target object as the recognition target. For example, if the recognition target is an old car, the attributes include color, front, rear, left and right, and whether or not it has been modified. Each attribute is individually associated with a priority in recognition. In this case, the recognition result includes the type of object to be recognized as well as the attributes included in the captured image. The model update unit 160 sets the recognition result to include the above attributes as a predetermined condition. For example, it is assumed that the recognition result includes the attributes of the blue front and right side of the old car. In this case, the model update unit 160 updates the dictionary 130 so as to lower the priorities of the attributes "blue", "front", and "right side" of the old car from now on. In other words, recognition and recording of the captured image of the old car with the attributes "blue", "front", and "right side" are suppressed from now on. Therefore, from now on, the dictionary 130 judges that the recognition was successful (exceeded the recognition accuracy) when the input captured image is a color other than "blue" of a certain old car, or is the left side or rear. In other words, "blue",
The recognition is performed with priority over the cases of "front" and "right". For example, the priority may be associated with a recognition accuracy threshold. Therefore, attributes of the recognition target that have already been recorded in the recording device 150 are less likely to be recorded, and unrecorded attributes are preferentially recorded.

The image recording device 100 includes a processor and a memory as components not shown. The storage device 150 stores a computer program that implements the processes of the image recognition unit 120, the recording control unit 140, and the model update unit 160. The processor then reads the computer program from the storage device 150 into the memory and executes the computer program. In this way, the processor realizes the functions of the image recognition unit 120, the recording control unit 140, and the model update unit 160.

2 is a flowchart showing the flow of the image recording process according to the first embodiment. First, the image acquisition unit 110 acquires a captured image (S101). Next, the image recognition unit 120
Using the recognition model (dictionary 130), the recognition target is recognized from the captured image of the surroundings of the vehicle (S
Then, the image recording device 100 judges whether the image recognition has been successful (S1
03) In other words, it is determined whether the recognition target has been recognized.

If the recognition target is recognized, the recording control unit 140 records the captured image in the recording device 150 (S104). If the recognition target is recognized, the model update unit 160 determines whether the recognition result satisfies a predetermined condition (S105). If the recognition result satisfies the predetermined condition, the model update unit 160 updates the recognition model so as to suppress recognition of the recognition target (S106).
).

After that, the process returns to step S101. If the determination in step S103 or S105 is NO, the process also returns to step S101.

In this way, according to this embodiment, when a user passes near an object that the user wants to collect while driving a car or the like, image recognition is performed on the image captured by the on-board camera using the dictionary 130, so that the image of the object can be automatically recorded and collected.
If a recorded image satisfies a predetermined condition, recognition of similar captured images is suppressed in the future, and additional recording of the captured image 151 in the recording device 150 is suppressed. Therefore, it is possible to record useful images while suppressing the use of memory capacity.

<Embodiment 2>
The second embodiment is an improvement of the first embodiment. FIG. 3 is a block diagram showing the configuration of an image recording device 100a according to the second embodiment. The image recording device 100a includes:
1, the model update unit 160 is replaced with a model update unit 160a, and an acquisition unit 170 is added. Other configurations are the same as those in FIG. 1, so duplicated descriptions will be omitted where appropriate.

The acquisition unit 170 acquires learning data for a new recognition target from the outside. Here, the new recognition target may be, for example, an image of a new target object desired by the user, or an image of an additional attribute of a recognition target that has already been learned. Alternatively, the new recognition target may be an image of a stolen car or a violator provided by the police, a local government, or the like. In other words, the acquisition unit 17
The acquisition unit 170 acquires a group of images received from a provider via a network or the like as learning data. The acquisition unit 170 acquires a recognition model (dictionary 13) that is a trained model of a new recognition target from the outside.
0) may be obtained.

In this embodiment, the model update unit 160a includes a learning unit for performing learning using the learning data acquired by the acquisition unit 170 and updating the recognition model (dictionary 130). This makes it possible to effectively learn the dictionary 130. As a modified example, the model update unit 160a may not include a learning unit, and may instead introduce the recognition model (dictionary 130) acquired by the acquisition unit 170 in place of a conventional recognition model.

4 is a flowchart showing the flow of the image recording process according to the embodiment 2. First, the acquisition unit 170 acquires learning data distributed from an external source (S201). Next, the model update unit 160a learns using the learning data and updates the recognition model (dictionary 130) (S202).

In this way, in the second embodiment, new recognition objects provided from outside are additionally learned in the dictionary 130, making the dictionary 130 more effective. For example, when performing a search in a specific area, images of the search object are simultaneously distributed to a large number of vehicles in the specific area, so that the dictionary 130 of each vehicle can be instantly trained. Therefore, a search can be performed efficiently in a short time.

<Other embodiments>
The image capturing apparatus further includes a position information acquiring unit (not shown) for acquiring position information at which the captured image was captured,
The model update unit 160 may set the predetermined condition as a case where the location information of the location where the recognition target is recognized matches the location information of the location where the same recognition target was recognized in the past. This makes it possible to prevent images with the same background from being recorded in duplicate. Therefore, it is possible to record a wider variety of images while suppressing the use of memory capacity. The location information can be obtained, for example, from GNSS (Global Navigation Satellite).
This can be obtained through the Digital Ignition Satellite System.

The model update unit 160 may further include a weather information acquisition unit (not shown) that acquires weather information when the photographed image is taken, and may set the weather information when the recognition target is recognized as being better than the weather information when the same recognition target was recognized in the past as a predetermined condition. This allows at least a minimum amount of recording to be made even when the weather is bad, for example, on a rainy day when the recognition target is not clearly visible. Furthermore, even for the same recognition target, additional recording is not made if the weather is as bad as the one shown, but additional recording is made only when the image is taken in better weather. Weather information can be acquired, for example, by communication from a server on the Internet.

In addition, when the image recognition unit 120 recognizes a recognition target from a captured image of the periphery of the vehicle, it is preferable that the image recognition unit 120 notifies the user that the recognition target has been recognized by a notifying unit (not shown). The notifying unit having a speaker or a monitor notifies the user that the recognition target has been recognized by voice or image. The notifying unit may be in a form that notifies the user that the captured image has been recorded when the recording control unit 140 records the captured image in the recording device 150.

While the present invention has been described above in accordance with the above-described embodiment, the present invention is not limited to the configuration of the above-described embodiment, and naturally includes various modifications, alterations, and combinations that may be made by a person skilled in the art within the scope of the invention as defined in the claims of the present application.

For example, in the above embodiment, when a recognition target is recognized and the recognition result satisfies a predetermined condition, the model update unit updates the recognition model so that the same recognition target is not recognized thereafter, but when a recognition target is recognized and the recognition result satisfies a predetermined condition, the recording control unit may suppress recording so that the same recognition target is not recorded thereafter. In this case, when a recognition target is included in a captured image, it is recognized but not recorded, and the same effect as the above embodiment can be obtained.

Any of the above-mentioned processes of the communication device can be realized by having a CPU (Central Processing Unit) execute a computer program. In this case, the computer program can be stored and provided to the computer using various types of non-transitory computer readable media. The non-transitory computer readable media includes various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)). The program may also be provided to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path, such as an electric wire or optical fiber, or via a wireless communication path.

The present invention also includes not only the case where the functions of the above-mentioned embodiments are realized by a computer executing a program that realizes the functions of the above-mentioned embodiments, but also the case where the functions of the above-mentioned embodiments are realized in cooperation with an OS (Operating System) or application software running on a computer. Furthermore, the present invention also includes the case where the functions of the above-mentioned embodiments are realized by a function expansion board inserted into a computer or a function expansion unit connected to a computer, in which all or part of the processing of the program is performed.

REFERENCE SIGNS LIST 100 Image recording device 110 Image acquisition unit 120 Image recognition unit 130 Dictionary 140 Recording control unit 150 Recording device 151 Photographed image 160 Model update unit

Claims (3)

an image recognition unit that calculates a recognition accuracy, which is a likelihood of recognizing a predetermined recognition target from an image of the surroundings of the vehicle captured by a camera mounted on the vehicle , and that has a recognition model that determines that the recognition target has been recognized when the recognition accuracy is equal to or greater than a predetermined threshold;
a recording control unit that records the captured image in a recording device when the recognition target is recognized;
a position information acquisition unit that acquires position information at the time when the photographed image was photographed by using a GNSS;
a model update unit that updates the recognition model so as to increase the threshold value of the recognition accuracy when the recognition target is recognized from the captured image , the recognition result satisfies a predetermined condition, and the position information when the recognition target is recognized matches position information when the same recognition target was recognized in the past;
An image recording device comprising: the image recognition unit outputs a recognition result including attributes of the object to be recognized;
the model update unit extracts the attribute included in the recognition result when the recognition target is recognized, and updates the recognition model for the recognition target including the extracted attribute so as to increase the threshold of the recognition accuracy.
2. The image recording device according to claim 1. an image recognition step of calculating a recognition accuracy, which is a likelihood of recognizing a predetermined recognition target from an image of the periphery of the vehicle captured by a camera mounted on the vehicle, and providing a recognition model that determines that the recognition target has been recognized when the recognition accuracy is equal to or greater than a predetermined threshold;
a recording control step of recording the captured image in a recording device when the recognition target is recognized;
a position information acquisition step of acquiring position information at the time when the photographed image was photographed by using a GNSS;
a model updating step of updating the recognition model so as to increase the threshold value of the recognition accuracy when the recognition target is recognized from the captured image , the recognition result satisfies a predetermined condition, and the position information when the recognition target is recognized matches position information when the same recognition target was recognized in the past;
An image recording method comprising:

Images