JP7552149B2 - IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and an image processing program that recognize objects contained in an image.

Conventionally, image processing devices that recognize vehicles, pedestrians, etc. from images in road traffic are known. Such image processing devices recognize vehicles, pedestrians, etc. from images captured by a camera or the like, using dictionary data learned from multiple example images in which vehicles, pedestrians, etc. are recorded. The image processing device improves the accuracy of image analysis by updating the dictionary data. For example, Patent Document 1 discloses an image processing system that updates the dictionary data by receiving input of erroneous recognition or missed recognition from a user.

Patent No. 5333080

In the image processing system of Patent Document 1, before updating the dictionary data, the image processing device evaluates by simulation or the like whether the updated dictionary data has an analysis accuracy equal to or higher than a predetermined standard, and performs the update if the analysis accuracy is calculated to be equal to or higher than the standard. In such an image processing device, depending on how it is actually operated, the analysis results of vehicles, pedestrians, etc. based on the updated dictionary data may not match the simulation. For this reason, the analysis accuracy may decrease after updating the dictionary data.

The present invention has been made to solve the above problems, and provides an image processing device, an image processing method, and an image processing program that prevent a decrease in analysis accuracy after updating dictionary data.

The image processing device according to the present invention includes a control device and a storage device in which first dictionary data and second dictionary data different from the first dictionary data are stored. The control device includes an image acquisition unit that acquires an image, an analysis unit that analyzes the image based on the first dictionary data over a first analysis time to detect objects and analyzes the image based on the second dictionary data over a second analysis time to detect objects, an evaluation unit that evaluates the analysis accuracy based on the second dictionary data for each environmental condition and time period for which a reference value range is set, depending on whether the number of objects detected using the second dictionary data is within a reference value range under the same conditions, and a change unit that gradually changes the first analysis time and the second analysis time for each time period for which analysis was performed using the second dictionary data , based on the evaluation result of the analysis accuracy for each time period for which analysis was performed using the second dictionary data. The reference value range is set in advance for each environmental condition and time period based on the actual measured value of the number of objects and the number of objects detected using the first dictionary data, and the evaluation unit evaluates the analysis accuracy only under the environmental conditions and time period for which the reference value range is set .

In the present invention, the image processing device changes the first analysis time and the second analysis time in stages based on the evaluation results of the analysis accuracy. That is, the image processing device does not completely replace the dictionary data used from the first dictionary data to the second dictionary data, but changes the ratio at which the first dictionary data and the second dictionary data are used. Therefore, the image processing device can limit the effect on the analysis accuracy due to updating the dictionary data, and prevent the analysis accuracy from decreasing after updating the dictionary data.

1 is a functional block diagram showing an image processing device 1 according to a first embodiment. FIG. 4 is a diagram showing a dictionary-in-use table 33 according to the first embodiment. FIG. 4 is a diagram for explaining a criterion for evaluating the analysis accuracy according to the first embodiment. 10A and 10B are diagrams for explaining changes in a first analysis time and a second analysis time according to the first embodiment; 4 is a flowchart showing an image processing method according to the first embodiment.

Embodiment 1.
An image processing device 1 according to the first embodiment will be described below with reference to the drawings. FIG. 1 is a functional block diagram showing the image processing device 1 according to the first embodiment. The image processing device 1 according to the first embodiment is a device that performs vehicle detection and traffic volume measurement on a road. As shown in FIG. 1, the image processing device 1 has a control device 2 and a storage device 3. The image processing device 1 is also connected to an imaging device 9. The imaging device 9 is a camera installed on the roadside to capture images of the road, and transmits the captured images to the image processing device 1.

The control device 2 recognizes objects from images captured by the imaging device 9 using the first dictionary data 31, the second dictionary data 32, and the used dictionary table 33 stored in the storage device 3. In this embodiment, the objects are vehicles. The control device 2 also recognizes vehicles by model. The control device 2 is connected to a server (not shown) and can update the dictionary data.

The control device 2 is configured with a CPU (Central Processing Unit) or an ASIC (Application Specific Integrated Circuit), etc. The control device 2 has an image acquisition unit 21, a condition acquisition unit 22, an analysis unit 23, an evaluation unit 24, and a change unit 25. The image acquisition unit 21, the condition acquisition unit 22, the analysis unit 23, the evaluation unit 24, and the change unit 25 are realized by the CPU or the ASIC, etc., executing a program. The detailed configurations of the image acquisition unit 21, the condition acquisition unit 22, the analysis unit 23, the evaluation unit 24, and the change unit 25 will be described later.

The storage device 3 stores first dictionary data 31, second dictionary data 32, and a dictionary table 33. The dictionary data is a detection model that learns the characteristics of an object based on multiple example images containing the object. The first dictionary data 31 is dictionary data that has been used since before the dictionary data was updated. The second dictionary data 32 is dictionary data that was stored in the storage device 3 when the dictionary data was updated. In other words, the second dictionary data 32 is dictionary data that was introduced into the image processing device 1 more recently than the first dictionary data 31. The second dictionary data 32 is, for example, obtained from a server (not shown) connected to the image processing device 1 via the Internet, or is input by the user via an input device (not shown).

Figure 2 is a diagram showing the dictionary in use table 33 according to the first embodiment. The dictionary in use table 33 is composed of time periods, start times, end times, and dictionary in use data. The dictionary in use table 33 indicates, based on the current time, whether the control device 2 is to use the first dictionary data 31 or the second dictionary data 32. The time periods are broadly divided into early morning, morning, afternoon, evening, and night. Each time period is associated with a plurality of combinations of start times, end times, and dictionary in use data. The start times and end times each store a time in minutes. Either the first dictionary data 31 or the second dictionary data 32 is stored as the dictionary in use data.

That is, each record of the dictionary usage table 33 corresponds to the first dictionary data 31 or the second dictionary data 32 to be used for the time from the start time to the end time. Also, the first dictionary data 31 and the second dictionary data 32 are linked to each time period. For example, image analysis is performed using the second dictionary data 32 from 5:00 to 5:20 in the early morning, and image analysis is performed using the first dictionary data 31 from 5:20 to 6:00 in the early morning. Here, the time when the first dictionary data 31 is used is referred to as the first analysis time. Also, the time when the second dictionary data 32 is used is referred to as the second analysis time. That is, the first analysis time in the early morning time period is 40 minutes, and the second analysis time is 20 minutes.

As described above, the first dictionary data 31 and the second dictionary data 32 are linked to each time period. That is, a first analysis time and a second analysis time exist for each time period. Therefore, when the first analysis time or the second analysis time is simply described, it means the general term for the first analysis time or the second analysis time for each time period. Also, when the time period is specified, for example, the first analysis time "early morning" or the second analysis time "early morning", it means that it is limited to the relevant time period. Furthermore, although there may be multiple first analysis times or multiple second analysis times in a particular time period, it means the general term for the first analysis time and the second analysis time included in the time period unless otherwise specified.

The image acquisition unit 21 acquires images captured by the imaging device 9. The condition acquisition unit 22 acquires the current time and environmental information. The environmental information is, for example, the weather at the location where the image processing device 1 is installed. Weather information broadly categorized as sunny, cloudy, rainy, etc. is received from the server.

The analysis unit 23 refers to the dictionary table 33 to determine the dictionary data to be used from the current time. The analysis unit 23 also uses the determined dictionary data to analyze the image and detect objects. This allows the image processing device 1 to measure traffic volume on roads. For example, a convolutional neural network is used to analyze the image.

The evaluation unit 24 evaluates the analysis accuracy based on the second dictionary data 32. Specifically, the evaluation unit 24 evaluates the analysis accuracy as follows. FIG. 3 is a diagram for explaining the criteria for evaluating the analysis accuracy according to the first embodiment. As shown in FIG. 3, a reference value range is set for some time periods and weather conditions with respect to the number of vehicles of some vehicle types detected based on the second dictionary data 32.

The reference value range is set when the second dictionary data is updated. Each reference value range is set, for example, in advance, based on the first dictionary data for the actual measurement value of the object when conditions such as the day of the week, time period, and weather are the same, and is set to a value that assumes that the object can be detected more accurately. For example, if the actual measurement value of the object, a large vehicle, is 30, and the number of large vehicles analyzed based on the first dictionary data is 100, the reference value range is set to 10 to 60. The reference value range may be adjusted each time, for example, based on the analysis results from a few minutes ago. The reference value range may also be set for multiple vehicle types.

The evaluation unit 24 evaluates the analysis accuracy based on the second dictionary data 32 only during the time period and weather conditions for which the reference value range is set. If the number of detected objects is within the reference value range, the evaluation unit 24 evaluates that the analysis accuracy is improved by using the second dictionary data 32 compared to using the first dictionary data 31. For example, according to FIG. 3, if it is sunny in the early morning hours and 10 to 60 large vehicles can be detected per hour, the evaluation unit 24 evaluates that the analysis accuracy is improved by using the second dictionary data 32. Also, if it is cloudy or rainy in the early morning hours, the evaluation unit 24 does not evaluate the analysis accuracy based on the second dictionary data 32. On the other hand, if the number of objects detected based on the second dictionary data 32 is outside the reference value range, the evaluation unit 24 evaluates that the analysis accuracy is reduced by using the second dictionary data 32. In addition, when the reference value range is set for multiple vehicle types, if the number of vehicles of each detected vehicle type is within the corresponding reference value range, it is evaluated that using the second dictionary data 32 will improve the analysis accuracy more than using the first dictionary data 31.

The change unit 25 changes the first analysis time and the second analysis time stepwise based on the evaluation result of the analysis accuracy. Specifically, when the analysis accuracy is evaluated to have improved by using the second dictionary data 32, the change unit 25 shortens the first analysis time by one step and lengthens the second analysis time by one step in the time period in which the analysis was performed. Here, one step is N minutes, which corresponds to 1/n of the second analysis time. On the other hand, when the analysis accuracy is evaluated to have decreased by using the second dictionary data 32, the change unit 25 lengthens the first analysis time by one step and shortens the second analysis time by one step in the time period in which the analysis was performed. The analysis of the day and the change based on the evaluation are reflected in the storage device 3 so as to be used in the analysis on the following day and thereafter. Note that the first analysis time and the second analysis time are not changed in the time period in which the reference value range is not set and in the case of weather.

For example, if it is evaluated that the analysis accuracy will be improved by using the second dictionary data 32 in the early morning hours of the day, the change unit 25 delays the end time of the analysis based on the second dictionary data 32 by 5 minutes. At the same time, the change unit 25 delays the start time of the analysis based on the first dictionary data 31 by 5 minutes. That is, the change unit 25 changes the analysis based on the second dictionary data 32 that was performed from 5:00 to 5:20 on the day to be performed from 5:00 to 5:25. Also, the change unit 25 changes the analysis based on the first dictionary data 31 that was performed from 5:20 to 6:00 to be performed from 5:25 to 6:00. As a result, the first analysis time in the early morning of the next day is shortened by 5 minutes, and the second analysis time in the early morning is lengthened by 5 minutes.

On the other hand, if the change unit 25 determines that the analysis accuracy will decrease by using the second dictionary data 32 in the early morning hours of the day, the change unit 25 advances the end time of the analysis based on the second dictionary data 32 by 5 minutes. At the same time, the change unit 25 advances the start time of the analysis based on the first dictionary data 31 by 5 minutes. That is, the change unit 25 changes the analysis based on the second dictionary data 32 that was performed from 5:00 to 5:20 on the day to be performed from 5:00 to 5:15. Also, the change unit 25 changes the analysis based on the first dictionary data 31 that was performed from 5:20 to 6:00 to be performed from 5:15 to 6:00. As a result, the first analysis time in the early morning of the next day is extended by 5 minutes, and the second analysis time in the early morning is shortened by 5 minutes.

When there are multiple first analysis times or multiple second analysis times in a particular time period, the evaluation unit 24 performs evaluation for each second analysis time. The modification unit 25 then modifies the second analysis time for which evaluation has been performed and the first analysis time that follows this second analysis time. For example, as shown in FIG. 3, the second analysis time from 9:00 to 10:00 and the first analysis time from 10:00 to 12:00 are modified based on the analysis accuracy results for the second analysis time from 9:00 to 10:00 in the daytime.

Figure 4 is a diagram for explaining changes to the first analysis time and the second analysis time according to embodiment 1. As shown in Figure 4, when analysis accuracy improves by using the second dictionary data 32, as days pass, the proportion of the first analysis time T1 decreases and the proportion of the second analysis time T2 increases. Eventually, all dictionary data used by the analysis unit 23 is switched to the second dictionary data 32.

Figure 5 is a flowchart showing an image processing method according to the first embodiment. The operation of the image processing device 1 will be described with reference to Figure 5. The image processing method is executed by the control device 2 by the image processing program. First, the image acquisition unit 21 executes an image acquisition step of acquiring an image from the imaging device 9 (S1). The condition acquisition unit 22 executes a condition acquisition step of acquiring the current time and environmental information (S2). Next, the analysis unit 23 refers to the dictionary table 33 in use and determines whether the current time corresponds to the time to perform analysis using the second dictionary data 32 (S3). If the current time corresponds to the time to perform analysis using the first dictionary data 31 (S3: NO), the analysis unit 23 executes a first analysis step of performing analysis using the first dictionary data 31 stored in the storage device 3 (S4).

If the current time corresponds to the time for performing analysis using the second dictionary data 32 (S3: YES), the analysis unit 23 executes a second analysis step in which analysis is performed using the second dictionary data 32 stored in the storage device 3 (S5). Next, the evaluation unit 24 executes a compilation step in which the analysis results during the time are compiled (S6). Then, an evaluation step is executed in which the analysis accuracy based on the second dictionary data 32 is evaluated (S7). If it is determined that the analysis accuracy will decrease (S7: NO), the modification unit 25 executes a modification step in which the first analysis time is increased by one step and the second analysis time is decreased by one step (S8). Also, if it is determined that the analysis accuracy will increase (S7: YES), the modification unit 25 executes a modification step in which the first analysis time is decreased by one step and the second analysis time is increased by one step (S9).

In the first embodiment, the image processing device 1 gradually changes the first analysis time and the second analysis time based on the evaluation result of the analysis accuracy. That is, the image processing device 1 does not completely replace the dictionary data used from the first dictionary data 31 to the second dictionary data 32, but changes the ratio at which the dictionary data is used between the first dictionary data 31 and the second dictionary data 32. Therefore, the image processing device 1 can limit the effect on the analysis accuracy due to the update of the dictionary data, and suppress a decrease in the analysis accuracy after the dictionary data is updated.

The image processing device 1 also switches between the first dictionary data 31 and the second dictionary data 32 at different times. This means that the image processing device 1 does not need to parallelize multiple dictionary data using a server or the like, and can update the dictionary data on a single device.

The above is a description of the image processing device 1 in the embodiment, but the image processing device 1 of the present invention can be modified in various ways in addition to the configuration disclosed in the embodiment.

For example, the image processing device 1 may be used for purposes other than traffic volume measurement by replacing the dictionary data with data learned from objects other than vehicles. Objects other than vehicles include, for example, pedestrians on the road.

The changes to the first analysis time and the second analysis time may be reflected in the storage device 3 immediately after the analysis accuracy is evaluated. Furthermore, based on the evaluation of the analysis accuracy in a specific time period, the second analysis time may be changed for other time periods or for all time periods. For example, if it is evaluated that the analysis accuracy is improved by using the second dictionary data 32 in the early morning hours, the second analysis time in the daytime may also be lengthened.

In addition, when there are multiple first analysis times or multiple second analysis times in a specific time period, the change unit 25 may change a first analysis time other than the second analysis time for which the evaluation was performed and a first analysis time subsequent to this second analysis time. That is, the change unit 25 may change other first analysis times and second analysis times that are not subsequent to the second analysis time in the same time period as the second analysis time for which the evaluation was performed, or all first analysis times and second analysis times in the same time period as the second analysis time for which the evaluation was performed.

However, if the modification unit 25 determines that the analysis accuracy would decrease by using the second dictionary data 32, it may maintain the second analysis time rather than shortening it.

The image processing device 1 may also have a detection unit that detects the weather. Furthermore, the image processing device 1 may be configured as an integral part of the imaging device 9.

In addition, the first dictionary data 31, the second dictionary data 32, and the used dictionary table 33 may be stored not in the storage device 3 but in a cloud server or the like external to the image processing device 1.

1 Image processing device, 2 Control device, 3 Storage device, 9 Imaging device, 21 Image acquisition unit, 22 Condition acquisition unit, 23 Analysis unit, 24 Evaluation unit, 25 Change unit, 31 First dictionary data, 32 Second dictionary data, 33 Used dictionary table.

Claims (7)

A control device;
a storage device in which first dictionary data and second dictionary data different from the first dictionary data are stored;
The control device includes:
an image acquisition unit that acquires an image;
an analysis unit that analyzes the image based on the first dictionary data over a first analysis time to detect an object, and analyzes the image based on the second dictionary data over a second analysis time to detect the object;
an evaluation unit that evaluates the analysis accuracy based on the second dictionary data depending on whether or not the number of the objects detected using the second dictionary data is within the reference value range under the same conditions for each environmental condition and time period for which a reference value range is set ; and
a change unit that gradually changes the first analysis time and the second analysis time for each time period in which analysis is performed using the second dictionary data , based on an evaluation result of the analysis accuracy for each time period in which analysis is performed using the second dictionary data ,
the reference value range is set in advance for each of the environmental conditions and the time period based on an actual measurement value of the number of objects and the number of objects detected using the first dictionary data,
The evaluation unit evaluates the analysis accuracy only under the environmental conditions and the time period for which the reference value range is set.
Image processing device. The change unit is
2. The image processing device according to claim 1, wherein when the number of objects detected based on the second dictionary data during the time period in which analysis was performed using the second dictionary data is evaluated to be within the reference value range, the first analysis time is shortened by one step and the second analysis time is lengthened by one step during the time period evaluated to be within the reference value range . The change unit is
3. The image processing device according to claim 1, wherein when the number of objects detected based on the second dictionary data during the time period in which analysis was performed using the second dictionary data is evaluated to be outside the reference value range, the first analysis time is lengthened by one step and the second analysis time is shortened by one step during the time period in which the number of objects detected based on the second dictionary data during the time period in which analysis was performed using the second dictionary data is evaluated to be within the reference value range. The storage device includes:
a dictionary table that is composed of a time period, a start time, an end time, and dictionary data to be used, and indicates, based on a current time, whether the analysis unit will perform analysis based on the first dictionary data or the second dictionary data;
During each of the aforementioned time periods,
The image processing device according to claim 1 , wherein the first analysis time and the second analysis time are included. The evaluation unit is
The image processing device according to claim 4 , wherein the analysis accuracy is evaluated for each of the time periods. an image acquisition step of acquiring an image;
a first analysis step of analyzing the image based on first dictionary data for a first analysis time to detect an object;
a second analysis step of analyzing the image based on second dictionary data different from the first dictionary data for a second analysis time to detect the object;
an evaluation step of evaluating the accuracy of analysis based on the second dictionary data according to whether or not the number of the objects detected using the second dictionary data is within the reference value range under the same conditions for each environmental condition and time period for which a reference value range is set ;
and a change process of gradually changing the first analysis time and the second analysis time for each time period in which analysis is performed using the second dictionary data , based on an evaluation result of the evaluation process for each time period in which analysis is performed using the second dictionary data ,
the reference value range is set in advance for each of the environmental conditions and the time period based on an actual measurement value of the number of objects and the number of objects detected using the first dictionary data,
In the evaluation step, the analysis accuracy is evaluated only under the environmental conditions and the time period in which the reference value range is set.
Image processing methods. an image acquisition step of acquiring an image;
a first analysis step of analyzing the image based on first dictionary data for a first analysis time to detect an object;
a second analysis step of analyzing the image based on second dictionary data different from the first dictionary data for a second analysis time to detect the object;
an evaluation step of evaluating the accuracy of analysis based on the second dictionary data according to whether or not the number of the objects detected using the second dictionary data is within the reference value range under the same conditions for each environmental condition and time period for which a reference value range is set ;
a change step of gradually changing the first analysis time and the second analysis time for each time period in which analysis is performed using the second dictionary data, based on an evaluation result of the evaluation step for each time period in which analysis is performed using the second dictionary data ;
the reference value range is set in advance for each of the environmental conditions and the time period based on an actual measurement value of the number of objects and the number of objects detected using the first dictionary data,
In the evaluation step, the analysis accuracy is evaluated only under the environmental conditions and the time period in which the reference value range is set.
Image processing program.

Images