JP7835564B2 - Information processing device, vehicle behavior determination method, and vehicle behavior determination program - Google Patents

Description

This invention relates to an information processing device, a vehicle behavior determination method, and a vehicle behavior determination program.

Traditionally, object movement has been detected by utilizing the difference between frames in video. A representative technique proposed for this is template matching. For example, images of a monitored area, such as a train station platform and tracks, are captured. Then, template matching is performed on multiple sub-regions within the captured images, comparing them to a prepared background image of the monitored area. Correlation values between these sub-regions are calculated for each sub-region. Based on these correlation values, the presence of vehicles such as trains within the monitored area is then determined.

Patent No. 5386744

By the way, recording devices that capture and record the outside view from inside a vehicle, such as dashcams, are well known.

However, the conventional technology described above uses template matching to detect whether there are moving objects, such as vehicles, within the captured image. This conventional technology cannot, for example, determine the vehicle's behavior from video footage captured from inside the vehicle (e.g., using a recording device).

The present invention aims to realize an information processing device, a vehicle behavior determination method, and a vehicle behavior determination program that can determine the behavior of a vehicle from video footage captured from inside the vehicle looking outwards.

The information processing device described in claim 1 is characterized by comprising: an acquisition unit that acquires a moving image of the outside taken from inside a vehicle; a comparison unit that compares the images of frames of the moving image acquired by the acquisition unit, for each predetermined region of the image, between frames; and a determination unit that determines the behavior of the vehicle based on the comparison result of the comparison unit.

The vehicle behavior determination method described in claim 8 is a vehicle behavior determination method executed by an information processing device, characterized by comprising: an acquisition step of acquiring a moving image of the outside from inside the vehicle; a comparison step of comparing the images of frames of the acquired moving image between frames for each predetermined region of the image; and a determination step of determining the behavior of the vehicle based on the comparison result.

The vehicle behavior determination program described in claim 9 is a vehicle behavior determination program that causes an information processing device to execute an acquisition procedure for acquiring a video image of the outside of a vehicle from inside the vehicle; a comparison procedure for comparing images of frames of the acquired video image across predetermined regions of the image; and a determination procedure for determining the behavior of the vehicle based on the comparison results.

Figure 1 shows an example of an information processing system according to the embodiment. Figure 2 shows an example of the configuration of an information processing device according to the embodiment. Figure 3 illustrates the comparison of images between frames according to the embodiment. Figure 4 is a diagram illustrating template matching according to an embodiment. Figure 5 is a diagram illustrating the determination of the vehicle's behavior according to this embodiment. Figure 6 illustrates the determination of the vehicle's behavior according to this embodiment. Figure 7 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. Figure 8 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. Figure 9 illustrates the movement of the image region within the frame due to the behavior of the vehicle VEx according to this embodiment. Figure 10 is a diagram illustrating the determination of the vehicle's behavior according to the embodiment. Figure 11 is a diagram illustrating the determination of the vehicle's behavior according to this embodiment. Figure 12 is a flowchart showing the procedure for the vehicle behavior determination process, which determines the behavior of a vehicle. Figure 13 is a hardware configuration diagram showing an example of a computer that implements the functions of an information processing device.

Below, an example of an embodiment for implementing the information processing device, vehicle behavior determination method, and vehicle behavior determination program (hereinafter referred to as "embodiment") will be described in detail with reference to the drawings. Note that the information processing device, vehicle behavior determination method, and vehicle behavior determination program are not limited by this embodiment. Furthermore, the same parts are denoted by the same reference numerals in the following embodiments, and redundant descriptions are omitted.

[1. System Configuration]
First, the configuration of the information processing system according to the embodiment will be explained using Figure 1. Figure 1 is a diagram showing an example of the information processing system according to the embodiment. Figure 1 shows information processing system 1 as an example of the information processing system according to the embodiment.

As shown in Figure 1, the information processing system 1 may comprise an in-vehicle device 10 and an information processing device 100. The in-vehicle device 10 and the information processing device 100 are connected via a network N, either by wired or wireless communication. Furthermore, the information processing system 1 shown in Figure 1 may include any number of in-vehicle devices 10 and any number of information processing devices 100.

The in-vehicle device 10 may be a dedicated navigation system built into or attached to the vehicle VEx, or it may be a recording device (dashcam) installed in the vehicle VEx for security purposes or to prevent aggressive driving.

Furthermore, the in-vehicle device 10 may consist of a navigation device and a recording device. As an example, the in-vehicle device 10 may be a composite device in which an independent navigation device and a recording device are communicated with each other. Another example is that the in-vehicle device 10 may be a single device having both navigation and recording functions.

Furthermore, users can substitute their everyday portable devices (e.g., smartphones, tablet devices, notebook PCs, desktop PCs, PDAs, etc.) with the in-vehicle device 10 by installing a predetermined application on them. For example, a portable device with a predetermined navigation application or recording application installed can be considered the in-vehicle device 10 as described herein. When a portable device is used as the in-vehicle device 10, it is installed, for example, on the dashboard of the vehicle VEx while driving.

Furthermore, the in-vehicle device 10 may be equipped with various sensors. For example, the in-vehicle device 10 may be equipped with various sensors such as a camera, accelerometer, gyroscope, GPS sensor, and barometric pressure sensor.

The information processing device 100 may acquire various types of data based on the sensor information detected by these sensors (for example, by analyzing the sensor information). For example, the information processing device 100 may acquire video data of the outside world captured from inside the vehicle VEx by a camera. The information processing device 100 may also acquire location information from a GPS sensor. The information processing device 100 may acquire sensor information detected not only by sensors provided in the in-vehicle device 10, but also by sensors provided in the vehicle VEx itself.

The information processing device 100 is a device that performs information processing according to the embodiment. For example, the information processing device 100 compares images of acquired video frames between frames, for each predetermined region of the image, and determines the behavior of the vehicle VEx based on the comparison results. Furthermore, the information processing device 100 can also provide various driving assistance to the user based on the determined behavior of the vehicle VEx.

Here, if the in-vehicle device 10 is considered an edge computer that performs edge processing near the user, then the information processing device 100 may be, for example, a cloud computer that performs processing on the cloud side. In other words, the information processing device 100 may be a server device.

Furthermore, the following embodiment shows an example in which information processing according to the embodiment is realized in the information processing system 1 by transmitting and receiving information between the in-vehicle device 10 and the information processing device 100. However, the information processing according to the embodiment may also be realized on the edge side, i.e., only on the in-vehicle device 10. In this case, the in-vehicle device 10 may be configured to behave like the information processing device 100, for example, by the vehicle behavior determination program according to the embodiment.

The in-vehicle device 10 transmits sensor information detected by the sensors to the information processing device 100. For example, the in-vehicle device 10 uses a camera to capture images of the outside from inside the vehicle VEx. For instance, the camera is positioned facing forward of the vehicle VEx, capturing images of the area in front of the vehicle VEx from inside the vehicle VEx. The in-vehicle device 10 transmits the video data of the images captured by the camera from inside the vehicle VEx to the information processing device 100. Furthermore, the in-vehicle device 10 transmits location information indicating the current position of the vehicle VEx, detected by the GPS sensor, to the information processing device 100. The in-vehicle device 10 may transmit the video images and sensor information captured by the camera in real time, or it may transmit them at periodic intervals, such as at regular intervals.

The information processing device 100 stores video data and sensor information acquired from the vehicle VEx. The information processing device 100 performs information processing according to the following procedure to determine the behavior of the vehicle VEx from the stored video data.

[2. Configuration of the Information Processing Device 100]
An information processing device 100 according to an embodiment will be described using Figure 2. Figure 2 is a diagram showing an example of the configuration of the information processing device 100 according to an embodiment. As shown in Figure 2, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130.

(Regarding Communications Unit 110)
The communication unit 110 is implemented, for example, by a NIC (Network Interface Card). The communication unit 110 is connected to the network N by wire or wireless connection and performs information transmission and reception, for example, with the in-vehicle device 10.

(Regarding memory unit 120)
The storage unit 120 is implemented by, for example, a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disc. The storage unit 120 includes dynamic image data 121, a sensor information database 122, and a map information database 123.

The video data 121 is video data captured by the camera of the in-vehicle device 10. The sensor information database 122 is a database that stores sensor information detected by sensors in the in-vehicle device 10 or sensors in the vehicle VEx itself. The map information database 123 is a database that stores various types of map-related information.

(Regarding the control unit 130)
The control unit 130 is implemented by a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc., which executes various programs (for example, a vehicle behavior determination program according to the embodiment) stored in the memory device inside the information processing device 100 using RAM as the working area. The control unit 130 is also implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

The control unit 130 comprises an acquisition unit 131, a comparison unit 132, a determination unit 133, a storage unit 134, and an output unit 135, and realizes or executes the information processing functions and operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 2; other configurations are also acceptable as long as they perform the information processing described later.

(Regarding acquisition unit 131)
The acquisition unit 131 acquires various types of information from the in-vehicle device 10. For example, the acquisition unit 131 acquires video data of the outside taken from inside the vehicle VEx by the camera of the in-vehicle device 10. The acquisition unit 131 also acquires sensor information detected by sensors of the vehicle VEx (for example, sensors of the in-vehicle device 10 installed in the vehicle VEx, or sensors of the vehicle VEx itself). The acquisition unit 131 stores the acquired video data in video data 121. The acquisition unit 131 also stores the acquired sensor information in the sensor information database 122.

(Regarding comparison section 132)
The comparison unit 132 performs various comparisons. For example, the comparison unit 132 compares the images of frames of the video acquired by the acquisition unit 131 between frames, for each predetermined region of the image.

A specific example of comparing images between frames will be explained. Figure 3 is a diagram illustrating the comparison of images between frames according to the embodiment. Figure 3 shows an example of an image 150 from a frame of a moving image. The comparison unit 132 divides the image 150 from the frame of the moving image into a predetermined number of regions. In Figure 3, the central portion of the image 150, excluding the periphery, is divided vertically and horizontally in a predetermined matrix, thereby dividing the image 150 into multiple rectangular regions 160.

The comparison unit 132 determines the correlation or similarity between frames for each region 160 of the image 150 of the video frame, for example, using template matching, which will be described below as template matching, but is not limited to this. For example, the comparison unit 132 performs template matching on each region 160 of the image 150 of the video frame with the image of the next frame. Figure 4 is a diagram illustrating template matching according to an embodiment. In Figure 4, the central region 160 to be compared in the images is shown as the region of interest 160a. The comparison unit 132 first performs template matching on each region 160 of the image of the video frame with the region 160 at the same position in the image of the next frame. For example, the comparison unit 132 compares the region 160 designated as the region of interest 160a in Figure 4 of the image of the frame with the region 160 at the same position in the image of the next frame and determines whether they match or not. As a result of the comparison, the comparison unit 132 terminates template matching for the matching regions 160. On the other hand, the comparison unit 132, for regions 160 that do not match as a result of the comparison, compares them with the surrounding area centered on the region 160 at the same position in the image of the next frame, and determines whether or not there is a matching area. For example, the comparison unit 132 uses the area 160 designated as the area of interest 160a in Figure 4 of the image of the frame, and sets the range of the surrounding area 160, centered on the region 160 at the same position in the image of the next frame, as the template matching search area. The comparison unit 132 moves the image of the area 160 designated as the area of interest 160a within the range of the search area, centered on the region 160 at the same position in the image of the next frame, and determines whether or not there is a matching area. The comparison unit 132 terminates template matching for regions 160 where there is no matching area. On the other hand, if there is a matching area, the comparison unit 132 identifies the matching area as the moved position of the area 160 designated as the area of interest 160a in the image of the next frame, and terminates template matching.

Furthermore, if the image frame of a moving image includes areas with few features, such as the sky or road surface, accurate matching of these featureless areas 160 is difficult even when performing template matching between frames because the features are sparse. Therefore, the comparison unit 132 may exclude these featureless areas 160 from the comparison. For example, the comparison unit 132 may calculate the standard deviation of the luminance values of the pixels in each area 160 of the frame's image and exclude areas 160 whose standard deviation is smaller than a predetermined value from the comparison. The predetermined value is set as the upper limit of the standard deviation considered to be featureless. This allows for accurate matching of the same image area 160 between frames.

(Regarding the determination unit 133)
The determination unit 133 performs various determinations. For example, the determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines whether the vehicle VEx is stopped based on the number of matching regions. For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions satisfies a predetermined condition. The determination unit 133 also determines the direction of movement of the vehicle VEx based on the result of the movement of each region of the image in the frame of the moving image.

Here, we will explain a specific example of determining the behavior of the vehicle VEx. First, we will explain a specific example of determining when the vehicle VEx is stopped, using Figures 5 and 6.

Figure 5 illustrates the determination of the vehicle's behavior according to this embodiment. Figure 5 shows an image 151 taken from inside a parked vehicle VEx, looking forward to the front of the vehicle VEx. Image 151 shows buildings 151a and 151b in front of the parked vehicle VEx.

The comparison unit 132 divides the image 151 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values of the pixels in each region 160 of the image 151. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. Figure 5 shows the regions 161 that match as a result of template matching. Figure 5 also shows regions 162 that were excluded from the comparison or where no matching parts were found. In Figure 5, regions such as the road surface 160 are excluded from the comparison because they lack distinctive features.

Figure 6 illustrates the determination of the vehicle's behavior according to this embodiment. Figure 6 shows an image 152 taken from inside the vehicle VEx while it is traveling on a road, showing the area in front of the vehicle VEx. Image 152 shows the road 152a and surrounding buildings 152b.

The comparison unit 132 divides the image 152 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the luminance values of the pixels in each region 160 of the image 152. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. This allows for the exclusion of regions with few features that are unsuitable for template matching. In addition to the standard deviation of luminance values, exclusion may also be performed if feature points are not extracted or edges are not extracted. Figure 6 shows the matching regions 161 as a result of template matching. In Figure 6, since the vehicle VEx is moving, regions 160 near the far end of the road 152a are shown as matching regions 161. Also in Figure 6, the movement position in the next frame when there is a part that matches region 160 as a result of template matching is shown as a matching region 163. Also in Figure 6, the direction of movement to the matching region 163 is indicated by an arrow 164. For example, in Figure 6, a matching region 163 is identified in region 160 near building 152b, and the direction of movement is indicated by an arrow 164. Figure 6 also shows regions 162 that were excluded from the comparison or where no matching parts were found. In Figure 6, since the vehicle VEx is moving, many regions 160 become regions 162.

The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines whether the vehicle VEx is stopped based on the number of matching regions 161. For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions 161 is greater than or equal to a predetermined threshold, or greater than or equal to a predetermined percentage of the number of regions 160 that were compared. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered stopped. For example, the predetermined percentage is set to either 30% or 60%. As a result, the determination unit 133 can determine that the vehicle VEx is stopped from the image 151 shown in Figure 5, and that the vehicle VEx is not stopped from the image 152 shown in Figure 6. Also, in Figure 5, the number of matching regions 161 is 78. On the other hand, in Figure 6, the number of matching regions 161 is 4. For example, by setting the threshold to a value between 30 and 78 (or a percentage between 30 and 60%), the determination unit 133 can determine from image 151 shown in Figure 5 that the vehicle VEx is stopped, and from image 152 shown in Figure 6 that the vehicle VEx is not stopped.

Next, we will explain a specific example of determining the direction of movement of the vehicle VEx using Figures 7 to 11. Figures 7 to 9 illustrate the movement position of the image region 160 in the frame due to the behavior of the vehicle VEx according to the embodiment. Figures 7 to 9 schematically show the movement position of the image region 160 in N frames over N+1 frames due to the behavior of the vehicle VEx. In Figures 7 to 9, the vertical column of region 160 located in the center of the image is designated as column C. Also in Figures 7 to 9, the vertical column of region 160 one column to the right of column C is designated as column R1, and the vertical column of region 160 two columns to the right is designated as column R2. Also in Figures 7 to 9, the vertical column of region 160 one column to the left of column C is designated as column L1, and the vertical column of region 160 two columns to the left is designated as column L2.

Figure 7 shows the movement of region 160 when the vehicle VEx is moving in a straight line. When the vehicle VEx is moving in a straight line, region 160 moves radially from the center of the image. For example, in an N-frame image, in N+1 frames, with column C as the center, the regions 160 of columns R1 and R2 move to the right, and the regions 160 of columns L1 and L2 move to the left. Furthermore, the regions 160 of columns C, R1, R2, L1, and L2 also move vertically. Additionally, the amount of movement of region 160 in an N-frame image increases as it moves further away from the central column C. For example, in N+1 frames, the region 160 of column L2 moves more than the region 160 of column L1.

Figure 8 shows the movement of region 160 when vehicle VEx is turning left. When vehicle VEx is turning left, region 160 moves vertically and expands to the right of the image. For example, in an N-frame image, in N+1 frames, regions 160 in columns L2, L1, C1, R1, and R2 move to the right. Furthermore, the further region 160 is from the vertical center of the image, the greater its vertical movement. Also, the further to the right the region 160 is, the greater its movement. For example, in N+1 frames, region 160 in column R2 moves further to the right than region 160 in column R1.

Figure 9 shows the movement of region 160 when vehicle VEx is turning right. When vehicle VEx is turning right, region 160 moves while expanding vertically to the left of the image. For example, in an N-frame image, in N+1 frames, the regions 160 of columns L2, L1, C1, R1, and R2 move to the left. Furthermore, the further region 160 is from the vertical center of the image, the greater its vertical movement. Also, when vehicle VEx is turning right, the amount of movement is greater for the region 160 on the left side of the example. For example, in N+1 frames, the region 160 of column L2 moves significantly to the left than the region 160 of column L1.

Therefore, the direction of movement of the vehicle VEx can be determined based on the movement of each region 160 of the image within the video frame.

Figure 10 illustrates the determination of the behavior of the vehicle VEx according to this embodiment. Figure 10 shows an image 153 taken from inside the vehicle VEx while it is turning left, showing the view in front of the vehicle VEx. Image 153 includes a pedestrian crossing 153a at an intersection, street trees 153b, and a vehicle 153c.

The comparison unit 132 divides the image 153 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the brightness values of the pixels in each region 160 of the image 153. The comparison unit 132 excludes regions 160 whose standard deviation is smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 whose standard deviation is greater than or equal to the predetermined value. For example, the comparison unit 132 compares region 160 with region 160 at the same position in the image of the next frame and determines whether they match. The comparison unit 132 calculates a correlation value indicating the similarity of the regions 160, and determines that they match if the correlation value is greater than a threshold. The comparison unit 132 terminates template matching for regions 160 that match as a result of the comparison. On the other hand, for regions 160 that do not match as a result of the comparison, the comparison unit 132 compares the surrounding area with the region 160 at the same position in the image of the next frame as the center and determines whether there is a matching part. The comparison unit 132 calculates a correlation value indicating the similarity of region 160 to the surrounding area, and determines that the area with a correlation value greater than a threshold is a matching area. If a matching area is found, the comparison unit 132 identifies the matching area as the movement position of region 160 in the next frame's image. The threshold used for determining the match may be changed. For example, when comparing region 160 to the same position in the next frame's image, a higher threshold can be used to determine the match with higher accuracy. On the other hand, when comparing region 160 to the surrounding area of the next frame, a lower threshold can be used to make it easier to identify the movement position. Figure 10 shows the matching area 161 as a result of template matching. In Figure 10, the vehicle VEx is turning left, and the objects in the image are generally moving to the right, so there are few matching areas 161, and the lower right area 160 of image 153 is shown as the matching area 161. Also in Figure 10, the movement position in the next frame when there is a matching area with region 160 as a result of template matching is shown as the matching area 163. For example, in Figure 10, a matching area 163 is identified in area 160 near the pedestrian crossing 153a and vehicle 153c. Figure 10 also shows areas 162 that were excluded from the comparison or where no matching portions were found.

Figure 11 illustrates the determination of the behavior of the vehicle VEx according to the embodiment. Figure 11 shows an image 154 taken from inside the vehicle VEx while it is turning right, showing the view in front of the vehicle VEx. Image 154 shows a building 154a at the corner of the intersection, the vehicle 154b, the road 154c ahead of the right turn, and the railway tracks 154d running alongside the road 154c.

The comparison unit 132 divides the image 154 into multiple rectangular regions 160. The comparison unit 132 calculates the standard deviation of the brightness values of the pixels in each region 160 of the image 154. The comparison unit 132 excludes regions 160 with a standard deviation smaller than a predetermined value from the comparison, and performs template matching with the image of the next frame for regions 160 with a standard deviation greater than or equal to the predetermined value. Figure 11 shows the regions 161 that match as a result of template matching. In Figure 11, the vehicle VEx is turning right, and the objects in the image are generally moving to the left, so there are few matching regions 161, and the lower right region 160 of the image 154 is shown as a matching region 161. Also in Figure 11, the movement position in the next frame when there is a part that matches a region 160 as a result of template matching is shown as a matching region 163. For example, in Figure 11, a matching region 163 is identified in the region 160 near the building 154a. Furthermore, Figure 11 shows the regions 162 that were excluded from the comparison or for which no matching parts were found.

The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. For example, the determination unit 133 determines the direction of movement of the vehicle VEx based on the movement results of each region 160. For example, the determination unit 133 determines that the vehicle VEx is turning left if the number of regions moving to the right in the entire image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. Also, the determination unit 133 determines that the vehicle VEx is turning right if the number of regions moving to the left in the entire image of the video frame is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered to be turning right or left. For example, in Figure 10, the number of regions 160 moving to the right in the entire image of the video frame is 22. On the other hand, in Figure 11, the number of regions 160 moving to the left in the entire image of the video frame is 24. For example, by setting the threshold value to approximately 10 to 20, the determination unit 133 can determine from image 153 in Figure 10 that vehicle VEx is turning left, and from image 154 in Figure 10 that vehicle VEx is not turning right.

Here, as shown in Figures 7 to 9, the region 160 located in the left half of the image frame of the moving image moves to the left when moving straight or turning right, and to the right when turning left. Similarly, the region 160 located in the right half of the image frame of the moving image moves to the right when moving straight or turning left, and to the left when turning right.

Therefore, the determination unit 133 may determine a right turn based on the movement of the region 160 located in the left half of the image frame to the right, and determine a left turn based on the movement of the region 160 located in the right half of the image frame to the left. For example, the determination unit 133 determines that the vehicle VEx is turning left if the number of regions 160 moving to the right in the left half of the image frame is greater than or equal to a predetermined threshold or a predetermined percentage. Also, the determination unit 133 determines that the vehicle VEx is turning right if the number of regions 160 moving to the left in the right half of the image frame is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or percentage is set to a value that allows the vehicle VEx to be considered to be turning right or left. Alternatively, the determination of a right or left turn may be made by focusing only on the amount of movement, and if the amount of movement is greater than when moving straight.

Furthermore, the determination unit 133 may determine the behavior of the vehicle VEx by calculating the average movement for each column of region 160. For example, the determination unit 133 calculates the average amount of movement and direction of movement for each column of region 160. Based on the average amount of movement and direction of movement for each column of region 160, the determination unit 133 determines the behavior of the vehicle VEx.

For example, in Figure 10, below image 153, the average displacement and direction of movement are indicated by bars 170 for each column of region 160.

Furthermore, in Figure 11, below image 154, the average displacement and direction of movement are indicated by bars 170 for each vertical column of region 160.

The determination unit 133 determines that the vehicle VEx is turning left if, in the entire frame of the moving image, the number of columns where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. Furthermore, the determination unit 133 determines that the vehicle VEx is turning right if, in the entire frame of the moving image, the number of columns where the direction of movement is to the left and the average amount of movement is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or percentage is set to a value that allows the vehicle VEx to be considered to be turning left or left. For example, by setting the threshold to a value of approximately 10 to 30, the determination unit 133 can determine from image 153 shown in Figure 10 that the vehicle VEx is turning left, and from image 154 shown in Figure 10 that the vehicle VEx is not turning right.

Furthermore, the determination unit 133 may determine the behavior of the vehicle VEx by calculating the average of the movement amounts for each column of region 160 in the left and right halves of the image frame of the moving image.

For example, in Figure 10, the number of columns determined to be left turns and the number of columns determined to be right turns are shown as "L: 8, R: 0" below Image 153. The number of columns determined to be left turns is the count of columns in the right half of Image 153 where the direction of movement was to the right and the average movement amount was above a certain value. The number of columns determined to be right turns is the count of columns in the left half of Image 153 where the direction of movement was to the right and the average movement amount was above a certain value.

Furthermore, Figure 11 shows the number of columns determined to be left turns and the number of columns determined to be right turns, labeled "L: 0, R: 8," below Image 154. The number of columns determined to be left turns is the count of columns in the right half of Image 154 where the direction of movement was to the right and the average movement amount was above a certain value. The number of columns determined to be right turns is the count of columns in the left half of Image 154 where the direction of movement was to the left and the average movement amount was above a certain value.

The determination unit 133 determines that the vehicle VEx is turning left if, in the left half of the image of the frame of the moving image, the number of columns where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The determination unit 133 also determines that the vehicle VEx is turning right if, in the right half of the image of the frame of the moving image, the number of columns where the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. The predetermined threshold or predetermined percentage is set to a value that allows the vehicle VEx to be considered to be turning left or right. For example, by setting the threshold to a value of about 5, the determination unit 133 can determine that the vehicle VEx is turning left from image 153 shown in Figure 10, and that the vehicle VEx is not turning right from image 154 shown in Figure 11. Alternatively, the determination unit 133 may determine whether a turn is left or right based on the difference between the number of columns determined to be turning left and the number of columns determined to be turning right. For example, the determination unit 133 may determine that vehicle VEx is turning left if the difference between the number of columns determined to be turning left and the number of columns determined to be turning right is greater than or equal to a predetermined threshold or a predetermined percentage. Alternatively, the determination unit 133 may determine that vehicle VEx is turning right if the difference between the number of columns determined to be turning right and the number of columns determined to be turning left is greater than or equal to a predetermined threshold or a predetermined percentage. For example, suppose the number of columns determined to be turning left is 2, the number of columns determined to be turning right is 8, and the threshold is 5. In this case, the difference between the number of columns determined to be turning right and the number of columns determined to be turning left is 6. Since the difference of 6 is greater than or equal to the threshold of 5, the determination unit 133 determines that it is turning right.

Furthermore, when a vehicle VEx and another vehicle are traveling side-by-side at the same speed on a multi-lane road, the video image may show the other vehicles traveling alongside at similar positions, resulting in a large number of matching regions 161. Therefore, the determination unit 133 may determine that the vehicle VEx has stopped if a predetermined number of consecutive frames satisfy a predetermined condition regarding the number of matching regions 161. For example, the determination unit 133 determines that the vehicle VEx has stopped if, for example, five consecutive frames have a number of matching regions 161 equal to or greater than a predetermined threshold, or equal to or greater than a predetermined ratio of the number of regions 160 used for comparison. This suppresses the misjudgment that the vehicle VEx has stopped when it is temporarily traveling side-by-side at the same speed as another vehicle, and allows for accurate determination of the vehicle VEx's stopping. The determination unit 133 may also determine that the vehicle VEx has made a right or left turn if a predetermined number of consecutive frames indicate a right or left turn. The predetermined number may be variable; setting it to a small value allows for quick determination of the vehicle's VEx behavior.

(Regarding the storage unit 134)
The storage unit 134 stores the determination result made by the determination unit 133. For example, the storage unit 134 stores the behavior of the vehicle VEx determined by the determination unit 133 in the video data 121 or data associated with the video data 121, associating it with the frame and time of capture of the video in which the behavior was determined. Alternatively, the storage unit 134 may store the determination result made by the determination unit 133 in the map information database 123, associating it with the location on the map where the behavior was detected.

(Regarding the output unit 135)
The output unit 135 performs various outputs. For example, the output unit 135 transmits various information to the in-vehicle device 10 based on the determination result by the determination unit 133. For example, if the determination unit 133 does not determine that a stop is required at a location stored in the map information database 123, the output unit 135 transmits a warning to the in-vehicle device 10.

As described above, the information processing device 100 according to this embodiment can determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards. For example, the information processing device 100 can determine the vehicle VEx's stopping, right turns, and left turns from the video footage. For example, the information processing device 100 can determine from the video footage behaviors of the vehicle VEx, such as momentary stops that cannot be determined by GPS location information alone. Therefore, the information processing device 100 can also be used in the following ways. For example, by determining the behavior of the vehicle VEx from video footage captured by a recording device such as a drive recorder using the information processing device 100, accident verification can be performed. For example, it can be verified from the video footage whether or not the vehicle VEx made a stop. Furthermore, by determining the behavior of the vehicle VEx from video footage using the information processing device 100, it is possible to evaluate whether the driver is driving safely, and this can be used, for example, by insurance companies to evaluate driver performance in telemastic insurance.

[3. Processing Procedure]
Next, the information processing procedure performed by the information processing device 100 will be explained using Figure 12. Figure 12 is a flowchart showing the procedure for the vehicle behavior determination process, which determines the behavior of the vehicle.

According to the example in Figure 12, the acquisition unit 131 acquires various types of information from the in-vehicle device 10 (step S10). For example, the acquisition unit 131 acquires video data of the outside of the vehicle VEx, captured by the camera of the in-vehicle device 10.

The comparison unit 132 compares the images of the video frames acquired by the acquisition unit 131 across frames, for each predetermined region of the image (step S11). For example, the comparison unit 132 divides the video frame images into multiple regions 160. The comparison unit 132 performs template matching across frames for each region 160 of the video frame images.

The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132 (step S12). For example, the determination unit 133 determines that the vehicle VEx is stopped if the number of matching regions 161 satisfies a predetermined condition. The determination unit 133 also determines the direction of movement of the vehicle VEx based on the movement of each region 160 of the image in the frame of the moving image.

The storage unit 134 stores the determination result from the determination unit 133 (step S13). For example, the storage unit 134 stores the behavior of the vehicle VEx determined by the determination unit 133 in the video image data 121 or data associated with the video image data 121, associating it with the frame and shooting time of the video image in which the behavior was determined.

The output unit 135 transmits various information to the in-vehicle device 10 based on the determination result from the determination unit 133 (step S14). For example, if the determination unit 133 does not determine that a stop is required at a location stored in the map information database 123, the output unit 135 transmits a warning to the in-vehicle device 10.

Thus, the information processing device 100 according to this embodiment can determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards.

In this embodiment, the vehicle behavior determination process described above was performed in the information processing device 100 to determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards. However, the embodiment is not limited to this. The in-vehicle device 10 may also perform the vehicle behavior determination process described above to determine the behavior of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards.

Furthermore, in this embodiment, the case where the shape of region 160 is rectangular was described as an example. However, it is not limited to this. The shape of region 160 may be other than rectangular. For example, the comparison unit 132 may divide the image of the video frame into equilateral triangles or regular hexagons and make region 160 into the shape of equilateral triangles or regular hexagons. Alternatively, for example, the comparison unit 132 may divide the image of the video frame into multiple types of polygons and make region 160 into multiple types of polygons. Furthermore, the comparison unit 132 may recognize the outlines of objects in the image of the video frame and divide the image of the frame into regions 160 based on the outlines of the objects.

Furthermore, in this embodiment, the case of determining right and left turns as the direction of movement of the vehicle VEx was described as an example. However, it is not limited to this. By identifying the matching region 163 through template matching by the comparison unit 132, vertical movement can also be determined. The determination unit 133 may also determine the behavior of the vehicle VEx in the vertical direction. This allows, for example, to determine whether the vehicle VEx is going uphill or downhill.

Furthermore, in this embodiment, the video was described using an example where a camera is positioned facing the front of the vehicle VEx, and the video is captured from inside the vehicle VEx, showing the front of the vehicle VEx. However, it is not limited to this. The video can be any video captured from inside the vehicle VEx, showing the outside. For example, the video could be a video where a camera is positioned facing the rear of the vehicle VEx, and the video is captured from inside the vehicle VEx, showing the rear of the vehicle VEx. If the video is a video captured from inside the vehicle VEx, the vehicle VEx's behavior, such as stopping, can be determined. Also, if the video is a video captured from inside the vehicle VEx, showing the front, rear, or side of the vehicle VEx, the vehicle VEx's movement direction, such as turning right or left, can be accurately determined.

Furthermore, in this embodiment, as shown in Figure 4, the search area for template matching was described using the region 160a as the focus area, with the surrounding region 160 as the center. However, it is not limited to this. The comparison unit 132 may change the search area range according to the vehicle VEx speed. For example, the comparison unit 132 may widen the search area range horizontally as the vehicle VEx speed increases.

Furthermore, when filming the exterior from inside the vehicle VEX, a portion of the vehicle VEX may be captured in the video. For example, when filming the front of the vehicle VEX from inside the vehicle VEX, the hood of the vehicle VEX may be captured in the video. When a portion of the vehicle VEX is captured in the video, the region 160 of the vehicle VEX portion captured in the frame image will be the same image, and as a result of template matching, it will always be determined to be a matching region 161. Therefore, the information processing device 100 may exclude the always matching region 161 and determine the behavior of the vehicle VEX. For example, the determination unit 133 may exclude the region 160 that is always region 161 in multiple different behaviors and determine the behavior of the vehicle VEX. Also, if the portion of the vehicle VEX captured in the frame image is large, the region 161 capturing the area around the vehicle VEX will be smaller, and the accuracy of determining the behavior of the vehicle VEX may decrease. Therefore, the information processing device 100 may output a warning to the in-vehicle device 10 if the vehicle VEx portion in the frame image is large. For example, the output unit 135 may output a warning to the in-vehicle device 10 prompting adjustment of the camera's field of view if the number of consistently matching regions 161 exceeds an acceptable value. Alternatively, the target rectangular region may be disabled.

[4. Summary]
The information processing device 100 according to this embodiment includes an acquisition unit 131, a comparison unit 132, and a determination unit 133. The acquisition unit 131 acquires moving images of the outside taken from inside the vehicle VEx. The comparison unit 132 compares the images of the frames of the moving images acquired by the acquisition unit 131 between frames, for each predetermined region 160 of the image. The determination unit 133 determines the behavior of the vehicle VEx based on the comparison result of the comparison unit 132. As a result, the information processing device 100 can determine the behavior of the vehicle VEx from moving images taken from inside the vehicle VEx.

Furthermore, the comparison unit 132 performs template matching on each region 160 of the image in the video frame with the region 160 at the same position in the image of the next frame. The determination unit 133 determines that the vehicle VEx has stopped based on the number of matching regions 161. The determination unit 133 also determines that the vehicle VEx has stopped if the number of matching regions 161 satisfies a predetermined condition. This allows the information processing device 100 to determine whether the vehicle VEx has stopped from video footage captured from inside the vehicle VEx looking outwards.

Furthermore, the determination unit 133 determines that the vehicle VEx has stopped if a predetermined number of consecutive frames satisfy a predetermined condition regarding the number of matching regions 161. This allows the information processing device 100 to accurately determine whether the vehicle VEx has stopped.

Furthermore, the comparison unit 132 calculates the standard deviation of the luminance values of the pixels in each region 160 of the frame's image, and excludes regions 160 whose standard deviation is smaller than a predetermined value from the comparison. This allows the information processing device 100 to accurately match the same image regions 160 across frames.

Furthermore, the comparison unit 132 performs template matching on the image of each frame of the video, region 160 at a time, with the image of the next frame to determine the movement position of each region 160 within the image of the next frame. The determination unit 133 determines the direction of movement of the vehicle VEx based on the movement results of each region 160. This allows the information processing device 100 to determine the direction of movement of the vehicle VEx from video footage captured from inside the vehicle VEx looking outwards.

The determination unit 133 determines a right turn based on the movement of region 160 located in the left half of the image frame to the right, and determines a left turn based on the movement of region 160 located in the right half of the image to the left. This allows the information processing device 100 to accurately determine whether the vehicle VEx is turning right or left from a video image captured from inside the vehicle VEx looking outwards.

[5. Hardware Configuration]
Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having the configuration shown in Figure 13. Figure 13 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device 100. The computer 1000 has a CPU 1100, RAM 1200, ROM 1300, HDD 1400, communication interface (I/F) 1500, input/output interface (I/F) 1600, and media interface (I/F) 1700.

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, controlling various components. The ROM 1300 stores boot programs executed by the CPU 1100 when the computer 1000 starts up, as well as programs dependent on the computer 1000's hardware.

The HDD 1400 stores programs executed by the CPU 1100, as well as data used by such programs. The communication interface 1500 receives data from other devices via a predetermined communication network and sends it to the CPU 1100, and transmits data generated by the CPU 1100 to other devices via the predetermined communication network.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice, via the input/output interface 1600. The CPU 1100 acquires data from input devices via the input/output interface 1600. Furthermore, the CPU 1100 outputs the generated data to output devices via the input/output interface 1600.

The media interface 1700 reads programs or data stored in the recording medium 1800 and provides them to the CPU 1100 via the RAM 1200. The CPU 1100 loads such programs from the recording medium 1800 onto the RAM 1200 via the media interface 1700 and executes the loaded programs. The recording medium 1800 can be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, or semiconductor memory.

For example, when the computer 1000 functions as an information processing device 100 according to the embodiment, the CPU 1100 of the computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, these programs may be obtained from other devices via a predetermined communication network.

[6. Other]
Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be changed at will unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those illustrated; all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

Furthermore, the above embodiments can be combined as appropriate, provided that the processing content is not contradictory.

Although several embodiments of this application have been described in detail based on the drawings, these are illustrative examples, and the present invention can be implemented in various modified and improved forms based on the knowledge of those skilled in the art, starting with the embodiments described in the disclosure section of the invention.

1 Information Processing System 10 In-vehicle Device 100 Information Processing Device 120 Storage Unit 121 Moving Image Data 122 Sensor Information Database 123 Map Information Database 130 Control Unit 131 Acquisition Unit 132 Comparison Unit 133 Judgment Unit 134 Storage Unit 135 Output Unit 160, 161, 162 Area 163 Matching Area VEx Vehicle

Claims (4)

An acquisition unit that acquires video footage of the outside taken from inside the vehicle,
A determination unit that, for each predetermined region of the image of the frame of the moving image acquired by the acquisition unit , determines the correspondence between the image and the image of the next frame, identifies the movement position in the next frame for each region, determines a left turn based on the movement to the right of the region located on the left side of the image of the frame of the moving image, and determines a right turn based on the movement to the left of the region located on the right side of the image ,
An information processing device characterized by comprising: The determination unit determines the average amount of movement and direction of movement for each column in the region, and determines that the vehicle is turning left if the number of columns in the left half of the image of the frame of the moving image where the direction of movement is to the right and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage, and determines that the vehicle is turning right if the number of columns in the right half of the image of the frame of the moving image where the direction of movement is to the left and the average amount of movement is greater than or equal to a certain value is greater than or equal to a predetermined threshold or a predetermined percentage. A method for determining vehicle behavior performed by an information processing device,
The acquisition process involves obtaining video footage of the exterior from inside the vehicle,
The acquired video frame image is used to determine the correspondence between the image of the next frame and a predetermined region of the image, the movement position in the next frame for each region is identified, a left turn is determined based on the movement to the right of the region located on the left side of the video frame, and a right turn is determined based on the movement to the left of the region located on the right side of the image .
A method for determining vehicle behavior, characterized by including the following: The procedure for acquiring video footage of the exterior from inside the vehicle,
A determination procedure that involves determining the correspondence between the image of the acquired video frame, for each predetermined region of the image, and the image of the next frame, identifying the movement position in the next frame for each region, determining a left turn based on the movement to the right of the region located on the left side of the video frame, and determining a right turn based on the movement to the left of the region located on the right side of the image ,
A vehicle behavior determination program to be executed by an information processing device.