JP7698978B2 - Image processing device and vehicle - Google Patents

Description

This disclosure relates to an image processing device that performs object tracking based on captured images, and a vehicle equipped with such an image processing device.

Images captured by an imaging device include images of various objects. For example, Patent Literature 1 discloses an image processing device that tracks objects based on such captured images.

JP 2018-132897 A

However, in such image processing devices, there is a demand for suppressing deterioration in object tracking accuracy caused by external factors and shortening processing time. It is desirable to provide an image processing device that can shorten processing time while suppressing deterioration in object tracking accuracy caused by external factors, and a vehicle equipped with such an image processing device.

An image processing device according to an embodiment of the present disclosure includes: an area setting unit that sets one or more image areas in a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image; a first extraction unit that extracts a first feature amount that indicates a sparse feature included in the image area of the captured image; a second extraction unit that extracts a second feature amount that indicates a clearer dense feature compared to the sparse feature included in the image area in the left image or the right image as a priority image obtained from the left camera or the right camera that is set as a priority camera; a first object tracking unit that determines whether object tracking using the first feature amount is feasible in each of the left image and the right image; a second object tracking unit that performs object tracking using the second feature amount in the priority image; and a switching unit that, when it is determined by the first object tracking unit that object tracking using the first feature amount in the priority image is not feasible, switches one of the stereo cameras that was not set as the priority camera to the priority camera and switches the captured image obtained from the camera that was not set as the priority camera to the priority image .

A vehicle according to an embodiment of the present disclosure includes an image processing device according to the embodiment of the present disclosure, and a vehicle control unit that controls the vehicle using the results of object tracking using the second feature amount obtained from the second object tracking unit.

1 is a block diagram illustrating an example of a schematic configuration of a vehicle according to an embodiment of the present disclosure. FIG. 2 is a top view illustrating an example of the external configuration of the vehicle illustrated in FIG. 1 . 2 is a schematic diagram illustrating an example of a left image and a right image generated by the stereo camera illustrated in FIG. 1 . 2 is a schematic diagram illustrating an example of an image region set by a region setting unit illustrated in FIG. 1 . FIG. 2 is a schematic diagram illustrating an example of sparse features illustrated in FIG. 1 . FIG. 2 is a schematic diagram illustrating an example of dense features illustrated in FIG. 1 . 11A and 11B are schematic diagrams illustrating an example of a priority camera and priority image switching process according to an embodiment. 10 is a flowchart illustrating an example of a priority camera and priority image switching process according to an embodiment. 7B is a flowchart showing an example of a switching process subsequent to FIG. 7A.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. The description will be made in the following order.
1. Embodiment (Example of setting image area using distance information or machine learning)
2. Modifications

1. Preferred embodiment
[composition]
Fig. 1 is a block diagram showing an example of a schematic configuration of a vehicle (vehicle 10) according to an embodiment of the present disclosure. Fig. 2 is a schematic top view showing an example of the exterior configuration of the vehicle 10 shown in Fig. 1.

As shown in FIG. 1, the vehicle 10 is equipped with a stereo camera 11, an image processing device 12, and a vehicle control unit 13. Note that in FIG. 1, the driving power source (engine, motor, etc.) of the vehicle 10 is omitted. The vehicle 10 is, for example, an electric vehicle such as a hybrid electric vehicle (HEV) or an electric vehicle (EV), or a gasoline vehicle.

(A. Stereo camera 11)
The stereo camera 11 is a camera that captures an image of the area ahead of the vehicle 10 to generate a pair of images (a left image PL and a right image PR) having a parallax therebetween, as shown in Fig. 2 for example. The stereo camera 11 includes a left camera 11L and a right camera 11R, as shown in Figs. 1 and 2.

The left camera 11L and the right camera 11R each include, for example, a lens and an image sensor. The left camera 11L and the right camera 11R are arranged near the top of the windshield 19 of the vehicle 10, for example, as shown in FIG. 2, at a predetermined distance apart along the width direction of the vehicle 10. The left camera 11L and the right camera 11R are configured to perform imaging operations in synchronization with each other. Specifically, as shown in FIG. 1, the left camera 11L generates a left image PL, and the right camera 11R generates a right image PR. The left image PL includes a plurality of pixel values, and the right image PR includes a plurality of pixel values. The left image PL and the right image PR constitute a stereo image PIC, as shown in FIG. 1.

Figure 3 shows an example of such a stereo image PIC. Specifically, Figure 3(A) shows an example of a left image PL, and Figure 3(B) shows an example of a right image PR. Note that x and y shown in Figure 3 represent the x-axis and y-axis, respectively. In this example, another vehicle (preceding vehicle 90) is traveling ahead of vehicle 10 on the road on which vehicle 10 is traveling. The left camera 11L generates the left image PL by capturing an image of the preceding vehicle 90, and the right camera 11R generates the right image PR by capturing an image of the preceding vehicle 90.

The stereo camera 11 is configured to generate a stereo image PIC that includes the left image PL and the right image PR. The stereo camera 11 also generates a series of stereo images PIC by performing an imaging operation at a predetermined frame rate (e.g., 60 fps).

(B. Image Processing Device 12)
The image processing device 12 is a device that performs various image processing (such as tracking processing of an object in front of the vehicle 10) based on the stereo image PIC supplied from the stereo camera 11. As shown in FIG. 1 , the image processing device 12 has an image memory 121, a distance information generating unit 122, an area setting unit 123, feature amount extracting units 124A and 124B, object tracking units 125A and 125B, a switching unit 126, and a movement estimating unit 127.

Such an image processing device 12 is configured to include, for example, one or more processors (CPU: Central Processing Unit) that execute programs, and one or more memories communicatively connected to these processors. In addition, such memory is configured, for example, with a RAM (Random Access Memory) that temporarily stores processing data, and a ROM (Read Only Memory) that stores programs, etc.

The feature extraction unit 124A corresponds to a specific example of a "first extraction unit" in this disclosure, and the feature extraction unit 124B corresponds to a specific example of a "second extraction unit" in this disclosure. Also, the object tracking unit 125A corresponds to a specific example of a "first object tracking unit" in this disclosure, and the object tracking unit 125B corresponds to a specific example of a "second object tracking unit" in this disclosure.

(Image memory 121)
1, the image memory 121 is a memory for temporarily storing the left image PL and the right image PR included in the stereo image PIC. The image memory 121 sequentially supplies at least one of the left image PL and the right image PR thus stored as a captured image P to the distance information generating unit 122 and the feature amount extracting units 124A and 124B (see FIG. 1).

(Distance information generation unit 122)
The distance information generating unit 122 generates distance information Iz by performing predetermined image processing including stereo matching processing and filtering processing based on the captured image P (here, the left image PL and the right image PR) read out from the image memory 121 (see FIG. 1). Specifically, the distance information generating unit 122 generates a distance image including a plurality of pixel values based on the left image PL and the right image PR. In this example, each of the plurality of pixel values is a parallax value. In other words, each of the plurality of pixel values corresponds to a distance to a point corresponding to each pixel in a three-dimensional real space. Note that this example is not limited, and for example, each of the plurality of pixel values may be a distance value indicating a distance to a point corresponding to each pixel in a three-dimensional real space. In this way, the distance information generating unit 122 generates distance information Iz, which is information indicating a distance to a point corresponding to each pixel.

(Area setting unit 123)
The region setting unit 123 sets one or more image regions R in the captured image P based on the distance information Iz supplied from the distance information generating unit 122. Specifically, the region setting unit 123 specifies a plurality of pixels that are located close to each other in the captured image P and have approximately the same parallax value based on the distance information Iz, and sets a rectangular region including the plurality of pixels as the image region R. In other words, if there is an object in the captured image P, the pixels in the region corresponding to the object are located close to each other and have approximately the same parallax value. Therefore, by setting the image region R in this manner, the region setting unit 123 sets the image region R so as to surround the object.

Figure 4 is a schematic diagram showing an example of an image region R set by the region setting unit 123. In the example shown in Figure 4, an image region R is set for each of two vehicles in the captured image P (here, one of the left image PL and the right image PR). Note that in this example, the region setting unit 123 sets the image region R for the vehicles, but is not limited to this example, and may also set the image region R for, for example, a person, a guardrail, a wall, etc.

In this manner, information about one or more image regions R set by the region setting unit 123 is supplied to the feature extraction units 124A and 124B, respectively, as shown in FIG. 1.

In the example shown in FIG. 1, the region setting unit 123 sets the image region R using the distance information Iz, but this is not limited to the example. That is, the region setting unit 123 may use a trained model such as a DNN (Deep Neural Network) to identify an object in the captured image P and output the coordinates of the identified object to set the image region R, which is a rectangular region. That is, the region setting unit 123 may set the image region R using machine learning.

(Feature Extraction Units 124A, 124B)
The feature extraction unit 124A extracts sparse features FA (features indicating sparse features) contained in one or more image regions R in a captured image P (here, one of the left image PL and the right image PR) (see Figure 1).

The feature extraction unit 124B extracts dense features FB (features indicating dense features) contained in one or more image regions R in the captured image P (here, one of the left image PL and the right image PR) (see FIG. 1). Specifically, the feature extraction unit 124B extracts dense features FB contained in the image region R in the left image PL or the right image PR set as the priority image Pp. In addition, the feature extraction unit 124B extracts dense features FB, for example, by reusing sparse features FA extracted by the feature extraction unit (see FIG. 1), as will be described in detail later.

The above-mentioned priority image Pp refers to the captured image P obtained from the camera (left camera 11L or right camera 11R) set as the priority camera 11p in the stereo camera 11. Such priority camera 11p and priority image Pp are each set in advance before use, and the settings can be changed (settings switched) at any time by the switching process in the switching unit 126, which will be described later.

Furthermore, the sparse features in the sparse feature amount FA described above are coarser features than the dense features in the dense feature amount FB. Conversely, the dense features in the dense feature amount FB are sharper features than the sparse features in the sparse feature amount FA. As will be described in detail later, object tracking using the dense feature amount FB enables highly accurate object tracking, but increases the processing time and processing costs. For this reason, in this embodiment, as will be described later, these two types of feature amounts are used appropriately to achieve both object tracking accuracy and processing time (processing costs).

Note that the sparse feature FA corresponds to a specific example of a "first feature" in this disclosure, and the dense feature FB corresponds to a specific example of a "second feature" in this disclosure.

Here, FIG. 5A is a schematic representation of an example of a sparse feature FA, and FIG. 5B is a schematic representation of an example of a dense feature FB.

First, in the sparse feature FA shown in FIG. 5A, four pixels PX are arranged in a matrix (2 vertical × 2 horizontal), and the pixel values of each pixel PX are pixel values PXa, PXb, PXc, and PXd. On the other hand, in the dense feature FB shown in FIG. 5B, 16 pixels PX are arranged in a matrix (4 vertical × 4 horizontal). The difference in the number of pixels PX between the sparse feature FA and the dense feature FB is caused by, for example, the difference in the size (number of parameters) of the filter applied during the extraction process using the DNN trained model described later, and the difference in the size of the stride (application range of the convolution operation). In addition, in this dense feature FB, some (4) of these 16 pixels PX have the above-mentioned pixel values PXa, PXb, PXc, and PXd. That is, in this dense feature FB, as described above, the sparse feature FA (its pixel values) are reused. Note that, in this dense feature FB, new pixel values are acquired for each of the remaining pixels PX (in the example of FIG. 5B, the remaining 12 pixels PX excluding the pixels PX showing pixel values PXa, PXb, PXc, and PXd).

Each of the feature extraction units 124A and 124B is configured to extract sparse features FA and dense features FB using, for example, a trained model of DNN. In this case, for example, each of the neural networks in the feature extraction units 124A and 124B has multiple convolution layers and multiple pooling layers.

The sparse features FA extracted by the feature extraction unit 124A in this manner are supplied to the object tracking unit 125A and the feature extraction unit 124B, respectively (see FIG. 1). The dense features FB extracted by the feature extraction unit 124B are supplied to the object tracking unit 125B (see FIG. 1).

(Object Tracking Units 125A, 125B)
The object tracking unit 125A judges whether or not object tracking using the sparse feature amount FA in each of the left image PL and the right image PR is possible, and supplies the judgment result JA of such a judgment to the switching unit 126 (see FIG. 1). The judgment of whether or not such object tracking is possible is made, for example, by whether or not the maximum value of the correlation value between the image of the object (tracked object) acquired in the immediately preceding frame period and the image of the object acquired in the current frame period is less than a predetermined threshold value. Alternatively, by recording the coordinates of the tracked object on the image, the movement status of the tracked object and the coordinates after the movement may be estimated, and whether or not object tracking is possible may be judged by the degree of deviation between the estimated value of the coordinates after the movement and the updated value of the actual coordinates.

The object tracking unit 125B performs object tracking using dense feature values FB in the priority image Pp described above, and supplies the results of such object tracking (tracking position TB of the tracked object) to the movement estimation unit 127 and the vehicle control unit 13, respectively (see Figure 1).

Each of the object tracking units 125A and 125B performs object tracking by estimating the position of the tracked object for each captured image P in each frame period using the sparse feature amount FA or the dense feature amount FB. Specifically, the position where the above-mentioned correlation value is the largest (indicates the maximum value) is estimated as the position (coordinates) of the tracked object, and is updated as needed. In addition, such object tracking is performed using an object tracking method that utilizes a CNN (Convolutional Neural Network), such as KCF (Kernelized Correlation Filter), template matching, or Siamese Network.

(Switching Unit 126)
The switching unit 126 switches the settings of the priority camera 11p and the priority image Pp when the object tracking unit 125A determines that object tracking using the sparse feature amount FA in the priority image Pp is impossible. Specifically, the switching unit 126 performs such a setting switching process using the determination result JA supplied from the object tracking unit 125A (see FIG. 1). In addition, the switching process of the settings of the priority camera 11p and the priority image Pp is performed by supplying a switching signal Ss from the switching unit 126 to each of the feature amount extraction unit 124B, the object tracking unit 125B, and the movement estimation unit 127 (see FIG. 1). Details of such a setting switching process will be described later (FIGS. 6, 7A, and 7B). In addition, when performing such a setting switching process, the switching unit 126 compensates for the amount of parallax between the left image PL and the right image PR, for example, by using various general methods.

(Movement estimation unit 127)
The movement estimation unit 127 estimates the three-dimensional movement of the tracked object based on the result of object tracking (tracking position TB of the tracked object) supplied from the object tracking unit 125B and the distance information Iz supplied from the distance information generation unit 122 (see FIG. 1). Specifically, the movement estimation unit 127 estimates a speed VB (movement speed) during such three-dimensional movement and supplies it to the vehicle control unit 13 (see FIG. 1). In addition, the movement estimation unit 127 estimates the three-dimensional coordinates of the tracked object based on the tracking position TB and the distance information Iz, and estimates the above-mentioned three-dimensional speed VB by calculating the difference in three-dimensional coordinates between the current frame period and the immediately previous frame period.

In addition, when the switching unit 126 performs the above-mentioned switching process (switching process of the settings of the priority camera 11p and the priority image Pp), the movement estimation unit 127 performs, for example, the following correction process. That is, when the switching unit 126 supplies the switching signal Ss, the movement estimation unit 127 corrects the movement error during the three-dimensional movement of the tracked object caused by the setting switching of the priority image Pp, for example, by using linear approximation or the like. Specifically, the movement estimation unit 127 uses the data point group before the setting switching (the three-dimensional coordinates of the tracked object) and the data point group after the setting switching to approximate the estimation of the movement of the tracked object using a linear function, a quadratic function, a cubic function, or the like. As a result, the movement error during the three-dimensional movement of the tracked object caused by the setting switching of the priority image Pp is corrected. Note that, in the curve fitting during the above-mentioned approximation, a plurality of methods, such as the least squares method, are used.

(C. Vehicle control unit 13)
The vehicle control unit 13 performs various vehicle controls on the vehicle 10 by using the object tracking result (tracking position TB of the tracked object) supplied from the object tracking unit 125B and the speed VB (three-dimensional moving speed of the tracked object) supplied from the movement estimation unit 127 (see FIG. 1). Specifically, the vehicle control unit 13 performs, for example, driving control of the vehicle 10 and operation control of various members in the vehicle 10 based on the information on the tracking position TB and the speed VB.

Similar to the image processing device 12, such a vehicle control unit 13 is configured to include, for example, one or more processors (CPUs) that execute programs, and one or more memories communicatively connected to these processors. Also, like the image processing device 12, such memories are configured, for example, with a RAM that temporarily stores processing data, and a ROM that stores programs, etc.

[Actions, actions and effects]
Next, the operation, functions, and effects of this embodiment will be described in detail.

(A. Regarding deterioration of object tracking accuracy due to external factors)
First, a decrease in object tracking accuracy due to external factors will be described with reference to Fig. 6(A) and Fig. 6(B). Fig. 6(A) shows an example of a stereo image PIC1 (left image PL1 and right image PR1) in a certain frame period. Fig. 6(B) shows an example of a stereo image PIC2 (left image PL2 and right image PR2) in a frame period (see dashed arrow t) following the stereo image PIC1.

Generally, when a vehicle is traveling, as in the right image PR2 shown in FIG. 6(B), there is a risk that an obstacle 8 (an unavoidable external factor when traveling), such as a windshield wiper or water droplets, may prevent the camera from capturing an image. Specifically, in the example of the right image PR2 in FIG. 6(B), an image contained in a portion of the multiple image regions R (image region R shown by the dashed line) may be hidden by this obstacle 8, making it difficult to track an object (tracked object) located within that image region R. As a result, if this continues, the tracking accuracy of the tracked object will decrease.

In addition, since the processing time required for an in-vehicle CPU is generally very short, it is desirable to speed up processing during object tracking (reducing processing time). In particular, in this embodiment, as described above, object tracking using dense feature FB enables highly accurate object tracking, but this increases processing time and processing costs, so reducing such processing time is important.

For these reasons, when tracking objects based on images captured by a camera, it is desirable to shorten processing time (reduce processing costs) while minimizing the degradation of object tracking accuracy caused by external factors such as those described above.

(B. Switching Process, etc., of the Present Embodiment)
Therefore, in the vehicle 10 of this embodiment, the image processing device 12 is configured to perform the various processes described below (such as the process of switching the settings of the priority camera 11p and the priority image Pp described above).

Below, an example of the above-mentioned switching process etc. in this embodiment will be described in detail with reference to Figs. 1 to 6 as well as Figs. 7A and 7B. Figs. 7A and 7B each show an example of the above-mentioned switching process etc. in this embodiment in the form of a flow chart. Note that below, an example of such switching process etc. will be described with reference to the examples of stereo images PIC1 and PIC2 shown in Figs. 6(A) and 6(B) described above.

In the processing example shown in Figures 7A and 7B, first, the stereo camera 11 captures an image of the area in front of the vehicle 10 to generate a stereo image PIC (left image PL and right image PR) (step S100 in Figure 7A). Specifically, for example, stereo image PIC1 (left image PL1 and right image PR1), stereo image PIC2 (left image PL2 and right image PR2), etc., as shown in Figures 6(A) and 6(B) are generated.

Next, the image memory 21 in the image processing device 12 temporarily stores the stereo image PIC (left image PL and right image PR) thus generated as the captured image P (step S101). Next, the area setting unit 123 determines whether or not the image area R in the captured image P has been set (step S102). If it is determined that such an image area R has been set (step S102: Y), the process proceeds to step S104, which will be described later.

On the other hand, if it is determined that such an image region R has not been set (step S102: N), then the region setting unit 123 uses the above-mentioned method (distance information Iz or machine learning, etc.) to set one or more image regions R in the captured image P (step S103). Then, in next step S104, the feature extraction unit 124A extracts the above-mentioned sparse feature amounts FA included in the image region R of the captured image P.

Then, the object tracking unit 125A judges whether or not object tracking using the sparse feature amount FA is possible in the above-mentioned priority image Pp (left image PL or right image PR) (step S105). Here, if it is judged that object tracking using the sparse feature amount FA is possible in the priority image Pp (step S105: Y), the process proceeds to step S108 (FIG. 7B) described later.

On the other hand, if it is determined that object tracking using the sparse feature amount FA is not possible in the priority image Pp (step S105: N), the following occurs. That is, in this case, the object tracking unit 125A then determines whether object tracking using the sparse feature amount FA is possible in the other captured image P (one of the left image PL and the right image PR that is different from the priority image Pp) (step S106 in FIG. 7B). Here, if it is determined that object tracking using the sparse feature amount FA is not possible in such other captured image P (step S106: N), object tracking ends, and the series of processing examples shown in FIG. 7A and FIG. 7B also ends. That is, if the object tracking unit 125A determines that object tracking using the sparse feature amount FA is not possible in both the left image PL and the right image PR, the processing in step S109 described later (object tracking using the dense feature amount FB in the object tracking unit 125B) is not performed.

On the other hand, if it is determined that object tracking using the sparse feature amount FA is possible in the other captured image P (step S106: Y), the following occurs. That is, in this case, the switching unit 126 then performs a process of switching the settings of the priority camera 11p and the priority image Pp described above (step S107). After that, the process proceeds to step S108, which will be described later.

Here, taking the examples of Figures 6(A) and 6(B) mentioned above as an example, the above switching process is specifically performed as follows. That is, in this example, first, in the stereo image PIC1 shown in Figure 6(A), of the left image PL1 and the right image PR1, the right image PR1 is set as the priority image Pp. In other words, of the left camera 11L and the right camera 11R, the right camera 11R is set as the priority camera 11p.

Now, in the stereo image PIC2 obtained in a subsequent frame period, as shown in FIG. 6B, for example, the right image PR2 set as the priority image Pp is as follows due to the external factor (obstacle 8) mentioned above. That is, in the right image PR2 as the priority image Pp, an image included in a part of the multiple image regions R (image region R shown by the dashed line) is hidden by this obstacle 8, and it may become difficult to track an object (tracked object) located within that image region R.

In this case, the switching unit 126 performs a process of switching the settings of the priority camera 11p and the priority image Pp described above. Specifically, in this case, the switching unit 126 switches the setting of the priority camera 11p from the right camera 11R to the left camera 11L, and switches the setting of the priority image Pp from the right image PR2 to the left image PL2 (see the dashed arrow SW in FIG. 6(B)). In the left image PL2 after such a setting switch of the priority image Pp, as shown in FIG. 6(B), for example, unlike the right image PR2, no external factor (obstacle 8) has occurred. This avoids the risk of it becoming difficult to track the tracked object as described above.

Next, in step S108 (FIG. 7B) described above, the feature extraction unit 124B extracts the dense feature FB described above contained in the image region R of the priority image Pp set at that time. Next, the object tracking unit 125B performs object tracking in the priority image Pp set at that time using the dense feature FB extracted in step S108 (step S109).

Next, the movement estimation unit 127 estimates the speed VB of the tracked object during three-dimensional movement based on the tracking position TB and distance information Iz of the tracked object described above (step S110). At this time, if the setting of the priority image Pp is switched (the switching process in step S107) as described above, the movement estimation unit 127 performs the correction process described above. That is, in this case, the movement estimation unit 127 corrects the movement error during the three-dimensional movement of the tracked object that occurs due to the setting switching of the priority image Pp.

Next, the vehicle control unit 13 performs various vehicle controls (such as the driving control of the vehicle 10 and the operation control of various components) on the vehicle 10 using the tracking position TB of the tracked object described above and the speed VB estimated in step S110 (step S111).

This completes the series of processing examples shown in Figures 7A and 7B.

(C. Actions and Effects)
In this manner, in the present embodiment, dense feature values FB (clearer than sparse feature values FA) are extracted from the image region R of the priority image Pp in the captured image P, and object tracking is performed using the dense feature values FB in the priority image Pp. This improves the accuracy of object tracking compared to object tracking using sparse feature values FA. In addition, since object tracking is performed using the priority image Pp set as one of the left image PL and the right image PR, the burden of the tracking process is reduced compared to object tracking using both of these images (left image PL and right image PR).

In addition, in this embodiment, when it is determined that object tracking using sparse features FA is not feasible, the settings of the priority camera 11p and the priority image Pp are switched, respectively. As a result, even if the above-mentioned external factors occur in the priority camera 11p or the priority image Pp, for example, when object tracking is performed using the priority image Pp, it is possible to avoid object tracking being hindered due to external factors. Furthermore, when object tracking using sparse features FA is feasible, object tracking using the above-mentioned dense features FB can also be performed reliably and easily, so in this respect, the processing burden when tracking objects using dense features FB is also reduced.

As a result, in this embodiment, it is possible to reduce processing time (reduce processing costs) while suppressing deterioration in object tracking accuracy caused by external factors.

In addition, in this embodiment, the sparse feature FA extracted by the feature extraction unit 124A is reused to extract the dense feature FB in the feature extraction unit 124B, as follows. In other words, the dense feature FB can be easily extracted, and the processing time can be further shortened.

Furthermore, in this embodiment, when the setting of the priority image Pp is switched as described above, the movement error during the three-dimensional movement of the tracked object, which occurs due to the setting switching of the priority image Pp, is corrected, as follows. In other words, by correcting the movement error due to the setting switching of the priority image Pp, it is possible to improve the estimation accuracy of the speed VB, etc. in the movement estimation unit 127.

In addition, in this embodiment, the image region R in the captured image P is set using the distance information Iz or machine learning described above, so that such an image region R can be easily set, making it possible to further shorten the processing time.

In addition, in this embodiment, if it is determined that object tracking using sparse features FA is not feasible in both the left image PL and the right image PR, object tracking using dense features FB in the object tracking unit 125B is not performed, resulting in the following. In other words, if object tracking using sparse features FA is not feasible, there is a high probability that object tracking using dense features FB will also be not feasible. Therefore, by terminating object tracking when it is not feasible in both images, it is possible to prevent unnecessary increases in the processing load. As a result, it is possible to further shorten the processing time.

2. Modifications
Although the present disclosure has been described above by way of the embodiment, the present disclosure is not limited to this embodiment and various modifications are possible.

For example, the configuration (type, shape, arrangement, number, etc.) of each component in the vehicle 10 and the image processing device 12 is not limited to that described in the above embodiment. In other words, the configuration of each of these components may be of other types, shapes, arrangements, numbers, etc. Furthermore, the values, ranges, magnitude relationships, etc. of the various parameters described in the above embodiment are not limited to those described in the above embodiment, and may be other values, ranges, magnitude relationships, etc.

Specifically, for example, in the above embodiment, the stereo camera 11 is configured to capture an image in front of the vehicle 10, but this is not limited to the configuration, and for example, the stereo camera 11 may be configured to capture an image of the side or rear of the vehicle 10.

In addition, for example, in the above embodiment, an example is described in which the image processing device 12 is provided with a movement estimation unit 127, but this is not limited to the example, and for example, the image processing device 12 may not be provided with a movement estimation unit 127.

Furthermore, in the above embodiment, the various processes performed in the vehicle 10 and the image processing device 12 are described using specific examples, but the present invention is not limited to these specific examples. In other words, the various processes may be performed using other methods. Specifically, for example, the switching process of the priority camera 11p and the priority image Pp described above is not limited to the method described in the above embodiment. Also, for example, in the above embodiment, an example is described in which the dense feature amount FB is extracted by reusing the sparse feature amount FA, but the present invention is not limited to this example, and for example, the dense feature amount FB may be extracted without reusing the sparse feature amount FA.

In addition, the series of processes described in the above embodiment may be performed by hardware (circuits) or by software (programs). When performed by software, the software is composed of a group of programs for causing a computer to execute each function. Each program may be, for example, pre-installed in the computer and used, or may be installed in the computer from a network or recording medium and used.

In addition, in the above embodiment, an example in which the image processing device 12 is provided in a vehicle has been described, but this is not limited to the example, and such an image processing device 12 may be provided, for example, in a moving body other than a vehicle or in a device other than a moving body.

Furthermore, the various examples described above may be applied in any combination.

Note that the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

The present disclosure can also be configured as follows.
(1)
a region setting unit that sets one or a plurality of image regions in a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image;
a first extraction unit that extracts a first feature amount indicating a sparse feature included in the image region of the captured image;
a second extraction unit that extracts a second feature amount that indicates a clearer dense feature compared to the sparse feature included in the image area in the left image or the right image as a priority image obtained from the left camera or the right camera that is set as a priority camera; and
a first object tracking unit that determines whether or not object tracking using the first feature amount is possible in each of the left image and the right image;
a second object tracking unit that performs object tracking in the priority image using the second feature amount;
and a switching unit that switches settings of the priority camera and the priority image when it is determined in the first object tracking unit that object tracking using the first feature in the priority image is not feasible.
(2)
The image processing device according to (1) above, wherein the second extraction unit extracts the second feature amount by reusing the first feature amount extracted by the first extraction unit.
(3)
a movement estimation unit that estimates a three-dimensional movement of the tracked object based on a tracking position of the tracked object obtained by object tracking using the second feature amount and distance information generated based on the left image and the right image,
The movement estimation unit
When the setting of the priority image is switched by the switching unit,
The image processing device according to (1) or (2) above, further comprising: correcting a movement error during the three-dimensional movement of the object to be tracked, the movement error being caused by a change in the setting of the priority image.
(4)
The image processing device according to any one of (1) to (3) above, wherein the region setting unit sets the image region using distance information generated based on the left image and the right image, or machine learning.
(5)
When it is determined that object tracking using the first feature amount is not possible in both the left image and the right image in the first object tracking unit,
The image processing device according to any one of (1) to (4) above, wherein object tracking using the second feature amount in the second object tracking unit is not performed.
(6)
An image processing device according to any one of (1) to (5) above;
a vehicle control unit that performs vehicle control by utilizing a result of object tracking using the second feature amount obtained from the second object tracking unit.
(7)
one or more processors; and one or more memories communicatively coupled to the one or more processors,
The one or more processors:
Setting one or a plurality of image regions in a captured image obtained from a stereo camera having a left camera for generating a left image and a right camera for generating a right image;
extracting a first feature amount indicating sparse features included in the image region of the captured image;
extracting a second feature amount indicating a dense feature that is clearer than the sparse feature included in the image area in the left image or the right image as a priority image obtained from the left camera or the right camera set as a priority camera; and
determining whether object tracking using the first feature amount is executable in each of the left image and the right image;
performing object tracking using the second feature amount in the priority image;
and when it is determined that object tracking using the first feature amount in the priority image is not practicable, switching settings of the priority camera and the priority image, respectively.

10...vehicle, 11...stereo camera, 11L...left camera, 11R...right camera, 11p...priority camera, 12...image processing device, 121...image memory, 122...distance information generation unit, 123...area setting unit, 124A, 124B...feature extraction unit, 125A, 125B...object tracking unit, 126...switching unit, 127...movement estimation unit, 13...vehicle control unit, 19...windshield, 8 ...obstacle, 90...preceding vehicle, PL, PL1, PL2...left image, PR, PR1, PR2...right image, PIC, PIC1, PIC2...stereo image, P...captured image, Pp...priority image, R...image area, Iz...distance information, FA...sparse feature amount, FB...dense feature amount, JA...judgment result, Ss...switching signal, TB...tracking position, VB...speed, PX...pixel, PXa-PXd...pixel value.

Claims (6)

a region setting unit that sets one or a plurality of image regions in a captured image obtained from a stereo camera having a left camera that generates a left image and a right camera that generates a right image;
a first extraction unit that extracts a first feature amount indicating a sparse feature included in the image region of the captured image;
a second extraction unit that extracts a second feature amount that indicates a clearer dense feature compared to the sparse feature included in the image area in the left image or the right image as a priority image obtained from the left camera or the right camera that is set as a priority camera; and
a first object tracking unit that determines whether or not object tracking using the first feature amount is possible in each of the left image and the right image;
a second object tracking unit that performs object tracking in the priority image using the second feature amount;
and a switching unit that, when it is determined in the first object tracking unit that object tracking using the first feature amount in the priority image is not feasible, switches one of the stereo cameras that has not been set as the priority camera to the priority camera, and switches the captured image obtained from the camera that has not been set as the priority camera to the priority image . The image processing device according to claim 1 , wherein the second extraction unit extracts the second feature amount by using the first feature amount extracted by the first extraction unit. a movement estimation unit that estimates a three-dimensional movement of the tracked object based on a tracking position of the tracked object obtained by object tracking using the second feature amount and distance information generated based on the left image and the right image,
The movement estimation unit
When the setting of the priority image is switched by the switching unit,
The image processing device according to claim 1 or 2, further comprising: correcting a movement error that occurs during the three-dimensional movement of the tracked object due to a change in the setting of the priority image. The image processing device according to claim 1 , wherein the region setting unit sets the image region by using distance information generated based on the left image and the right image, or by using machine learning. When it is determined that object tracking using the first feature amount is not possible in both the left image and the right image in the first object tracking unit,
The image processing device according to claim 1 , wherein object tracking using the second feature amount in the second object tracking section is not performed. An image processing device according to any one of claims 1 to 5,
a vehicle control unit that performs vehicle control by utilizing a result of object tracking using the second feature amount obtained from the second object tracking unit.

Images