JP7621828B2 - IMAGE PROCESSING DEVICE, STEREO CAMERA DEVICE, MOBILE BODY, AND IMAGE PROCESSING METHOD - Google Patents

Description

The present invention relates to an image processing device, a stereo camera device, a moving object, and an image processing method.

It is known that the position of an object in real space is calculated based on the difference in the position (parallax) of the same object in each stereo image captured by a stereo camera. For example, a block matching method is used as a method of calculating parallax. In block matching, if a background is included in a block, objects at different distances are likely to be included, and the degree of match of the block matching decreases. This can reduce the accuracy of the parallax calculation. Furthermore, in an outdoor environment, there may be a lot of vertical edge information and horizontal edge information specific to man-made objects such as buildings, and this edge information can have a significant impact on the degree of match of the block matching. This significant impact on the degree of match can reduce the accuracy of the parallax calculation.

Therefore, it has been proposed to calculate the primary disparity in a stereo pair and expand or reduce the window size of the matching window based on the primary disparity (see Patent Document 1).

JP 2014-174930 A

In block matching, if the size of a block could be changed depending on the distance of the object to which the pixels contained in the block belong, it is thought that the degree of agreement in block matching could be improved. However, if the accuracy of the primary disparity is low, it is difficult to improve the accuracy of the disparity calculation even if the block size is changed. Furthermore, for images that contain edge information in the vertical and horizontal directions, it is difficult to improve the accuracy of the disparity calculation even if the block size is changed.

Therefore, in consideration of the problems with the conventional technology described above, the objective of the present disclosure is to provide an image processing device, a stereo camera device, a moving object, and an image processing method that improve the accuracy of calculating disparity.

In order to solve the above-mentioned problems, an image processing apparatus according to a first aspect comprises:
An acquisition unit that acquires a standard image and a reference image captured by a stereo camera;
a control unit that calculates a disparity of a subject image within the search area based on a first degree of difference between a search area including a reference point in the base image and a comparison area that is displaced in pixel units along a baseline length direction of the stereo camera in the reference image,
The control unit is
clustering the pixels in the reference image so that the pixels are grouped by object;
In the search region, pixels that belong to the same subject as the subject to which the reference point belongs are weighted differently from pixels that belong to subjects other than the same subject, to calculate the first dissimilarity.

Moreover, a stereo camera device according to a second aspect of the present invention comprises:
A stereo camera that captures a standard image and a reference image;
an image processing device that calculates a disparity of a subject image within the search area based on a first degree of difference between a search area including a reference point in the standard image and a comparison area that is displaced in pixel units along a baseline length direction of the stereo camera in the reference image;
The image processing device includes:
In the reference image, pixels are clustered by subject;
In the search region, pixels that belong to the same subject as the subject to which the reference point belongs are weighted differently from pixels that belong to subjects other than the same subject, to calculate the first dissimilarity.

A moving body according to a third aspect of the present invention comprises:
a stereo camera device including a stereo camera for capturing a standard image and a reference image; and an image processing device for calculating a parallax of a subject image in a search area based on a first degree of difference between a search area including a standard point in the standard image and a comparison area displaced in pixel units along a base line length direction of the stereo camera in the reference image;
The stereo camera device includes:
In the reference image, pixels are clustered by subject;
In the search region, pixels that belong to the same subject as the subject to which the reference point belongs are weighted differently from pixels that belong to subjects other than the same subject, to calculate the first dissimilarity.

Further, an image processing method according to a fourth aspect comprises:
A step of capturing a standard image and a reference image by a stereo camera;
clustering pixels in the reference image by object;
determining a weighting for each pixel in a search area including a reference point in the reference image such that pixels belonging to the same subject as the subject to which the reference point belongs are weighted differently from pixels belonging to subjects other than the same subject;
The method includes a step of calculating a disparity of a subject image within the search area based on a plurality of first disparities between a search area including a reference point in the base image and a comparison area in the reference image which is displaced on a pixel-by-pixel basis along a baseline length direction of the stereo camera, and a weighting of each pixel.

The image processing device, stereo camera device, moving body, and image processing method according to the present disclosure configured as described above improve the accuracy of calculating disparity.

1 is a block diagram showing a schematic configuration of a stereo camera device including an image processing device according to an embodiment of the present invention. 2 is a side view conceptually illustrating the position of the stereo camera of FIG. 1 on the moving body. 2 is a front view conceptually illustrating the position of the stereo camera of FIG. 1 on the moving body. 4 is a flowchart for explaining a first position calculation process executed by a control unit in FIG. 1 . 10 is a flowchart for explaining a second position calculation process executed by the control unit in FIG. 1 . 10 is a flowchart for explaining a third position calculation process executed by the control unit in FIG. 1 . FIG. 11 is a diagram showing a set of pixels in a search region illustrated as an example for explaining the accuracy of disparity calculation. 8 is a diagram showing a set of pixels of a reference image cut out in a range corresponding to the search region exemplified in FIG. 7 . 9 is a diagram showing the ranges of a first comparison region, a second comparison region, and a third comparison region specified for the reference image of FIG. 8 .

Below, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

As shown in FIG. 1, a stereo camera device 11 including an image processing device 10 to which the present disclosure is applied includes a stereo camera 12 and the image processing device 10. As shown in FIG. 2, the stereo camera device 11 may calculate the position of a subject corresponding to the subject image included in a stereo image in a world coordinate system based on a moving body 13, based on a captured stereo image. As shown in FIG. 1, the stereo camera device 11 may notify an external device 14 of the position of the subject in the world coordinate system. The external device 14 may include a driving support device such as an auto cruise control, a driving support device such as an automatic braking device, a notification device that notifies the distance to the subject, and a warning device that issues a warning.

The moving body 13 may include, for example, vehicles, ships, aircraft, etc. The vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, residential vehicles, fixed-wing aircraft running on runways, etc. The automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses, etc. The industrial vehicles may include, for example, industrial vehicles for agriculture and construction. The industrial vehicles may include, for example, forklifts, golf carts, etc. The agricultural industrial vehicles may include, for example, tractors, cultivators, transplanters, binders, combines, lawnmowers, etc. The construction industrial vehicles may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, road rollers, etc. The vehicles may include those that run by human power. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can run on roads. The same vehicle may be included in multiple classifications. The ships may include, for example, marine jets, boats, and tankers. Aircraft may include, for example, fixed-wing aircraft, rotorcraft, etc.

The stereo camera 12 and the image processing device 10 may be capable of communicating with each other via wired or wireless communication. The stereo camera 12 and the image processing device 10 may communicate with each other via a network. The network may include, for example, a wired or wireless LAN (Local Area Network), CAN (Controller Area Network), etc. The stereo camera 12 and the image processing device 10 may be housed in the same housing and configured as an integrated unit. The stereo camera 12 and the image processing device 10 may be located in a moving object 13 and configured to be capable of communicating with an ECU (Electronic Control Unit) in the moving object 13.

As shown in FIG. 2, the stereo camera device 11 may be mounted on, for example, a moving body 13. The stereo camera 12 may be installed, for example, so as to capture an image in front of the moving body 13. The stereo camera 12 is arranged, for example, so that the optical axis OX of at least one of the multiple cameras included in the stereo camera 12, which will be described later, is approximately parallel to the front-rear direction of the moving body 13.

The stereo camera 12 may be mounted in various locations on the moving body 13. For example, the stereo camera 12 is mounted inside the moving body 13 and captures an image of the outside of the moving body 13 through the windshield. The stereo camera 12 may be disposed in front of the rearview mirror or on the dashboard. The stereo camera 12 may be fixed to any of the front bumper, fender grille, side fender, light module, and bonnet of the moving body 13.

The stereo camera 12 may include multiple cameras having parallax with respect to each other. As shown in FIG. 1, the stereo camera 12 may include, for example, a first camera 15 and a second camera 16. The stereo camera 12 may capture an image of an object from multiple directions by using multiple cameras in cooperation with each other. The stereo camera 12 may include multiple cameras in a single housing. The stereo camera 12 may include two or more cameras that are independent of each other and located apart from each other. The stereo camera 12 is not limited to multiple cameras that are independent of each other. In the present disclosure, for example, a camera having an optical mechanism that guides light incident at two separate points to one light receiving element may be adopted as the stereo camera 12.

In the stereo camera 12, the first camera 15 and the second camera 16 may be positioned so that their optical axes OX face in directions that allow them to capture the same subject. The optical axes OX and positions of the first camera 15 and the second camera 16 may be determined so that the captured images include at least the same subject image. The optical axes OX of the first camera 15 and the second camera 16 are, for example, parallel to each other. The optical axes OX being parallel is not limited to being strictly parallel, and may allow for misalignment in assembly, misalignment in installation, and misalignment over time. The optical axes OX of the first camera 15 and the second camera 16 are not limited to being parallel, and may face in different directions.

The first camera 15 and the second camera 16 may be positioned apart in a direction intersecting the optical axis OX. As shown in FIG. 3, the first camera 15 and the second camera 16 may be positioned along the left-right direction in a posture determined when they are attached to the moving body 13. Due to differences in the positions of the first camera 15 and the second camera 16, the positions of the objects corresponding to each other in the two images captured by each camera are different. The base image generated by the first camera 15 and the reference image generated by the second camera 16 are stereo images captured from different viewpoints. The first camera 15 and the second camera 16 may capture the object at a predetermined frame rate (e.g., 30 fps).

The first camera 15 and the second camera 16 may each include an optical system that defines an optical axis OX and an imaging element. The first camera 15 and the second camera 16 may each have a different optical axis OX. The optical system may include a lens or a mirror, and may form an image of a subject on the light receiving surface of the imaging element. The imaging element may be, for example, a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor). The imaging elements of the first camera 15 and the second camera 16 may be present in the same plane perpendicular to the optical axis OX of each camera. The imaging elements of the first camera 15 and the second camera 16 may have a plurality of pixels arranged in a matrix. In the imaging elements of the first camera 15 and the second camera 16, the row direction in which pixels are arranged may be parallel to the base length direction of the stereo camera 12. The base length direction is the direction of a straight line passing through the intersection of the optical axis OX to the light receiving surface of the imaging element of the first camera 15 and the intersection of the optical axis OX to the light receiving surface of the imaging element of the second camera 16. The first camera 15 and the second camera 16 generate the image formed on the imaging element as a signal. The first camera 15 and the second camera 16 may perform any image processing such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction on the captured image.

As shown in FIG. 1, the image processing device 10 includes an acquisition unit 17, an output unit 18, a memory 19, and a control unit 20. The image processing device 10 may be disposed at any position within the mobile object 13. The image processing device 10 is disposed, for example, within the dashboard of the mobile object 13.

The acquisition unit 17 is an input interface of the image processing device 10 that acquires information from the stereo camera 12 and external devices. The acquisition unit 17 may employ a physical connector and a wireless communication device. The physical connector may include an electrical connector compatible with transmission by electrical signals, an optical connector compatible with transmission by optical signals, and an electromagnetic connector compatible with transmission by electromagnetic waves. The electrical connector may include a connector conforming to IEC 60603, a connector conforming to the USB standard, a connector compatible with an RCA terminal, a connector compatible with an S terminal defined in EIAJ CP-1211A, a connector compatible with a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector compatible with a coaxial cable including BNC. The optical connector may include various connectors conforming to IEC 61754. The wireless communication device may include a wireless communication device conforming to each standard including Bluetooth (registered trademark) and IEEE 802.11. The wireless communication device may include at least one antenna.

The acquisition unit 17 acquires the standard image and the reference image generated by the stereo camera 12 through imaging as signals. The acquisition unit 17 passes the acquired standard image and reference image to the control unit 20. The acquisition unit 17 may be compatible with a transmission method for the imaging signal of the stereo camera 12. The acquisition unit 17 may be connected to the output interface of the stereo camera 12 via a network.

The output unit 18 is an output interface of the image processing device 10 that can output the position of the subject in the world coordinate system calculated by the image processing device 10 to other devices within the mobile body 13 or to external devices 14 outside the mobile body 13, such as other mobile bodies or roadside units. The output unit 18 may include various interfaces compatible with wired and wireless communication, similar to the acquisition unit 17. The output unit 18 has, for example, a CAN interface and communicates with other devices within the mobile body 13.

The memory 19 may include any storage device, such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 19 may store various programs that cause the control unit 20 to function, and various information used by the control unit 20.

The control unit 20 may include one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 20 may be either a system-on-a-chip (SoC) in which one or more processors work together, or a system in a package (SIP).

The control unit 20 performs clustering so that each pixel in the reference image is grouped by subject. The control unit 20 may perform clustering using various methods. The control unit 20 may perform clustering using at least one method. The control unit 20 may perform clustering using a combination of multiple methods.

For example, the control unit 20 may perform clustering based on optical flow, which will be described below. To perform clustering based on optical flow, the control unit 20 may continuously acquire reference images. Continuously acquiring reference images means acquiring reference images of two or more frames captured at different times at an arbitrary frame rate. The reference images of two or more frames captured at different times may be reference images of frames captured at consecutive times.

The control unit 20 calculates the optical flow for each pixel of the reference image using any method, such as the Lucas-Kanade method. The control unit 20 may consider pixels that have the same optical flow, i.e., the same motion vector between successive reference images, and are close to each other, as belonging to the same subject, among the calculated optical flows for each pixel, as belonging to the same subject. The control unit 20 may group the pixels by subject by considering each pixel as belonging to any subject.

For example, the control unit 20 may perform clustering based on occlusion regions, which will be described below. An occlusion region is a region that is captured in one of the base image and the reference image and not captured in the other, such as a subject region such as a distant background around a close subject.

The control unit 20 may determine the occlusion region by comparing the parallax from the standard image to the reference image and the parallax from the reference image to the standard image. The control unit 20 may calculate a first parallax from each pixel of the standard image to each pixel of the reference image by block matching. The control unit 20 may also calculate a second parallax from each pixel of the reference image to each pixel of the standard image by block matching.

The control unit 20 may calculate the dissimilarity based on a comparison of the first dissimilarity and the second dissimilarity. The dissimilarity may be the absolute value of the sum of the first dissimilarity and the second dissimilarity with the polarity reversed. In an occlusion region, the first dissimilarity is likely to differ from the second dissimilarity with the polarity reversed. Therefore, the control unit 20 may consider a pixel whose dissimilarity is equal to or greater than a threshold value to be a pixel in an occlusion region.

The control unit 20 may recognize contours formed by pixels that are considered to be within an occlusion region and are adjacent to each other in the reference image. The control unit 20 may divide the reference image into a plurality of regions based on the recognized contours. The control unit 20 may group each pixel of the reference image by subject based on which region it is included in.

The control unit 20 may create at least a first fitting function as described below. The control unit 20 designates an arbitrary region in the reference image as a search region. For example, the control unit 20 designates an arbitrary pixel as a pixel of interest and designates an area of a predetermined size including the pixel of interest as the search region. The search region may be, for example, an area of m rows and n columns (m and n are natural numbers) centered on the pixel of interest. In this specification, a point in the reference image that corresponds to the pixel of interest is called a reference point.

The control unit 20 designates, as a comparison region, a region in the reference image that is displaced in pixel units from the position of the search region along the baseline length direction. The control unit 20 designates multiple comparison regions by displacing one pixel at a time in both directions along the baseline length direction.

The control unit 20 calculates a first dissimilarity between the partial images included in the search area and the comparison area. In calculating the first dissimilarity, the control unit 20 performs weighting based on clustering, as described below. The control unit 20 determines the subject to which the reference point belongs in the search area, in other words, the subject to which the pixel of interest belongs. The control unit 20 determines pixels in the search area that belong to the same subject as the subject to which the pixel of interest belongs. The control unit 20 performs weighting on pixels that belong to the same subject as the pixel of interest differently from pixels that belong to subjects other than the subject. For example, the control unit 20 performs weighting on pixels that belong to the same subject as the pixel of interest more heavily than pixels that belong to subjects other than the subject. More specifically, the control unit 20 may multiply the difference between the pixel value of a pixel that belongs to the same subject as the pixel of interest and the pixel value of a pixel located at the same position as the pixel of interest in the comparison area by a coefficient that is weighted more heavily than other pixels. For example, the coefficient is 2 for pixels that belong to the same subject as the pixel of interest, and 1 for pixels that belong to a different subject than the pixel of interest.

The control unit 20 calculates the first dissimilarity using the difference between the pixel values of pixels at the same position in the search area and the comparison area multiplied by the above-mentioned coefficient. The first dissimilarity is an index that changes depending on the difference in pixel values of pixels that make up the partial image, such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference). The control unit 20 calculates multiple first dissimilarity values for each displaced position.

In general, the comparison area in which the dissimilarity is an extreme value is the most similar to the search area among the comparison areas for which the first dissimilarity is calculated. In this embodiment, the extreme value of the first dissimilarity is a minimum value in a configuration in which the dissimilarity increases as the difference between the search area and the comparison area increases, such as SAD and SSD.

The control unit 20 may create a first fitting function based on the first dissimilarity calculated for each of the multiple comparison regions. The first fitting function is a function that indicates the relationship of the first dissimilarity to the amount of movement of the comparison region in the baseline length direction relative to the search region. The first fitting function may be created by performing equiangular linear fitting or parabolic fitting, etc., so that the first fitting function passes through the dissimilarity corresponding to the amount of movement.

The control unit 20 may further create a second fitting function, as described below. The control unit 20 may displace one of the base image and the reference image by 0.5 pixels in the baseline length direction. Displacing by 0.5 pixels means, for example, performing pixel interpolation by averaging the pixel values of pixels that make up the image adjacent to each other along the displacement direction, and calculating the pixel value at the position displaced by 0.5 pixels. The displacement of one of the base image and the reference image may not be the entire image, but may be the periphery of the search area and comparison area where the parallax is calculated. The displacement direction may be a negative direction of -0.5 pixels.

The control unit 20 may designate as the search area an area of the same size and at the same position within the entire image when the first fitting function was created, with one of the base image and the reference image displaced by 0.5 pixels in the baseline length direction.

The control unit 20 may designate, as a comparison region, a region in the reference image that has been displaced pixel by pixel from the position of the search region along the baseline length direction, with one of the base image and the reference image displaced by 0.5 pixels in the baseline length direction. The control unit 20 may designate multiple comparison regions by displacing one pixel at a time in both directions along the baseline length direction.

The control unit 20 may calculate a second dissimilarity between the partial images included in the search area and the comparison area with one of the standard image and the reference image displaced by 0.5 pixels in the baseline length direction. In calculating the second dissimilarity, the control unit 20 may apply weighting based on clustering, similar to the calculation of the first dissimilarity.

In a configuration in which the reference image is displaced by 0.5 pixels, the control unit 20 may perform clustering on the reference image displaced by 0.5 pixels, thereby grouping each pixel of the reference image displaced by 0.5 pixels by subject. Alternatively, the control unit 20 may group each pixel of the reference image displaced by 0.5 pixels by displacing the clustering of the reference image used to calculate the first dissimilarity by 0.5 pixels. In a configuration in which the reference image is displaced by 0.5 pixels, the control unit 20 may apply the clustering used to calculate the first dissimilarity directly to the calculation of the second dissimilarity.

In calculating the second dissimilarity, the control unit 20 may determine pixels within the search area that belong to the same subject as the subject to which the pixel of interest belongs. The control unit 20 may weight pixels that belong to the same subject as the pixel of interest differently from pixels that belong to subjects other than the pixel of interest. For example, the control unit 20 may weight pixels that belong to the same subject as the pixel of interest more heavily than pixels that belong to subjects other than the pixel of interest.

The control unit 20 may calculate the second dissimilarity using the difference between the pixel values of pixels at the same positions in the search area and the comparison area multiplied by a coefficient according to the weighting, similar to the calculation of the first dissimilarity. The second dissimilarity is an index that changes according to the difference in pixel values of pixels constituting a partial image, such as SAD (Sum of Absolute Difference) or SSD (Sum of Squared Difference), and is calculated using the same calculation formula as that used to calculate the first dissimilarity. The control unit 20 may calculate multiple second dissimilarity values for different displaced positions.

The control unit 20 may create a second fitting function based on the second dissimilarity calculated for each of the multiple comparison regions while one of the standard image and the reference image is displaced by 0.5 pixels in the baseline length direction. The second fitting function may be a function that indicates the relationship of the dissimilarity to the amount of movement of the comparison region in the baseline length direction relative to the search region. The control unit 20 may create the second fitting function using the same method as that used to calculate the first fitting function, such as equiangular linear fitting or parabolic fitting.

The control unit 20 calculates the disparity of the subject image in the search area based on the first disparity. More specifically, the control unit 20 calculates the disparity based on a first fitting function created from the first disparity. The control unit 20 may calculate, as the disparity, the amount of movement that minimizes the first disparity of the first fitting function.

In a configuration in which a second fitting function is further created, the control unit 20 may calculate the disparity based on the second disparity as well. More specifically, the control unit 20 may calculate the disparity based on the second fitting function created from the second disparity. More specifically, the control unit 20 may calculate, as the disparity, the movement amount at which the sum or average value of the first disparity and the second disparity is minimum in a fitting function obtained by summing or averaging the first disparity and the second disparity at the same movement amount of the first fitting function and the second fitting function.

The control unit 20 may calculate the position in the world coordinate system of the subject corresponding to the subject image in the search area based on the calculated parallax, exterior orientation parameters, and interior orientation parameters. The control unit 20 may control the output unit 18 to output the calculated position in the world coordinate system of the subject to the external device 14. The control unit 20 may sequentially designate a part of the reference image or the entire reference image as a comparison area and calculate the position of the subject for each pixel.

Next, the first position calculation process executed by the control unit 20 in this embodiment will be described with reference to the flowchart in FIG. 4. In the first position calculation process, the reference image is displaced by 0.5 pixels to calculate the second dissimilarity, and clustering is performed on the displaced reference image. The first position calculation process starts every time one frame of the reference image and the reference image is acquired from the stereo camera 12.

In step S100, the control unit 20 displaces the reference image by a movement amount of 0.5 pixels. After the displacement, the process proceeds to step S101.

In step S101, the control unit 20 performs clustering on the reference image and the reference image displaced by 0.5 pixels in step S100. In a configuration in which clustering based on optical flow is performed, the reference image of the immediately preceding frame is used together with the reference image of the frame acquired at the start of the first position calculation process to calculate the optical flow. In a configuration in which clustering based on occlusion regions is performed, the occlusion regions may be calculated using the reference image of the frame acquired at the start of the first position calculation process. After clustering, the process proceeds to step S102.

In step S102, the control unit 20 designates a search area from among undesignated areas among the areas to be designated for the reference image and the reference image displaced by 0.5 pixels in step S100. After designation, the process proceeds to step S103.

In step S103, the control unit 20 determines to which of the objects clustered in step S101 the pixel of interest in the search area belongs. Furthermore, the control unit 20 determines the weighting of each pixel in the search area based on whether or not the pixel belongs to the same object as the pixel of interest. After the determination, the process proceeds to step S104.

In step S104, the control unit 20 specifies multiple comparison areas for the search area specified in step S102. Furthermore, the control unit 20 calculates a first dissimilarity and a second dissimilarity based on the search area specified in step S102, the multiple comparison areas, and the weighting determined in step S103. After the calculation, the process proceeds to step S105.

In step S105, the control unit 20 creates a first fitting function based on the first dissimilarity calculated in step S104. Furthermore, the control unit 20 creates a second fitting function based on the second dissimilarity calculated in step S104. After the second fitting function is created, the process proceeds to step S106.

In step S106, the control unit 20 calculates the disparity based on the first fitting function and the second fitting function created in step S105. After calculating the disparity, the process proceeds to step S107.

In step S107, the control unit 20 calculates the position of the subject corresponding to the subject image included in the search area specified in step S102 based on the parallax, exterior orientation parameters, and interior orientation parameters calculated in step S106. After the calculation, the process proceeds to step S108.

In step S108, the control unit 20 controls the output unit 18 to output the position calculated in step S107 to the external device 14. After the output, the process proceeds to step S109.

In step S109, the control unit 20 determines whether all of the areas to be specified in the reference image and the reference image displaced by 0.5 pixels in step S100 have been specified as search areas. If some areas have not been specified, the process returns to step S102. If all of the areas to be specified have been specified, the first position calculation process ends.

Next, the second position calculation process executed by the control unit 20 in this embodiment will be described with reference to the flowchart in FIG. 5. In the second position calculation process, in order to calculate the second dissimilarity, the reference image is displaced by 0.5 pixels, and the grouping of each pixel by subject performed by clustering on the original reference image is displaced by 0.5 pixels in the same direction as the reference pixel, thereby obtaining the clustering result for the reference image displaced by 0.5 pixels. The first position calculation process is started each time one frame of the reference image and the reference image are acquired from the stereo camera 12.

In step S200, the control unit 20 displaces the reference image by a movement amount of 0.5 pixels. After the displacement, the process proceeds to step S201.

In step S201, the control unit 20 performs clustering on the reference image. In a configuration in which clustering based on optical flow is performed, the reference image of the immediately preceding frame is used together with the reference image of the frame acquired at the start of the first position calculation process to calculate the optical flow. In a configuration in which clustering based on occlusion regions is performed, the occlusion regions may be calculated using the reference image of the frame acquired at the start of the first position calculation process. After clustering, the process proceeds to step S202.

In step S202, the control unit 20 displaces the grouping of each pixel by subject, which was performed by the clustering in step S201, in the same direction as the reference pixel by a movement amount of 0.5 pixels. After the displacement, the process proceeds to step S203.

In step S203, the control unit 20 specifies a search area from among unspecified areas among the areas to be specified for the reference image and the reference image displaced by 0.5 pixels in step S100. After the search area is specified, the process proceeds to step S204.

In step S204, the control unit 20 determines to which of the objects clustered in step S201 the pixel of interest in the search area of the reference image belongs. The control unit 20 also determines to which of the objects clustered in step S202 the pixel of interest in the search area of the reference image displaced by 0.5 pixels belongs. Furthermore, the control unit 20 determines the weighting of each pixel in the search area based on whether or not the pixel of interest belongs to the same object as the pixel of interest. After the determination, the process proceeds to step S205.

In steps S205 to S210, the control unit 20 performs processing similar to steps S104 to S109 in the first position calculation process. In step S210, if all areas to be specified have been specified, the second position calculation process ends.

Next, the third position calculation process executed by the control unit 20 in this embodiment will be described with reference to the flowchart in FIG. 6. In the third position calculation process, the reference image is displaced by 0.5 pixels to calculate the second dissimilarity. The third position calculation process starts every time one frame of the base image and the reference image is acquired from the stereo camera 12.

In step S300, the control unit 20 displaces the reference image by a movement amount of 0.5 pixels. After the displacement, the process proceeds to step S302.

In step S301, the control unit 20 performs clustering on the reference image. In a configuration in which clustering based on optical flow is performed, the reference image of the immediately preceding frame is used together with the reference image of the frame acquired at the start of the first position calculation process to calculate the optical flow. In a configuration in which clustering based on occlusion regions is performed, the occlusion regions may be calculated using the reference image of the frame acquired at the start of the first position calculation process. After clustering, the process proceeds to step S302.

In step S302, the control unit 20 specifies a search area from among the areas to be specified for the reference image that are not specified. After specification, the process proceeds to step S303.

In step S303, the control unit 20 determines to which of the objects clustered in step S301 the pixel of interest in the search area belongs. Furthermore, the control unit 20 determines the weighting of each pixel in the search area based on whether or not the pixel belongs to the same object as the pixel of interest. After the determination, the process proceeds to step S304.

In step S304, the control unit 20 specifies multiple comparison regions in the reference image for the search region specified in step S302. The control unit 20 also specifies multiple comparison regions in the reference image displaced by 0.5 pixels in step S300 for the search region. Furthermore, the control unit 20 calculates a first dissimilarity and a second dissimilarity based on the search region specified in step S302, the multiple comparison regions, and the weighting determined in step S303. After the calculation, the process proceeds to step S305.

In steps S305 to S309, the control unit 20 performs processing similar to steps S105 to S109 in the first position calculation process. If, in step S309, all areas to be specified have been specified, the third position calculation process ends.

The image processing device 10 of the present embodiment configured as described above has a control unit 20 that calculates the parallax of the subject image in the search area based on the first dissimilarity between the search area including the reference point in the reference image and the comparison area displaced in pixel units along the baseline length direction in the reference image. The control unit 20 performs clustering so that each pixel in the reference image is grouped by subject, and calculates the first dissimilarity by weighting pixels belonging to the same subject as the subject to which the reference point belongs in the search area more heavily than pixels belonging to subjects other than the subject. In general, the captured image contains an image of a subject that is the background of the subject, along with the type of subject to be observed. Since the background may contain multiple subjects at different distances, the accuracy of the calculation of the parallax by block matching decreases. In particular, in outdoor natural environments, the image contains a lot of vertical and horizontal edge information of buildings, etc., so the accuracy of the calculation of the parallax using block matching further decreases due to the strong influence of the edge information. In response to such an event, the image processing device 10 configured as described above heavily weights pixels belonging to the same subject as the pixel of interest, as will be described below using a specific example, and therefore greatly increases the fitting function for parallax other than the accurate parallax. Therefore, the image processing device 10 increases the likelihood that the value of the fitting function for the accurate parallax will be smaller than the value of the fitting function for the inaccurate parallax. Therefore, the image processing device 10 improves the accuracy of calculating the parallax.

To explain the accuracy of calculating the parallax, a search area SR with 5 rows and 5 columns is illustrated as shown in FIG. 7. In the illustrated search area SR, the pixel of interest FP in the center belongs to subject A located at a short distance from the stereo camera 12. In addition, in the search area SR, six pixels including the pixel of interest FP, which are one column to the right and two columns below the pixel of interest FP, belong to the subject A. The luminance of the pixels belonging to the subject A is 50. The five pixels lined up in the leftmost column in the search area SR belong to subject B at a medium distance. The luminance of the pixels belonging to the subject B is 100. In the search area SR, all pixels other than those belonging to subjects A and B belong to subject C at a long distance. The luminance of the pixels belonging to the subject C is 200.

As shown in FIG. 8, a reference image RI cut out into 5 rows and 7 columns corresponding to the search region SR in FIG. 7 is exemplified. In the illustrated reference image RI, a total of 15 pixels arranged in the two leftmost columns and the one rightmost column belong to subject B at a medium distance. In addition, in the reference image RI, a total of 6 pixels arranged in the first to third rows from the bottom of the second and third columns from the right end belong to subject A at a close distance. In the reference image RI, all pixels other than those belonging to subjects A and B belong to subject C at a long distance.

As shown in FIG. 9, in the reference image RI, a region of 5 rows and 5 columns of pixels including the first column from the left end is defined as a first comparison region CR1. In the reference image RI, a region of 5 rows and 5 columns of pixels including the second column from the left end is defined as a second comparison region CR2. In the reference image RI, a region of 5 rows and 5 columns of pixels including the third column from the left end is defined as a third comparison region CR3. For the reference image RI shown in the figure, the third comparison region CR3 is the region that corresponds to the accurate parallax.

For the search area SR illustrated in FIG. 7, and the first comparison area CR1, second comparison area CR2, and third comparison area CR3 illustrated in FIG. 9, the first dissimilarity is calculated as follows. The first dissimilarity is calculated separately for a configuration in which no weighting is applied and a configuration in which weighting is applied as in this embodiment. In this example, the first dissimilarity is calculated by SAD. The weighting coefficient is 2 for pixels that belong to the same subject as the pixel of interest FP, and 1 for pixels that belong to a different subject from the subject.

In a configuration without weighting, the first dissimilarity is 1850 for the first comparison region CR1. In a configuration without weighting, the first dissimilarity is 900 for the second comparison region CR2. In a configuration without weighting, the first dissimilarity is 1000 for the third comparison region CR3. Thus, in a configuration without weighting, the first dissimilarity is minimal in the second comparison region CR2.

In the weighted configuration, the first dissimilarity is 2750 for the first comparison region CR1. In the weighted configuration, the first dissimilarity is 1350 for the second comparison region CR2. In the weighted configuration, the first dissimilarity is 1000 for the third comparison region CR3. Thus, in the weighted configuration, the first dissimilarity is minimal in the third comparison region CR3.

In addition, the image processing device 10 of this embodiment performs clustering based on optical flow. Pixels with the same optical flow are likely to display the same subject image. Therefore, the image processing device 10 having the above-mentioned configuration improves the accuracy of clustering subject images corresponding to subjects.

The image processing device 10 of this embodiment also calculates disparity by comparing the parallax from each pixel of the base image to each pixel of the reference image with the parallax from each pixel of the reference image to each pixel of the base image, and performs clustering by dividing the pixels whose disparity is equal to or greater than a threshold into contours. With this configuration, the image processing device 10 improves the accuracy of clustering of subject images corresponding to the subject, even if the subject is stationary.

In addition, the image processing device 10 of this embodiment performs clustering by combining multiple methods. With this configuration, the image processing device 10 can more accurately group actually different subjects in images. For example, with clustering based on optical flow, it is difficult to group subjects with little movement by image. On the other hand, with clustering based on occlusion regions, it is possible to group subjects with little movement by image. Therefore, by combining these methods, as described above, actually different subjects can be more accurately grouped.

The image processing device 10 also calculates the parallax of the subject image within the search area based on a first degree of difference between a search area including a reference point in the reference image and a comparison area displaced in pixel units along the baseline length direction in the reference image, and a second degree of difference between the search area of the reference image and a comparison area displaced in pixel units along the baseline length direction in the reference image with one of the reference image and the reference image displaced by 0.5 pixels in the baseline length direction, in other words, based on a first fitting function and a second fitting function. With this configuration, in other words, the EEC method, the image processing device 10 can reduce systematic errors in sub-pixel estimation.

Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined into one or divided. Although the embodiments of the present disclosure have been described mainly with respect to the device, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments of the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor provided in the device. It should be understood that these are also included in the scope of the present disclosure.

In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure may have their numbers exchanged. For example, a first camera may exchange identifiers "first" and "second" with a second camera. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the identifier exchange. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. Descriptions of identifiers such as "first" and "second" in this disclosure alone should not be used to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

REFERENCE SIGNS LIST 10 Image processing device 11 Stereo camera device 12 Stereo camera 13 Moving object 14 External device 15 First camera 16 Second camera 17 Acquisition unit 18 Output unit 19 Memory 20 Control unit CR1 First comparison area CR2 Second comparison area CR3 Third comparison area FP Target pixel RI Reference image SR Search area

Claims (7)

An acquisition unit that acquires a standard image and a reference image captured by a stereo camera;
a control unit that calculates a disparity of a subject image within the search area based on a first degree of difference between a search area including a reference point in the base image and a comparison area that is displaced in pixel units along a baseline length direction of the stereo camera in the reference image,
The control unit is
clustering the pixels in the reference image so that the pixels are grouped by object;
calculating the first dissimilarity degree by weighting pixels in the search area that belong to the same object as the object to which the reference point belongs differently from pixels that belong to objects other than the same object ;
calculating a second dissimilarity between a search area of the standard image and a comparison area in the reference image that is displaced by a pixel unit along the base line direction in a state in which one of the standard image and the reference image is displaced by 0.5 pixels in the base line length direction, and calculating the dissimilarity based on a sum or an average value of the first dissimilarity and the second dissimilarity;
In the search area used for calculating the second dissimilarity, a weighting different from that of pixels belonging to an object other than the object to which the reference point of the search area belongs is applied to pixels belonging to the same object.
Image processing device. 2. The image processing device according to claim 1,
The control unit performs the clustering by at least one method. 3. The image processing device according to claim 1,
The acquisition unit continuously acquires the reference images,
The control unit performs the clustering based on an optical flow between reference images captured at different times. 4. The image processing device according to claim 1,
The control unit calculates disparity by comparing the disparity from each pixel of the base image to each pixel of the reference image with the disparity from each pixel of the reference image to each pixel of the base image, and performs the clustering by dividing the image into contours made up of pixels whose disparity is equal to or greater than a threshold. A stereo camera that captures a standard image and a reference image;
an image processing device that calculates a disparity of a subject image within the search area based on a first degree of difference between a search area including a reference point in the standard image and a comparison area that is displaced in pixel units along a baseline length direction of the stereo camera in the reference image;
The image processing device includes:
In the reference image, pixels are clustered by subject;
calculating the first dissimilarity degree by weighting pixels in the search area that belong to the same object as the object to which the reference point belongs differently from pixels that belong to objects other than the same object ;
calculating a second dissimilarity between a search area of the standard image and a comparison area in the reference image that is displaced by a pixel unit along the base line direction in a state in which one of the standard image and the reference image is displaced by 0.5 pixels in the base line length direction, and calculating the dissimilarity based on a sum or an average value of the first dissimilarity and the second dissimilarity;
In the search area used for calculating the second dissimilarity, a weighting different from that of pixels belonging to an object other than the object to which the reference point of the search area belongs is applied to pixels belonging to the same object.
Stereo camera device. a stereo camera device including a stereo camera for capturing a standard image and a reference image; and an image processing device for calculating a parallax of a subject image in a search area based on a first degree of difference between a search area including a standard point in the standard image and a comparison area displaced in pixel units along a base line length direction of the stereo camera in the reference image;
The stereo camera device includes:
In the reference image, pixels are clustered by subject;
calculating the first dissimilarity degree by weighting pixels in the search area that belong to the same object as the object to which the reference point belongs differently from pixels that belong to objects other than the same object ;
calculating a second dissimilarity between a search area of the standard image and a comparison area in the reference image that is displaced by a pixel unit along the base line direction in a state in which one of the standard image and the reference image is displaced by 0.5 pixels in the base line length direction, and calculating the dissimilarity based on a sum or an average value of the first dissimilarity and the second dissimilarity;
In the search area used for calculating the second dissimilarity, a weighting different from that of pixels belonging to an object other than the object to which the reference point of the search area belongs is applied to pixels belonging to the same object.
Mobile body. A step of capturing a standard image and a reference image by a stereo camera;
clustering pixels in the reference image by object;
determining a weight for each pixel in a search area including a reference point in the reference image such that pixels belonging to the same subject as the subject to which the reference point belongs are weighted differently from pixels belonging to subjects other than the same subject;
calculating a disparity of a subject image within the search area based on a plurality of first dissimilarities between a search area including a reference point within the standard image and a comparison area displaced in pixel units along a baseline length direction of the stereo camera within the reference image, and a weighting of each pixel ;
calculating a second dissimilarity between a search area of the standard image and a comparison area in the reference image that is displaced by a pixel unit along the base line direction while one of the standard image and the reference image is displaced by 0.5 pixels in the base line length direction, and calculating the dissimilarity based on a sum or average value of the first dissimilarity and the second dissimilarity;
and weighting pixels in a search area used to calculate the second dissimilarity differently from pixels that belong to an object that is the same as an object to which a reference point of the search area belongs .

Images