JP7532238B2 - GENERATION APPARATUS, GENERATION METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to distance measurement technology that calculates the distance to a subject.

In recent years, distance measurement technology has come to be used for a variety of purposes, such as autonomous driving and factory automation (FA), and there are a variety of distance measurement technologies. For example, one distance measurement technology uses images captured by a stereo camera. In this method, a stereo camera is used to simultaneously capture images of a subject from different directions using multiple image capture devices, and distance information to the subject is calculated based on the parallax of the multiple captured images (Patent Document 1).

Another distance measurement technology is one in which pixels with distance measurement capabilities (distance measurement pixels) are arranged in some or all of the pixels of an image sensor, and the distance to the subject is calculated using a phase difference method (Patent Document 2). In this method, multiple photodiodes are arranged for one distance measurement pixel, and separate captured images (referred to as image A and image B, respectively) are generated, and the distance to the subject is calculated based on the amount of positional deviation of the captured images.

There are also distance measurement methods that do not use image sensors. For example, the Time Of Flight (TOF) method measures the distance to a subject by shining light on the subject and detecting the time it takes for the light to reflect back.

Distance measuring devices using these distance measuring technologies suffer from distortion due to optical factors. For example, in distance measuring devices that use an image sensor, lens distortion aberration causes distortion in the captured image, and if a distance image is generated by calculating the distance based on the captured image containing this distortion, the distance image will also be distorted. For this reason, it is common to either perform distortion correction on the captured image and generate a distance image based on the image after distortion correction, or to perform distortion correction on the generated distance image.

There are various methods for distortion correction, but all of them require referencing the pixel values of the pre-correction image that correspond to each pixel in the post-correction image, as well as the surrounding pixels.

JP 2018-105682 A Patent No. 5192096

However, the more accurate the correction method, the greater the memory bandwidth required to read pixel values of the pre-correction image from memory for distortion correction. This means that calculating highly accurate distance information requires a larger memory bandwidth, which can lead to memory bandwidth shortages.

The present invention is a generating device that generates a distance image that represents the distance to a subject, and is characterized by comprising: a generating means that generates the distance image based on a pair of captured images having parallax; and a correcting means that corrects distortion in the distance image caused by the optical system of the imaging means that generated the pair of captured images for areas of the distance image where the reliability is equal to or greater than a predetermined threshold value.

The present invention can reduce the memory bandwidth required to calculate distance information.

1 is a diagram showing a configuration of a distance measuring device according to a first embodiment. FIG. 2 is a diagram showing a processing flow in the first embodiment. FIG. 4 is a diagram showing an example of a received image in the first embodiment. 5A and 5B are diagrams illustrating an example of band information required for the distortion correction method according to the first embodiment. 5A to 5C are diagrams illustrating a distortion correction method according to the first embodiment. FIG. 11 is a diagram showing the configuration of a distance measuring device according to a second embodiment. FIG. 11 is a diagram showing a processing flow in the second embodiment. FIG. 11 is a diagram showing the configuration of a distance measuring device according to a third embodiment. FIG. 13 is a diagram illustrating a cache configuration according to the third embodiment. FIG. 13 is a diagram illustrating the reliability in the third embodiment. 13A to 13C are diagrams illustrating a distortion correction process according to the third embodiment. FIG. 13 is a diagram showing the configuration of a distance measuring device according to a fourth embodiment. 13A and 13B are diagrams illustrating an example of a distortion correction method and bandwidth information of additional processing according to the fourth embodiment. FIG. 13 is a diagram showing the configuration of a distance measuring device according to a fifth embodiment. FIG. 13 is a diagram illustrating an example of a processing flow in the fifth embodiment. 13A to 13C are diagrams illustrating an image shape transformation processing method according to a fifth embodiment. FIG. 13 is a diagram showing the configuration of a distance measuring device according to a sixth embodiment. FIG. 13 is a diagram showing another example of the configuration of a distance measuring device according to the sixth embodiment. FIG. 23 is a diagram showing an example of a processing flow in the sixth embodiment. FIG. 23 is a diagram illustrating an example of a processing flow according to a seventh embodiment.

Below, an embodiment of the distance measuring device of the present invention will be described with reference to the drawings. In the following embodiments, distance measuring methods will be described using a method in which multiple photodiodes are arranged for one distance measuring pixel and a method in which distance is calculated using a phase difference method and a method in which a stereo camera is used, but the distance measuring methods to which the present invention can be applied are not limited to the above. The present invention can be applied to all distance measuring devices in which distortion due to an optical system such as TOF occurs. In addition, in this embodiment, the captured image and distance image are stored in a DRAM (Dynamic Random Access Memory), but other memories may also be used. In that case, it is necessary to provide bandwidth information corresponding to the memory in which the distance image is stored as information on the memory bandwidth required for each distortion correction described later.

(Embodiment 1)
A distance measuring device according to a first embodiment of the present invention will be described. In the first embodiment, a method is applied in which a plurality of photodiodes are arranged at intervals of several μm for one distance measuring pixel, and a method of calculating distance using a phase difference method is applied to a pair of captured images having parallax generated by an imaging device having at least one common optical system. When two pairs of captured images are obtained from one optical system in this way, the distortion amounts of the two captured images are close to each other, so it is preferable to first generate a distance image from the two captured images and then perform distortion correction. In this way, it is only necessary to perform distortion correction on one distance image, so that the access bandwidth (DRAM bandwidth) of the DRAM for reading pixel values associated with distortion correction can be reduced compared to the case where distortion correction is performed on each of a pair of captured images.

Figure 1 is a hardware configuration diagram of a distance measuring device according to the first embodiment. Reference numeral 100 denotes the distance measuring device of this embodiment, which generates a distance image of a subject that has been subjected to distortion correction. Reference numeral 101 denotes an imaging unit, which captures an image of the subject and generates image data. Reference numeral 102 denotes a distance image generating unit, which receives image data from the imaging unit 101 and generates a distance image.

Next, the internal configuration of the imaging unit 101 will be described. Reference numeral 103 denotes a lens that collects light reflected from the subject and directs it to the imaging element 104, which forms an image on the imaging surface. Reference numeral 104 denotes the imaging element, which is an image sensor such as a CMOS sensor or a CCD sensor. The imaging element 104 converts the light received through the lens 103 into an electrical signal. Reference numeral 105 denotes an image transmission unit that transmits the electrical signal generated by the imaging element 104 to the distance image generation unit 102 as image data.

Next, the internal configuration of the distance image generating unit 102 will be described. 106 is an image receiving unit that receives image data transmitted from the imaging unit 101. 107 is an image correction unit that performs necessary correction processing on the image data received by the image receiving unit 106 as a pre-processing for generating a distance image, but does not include distortion correction for correcting distortion in the image data caused by the optical system of the imaging unit 101. 108 is a parallax calculation unit that calculates parallax for each pixel of the corrected image corrected by the image correction unit 107. 109 is a distance calculation unit that converts the parallax data calculated by the parallax calculation unit 108 into a distance image. 110 is a DRAM that stores the distance image generated by the distance calculation unit 109 in this embodiment. Here, the distance image generated by the distance calculation unit 109 and stored in the DRAM 110 is a distance image before distortion correction in this embodiment. In addition, it is assumed that information on the coordinate values of the pre-correction image corresponding to each pixel in the post-correction image in distortion correction (hereinafter referred to as coordinate conversion map information) is stored in advance in the DRAM 110. Reference numeral 111 denotes a reliability calculation unit, which calculates reliability for each arbitrary pixel unit in the corrected image corrected by the image correction unit 107. In FIG. 1, the reliability calculation unit 111 is configured to acquire image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without via the DRAM 110. Reference numeral 112 denotes an SRAM, which stores the reliability calculated for each pixel in this embodiment. The reliability may be calculated for each plurality of pixels instead of for each pixel, and the storage destination of the reliability does not necessarily have to be an SRAM but may be other memories. Reference numeral 113 denotes a memory access processing content selection unit, which in this embodiment selects a distortion correction method based on the reliability calculated by the reliability calculation unit 111 and the memory access bandwidth required for each preset distortion correction method. In this embodiment, the selection is made based on the DRAM bandwidth. Reference numeral 114 denotes a distortion correction necessity determination unit, which determines whether or not to perform distortion correction according to the reliability of the pixel of the pre-correction image corresponding to each pixel of the distortion-corrected image. At this time, the coordinate transformation map stored in advance in the DRAM 110 is referenced to obtain the coordinate values of the pixels of the pre-correction image corresponding to each pixel of the distortion-corrected image. 115 is a distortion correction unit that performs distortion correction of pixels that are determined to require distortion correction by the distortion correction necessity determination unit 114 using the method selected by the memory access processing content selection unit 113, and outputs the corrected pixel value. 116 is a mask unit that sets a predetermined value to pixels of the distortion-corrected image that are determined not to require distortion correction by the distortion correction necessity determination unit 114, and outputs the result.

Next, each part will be described in detail with reference to a flowchart. FIG. 2 is a flowchart showing the processing flow of the distance image generating unit 102 in the distance measuring device 100 of this embodiment. Note that the distance image generating unit 102 can be realized by an information processing device, and the components other than the DRAM 110 and SRAM 112 can be realized by one or more processors executing programs. The same applies to the distance image generating units of the other embodiments described below.

In S201, the image receiving unit 106 receives image data from the imaging unit 101. In this embodiment, it is assumed that an image like that shown in FIG. 3(a) is received.

In S202, the image correction unit 107 performs necessary correction processing other than distortion correction on the received image. Examples of correction processing include scratch correction when the sensor is scratched, and shading correction that corrects brightness unevenness that occurs when brightness decreases with distance from the center of the lens. However, the corrections performed here are not limited to the above.

In S203, the parallax calculation unit 108 calculates parallax for each pixel of the image corrected in S202. Here, a parallax calculation method called template matching will be described. It is assumed that the imaging unit 101 has an imaging element 104 with multiple photodiodes in one pixel, and one of the captured images (image A, image B) obtained from the multiple photodiodes in one pixel is used as a standard image, and the other is used as a reference image. Then, an area of any size centered on a point of interest in the standard image (called a matching area) is set, and the image of the matching area is moved sequentially in the reference image to identify the area with the highest similarity in the reference image. The distance between the point of interest in the matching area in the standard image and the center point of the area in the reference image with the highest similarity to the matching area is calculated as the parallax. Here, the size of the matching area and the number of movement steps (unit movement amount) of the matching area in the reference image can be set to any value. In addition, any method can be applied to obtain the similarity between images. For example, Sum of Squared Difference (SSD), which evaluates similarity using the sum of the squares of the differences in pixel values, and Sum of Absolute Difference (SAD), which evaluates similarity using the sum of the absolute values of the differences in pixel values, can be used.

In S204, the distance calculation unit 109 calculates the distance for each pixel based on the parallax calculated in S203. The distance is calculated from the parallax using the principle of triangulation, and a distance image before distortion correction is generated.

In S205, the reliability calculation unit 111 calculates the reliability for each predetermined pixel unit in the image corrected in S202. In this embodiment, the reliability is calculated in units of one pixel, but the reliability may be calculated in units of any pixel. In this embodiment, the reliability is calculated in one bit, but the reliability may be calculated in any number of bits. The reliability is calculated based on factors that affect the distance measurement accuracy, but in the calculation of distance using the phase difference method in this embodiment, the reliability is calculated based on the amount of change in contrast. The amount of change in contrast can be calculated from the variance of the pixel values of the pixel at the point of interest and its surrounding pixels. In this embodiment, the reliability calculation algorithm is adjusted so that the area 300 (area corresponding to the sky) in FIG. 3(b) is calculated as reliability 0 and the other areas as reliability 1 for the input image in FIG. 3(a). Here, the threshold is 1, and 0 below the threshold means low reliability, and 1 above the threshold means high reliability.

In this embodiment, the calculated reliability is stored in a static random access memory (SRAM) in association with the coordinate value so that it can be read out quickly, but it may be stored in other memory. In this embodiment, the area 300 with a reliability of 0 is approximately 14% of the entire image area.

In S206, the memory access process content selection unit 113 selects a distortion correction method based on the reliability calculated in S204 and the DRAM bandwidth required for each preset distortion correction method. In this embodiment, the distortion correction method is selected from three distortion correction methods, nearest neighbor, bilinear, and bicubic, depending on the proportion of areas with low reliability. However, the types of distortion correction methods are not limited to these. In addition, the method of providing information on the DRAM bandwidth required for each distortion correction method may be any method. For example, an input unit may be prepared for the user, or the information may be stored in a memory such as a DRAM or SRAM in advance. In this embodiment, bandwidth information such as that shown in the table in FIG. 4 is provided, but this format is merely an example, and any format may be used as long as the bandwidth required for each distortion correction method is known. The column 400 in the table shown in FIG. 4 indicates the type of distortion correction method.

Here, we will explain three distortion correction methods: nearest neighbor, bilinear, and bicubic. Figure 5(a) is a diagram explaining distortion correction.

500 represents an image after distortion correction, and now, suppose that correction is to be performed on the pixel of the point of interest 503. First, the coordinate values 504 (hereinafter referred to as pre-correction coordinate values) of the pixel of the pre-correction image corresponding to the point of interest 503 are referenced. In this embodiment, the pre-correction coordinate values 504 are obtained by referencing values corresponding to the post-correction coordinate values from those stored in advance in the DRAM 110 as a coordinate transformation map 502, but they may be stored in a memory other than the DRAM 110. The coordinate transformation map 502 is calculated in advance based on a coordinate transformation formula corresponding to the distortion aberration of the lens 103, but the method of obtaining the pre-correction coordinate values 504 is not limited to the above method. The method of obtaining the pre-correction coordinate values 504 may be a method of calculating the pre-correction coordinate values 504 one by one within the distance measuring device 100 using the above coordinate transformation formula.

501 represents the pre-correction image, and 505 (hereinafter referred to as the pre-correction coordinate position) represents the coordinate position of the pre-correction coordinate value 504 corresponding to the point of interest 503. However, the pre-correction coordinate value 504 calculated by the coordinate conversion formula generally includes a decimal point. Therefore, in the nearest neighbor method, as shown in FIG. 5B, the pixel value of pixel 508, which is the pixel closest to the pre-correction coordinate position 505 in the pre-correction image 501, is set as the post-correction pixel value at the point of interest 503. On the other hand, in the bilinear method, the value interpolated from the pixel values of the four pixels 506, 507, 508, and 509 surrounding the pre-correction coordinate position 505 in the pre-correction image 501 is set as the post-correction pixel value at the point of interest 503. Furthermore, in the bicubic method, the value interpolated from the pixel values of 16 pixels surrounding the pre-correction coordinate position 505 in the pre-correction image 501 is set as the post-correction pixel value of the point of interest 503. Therefore, with nearest neighbor, only one pixel value needs to be read from the DRAM 110, but with bilinear, four pixel values need to be read, and with bicubic, 16 pixel values need to be read, and the DRAM bandwidth increases depending on the type of distortion correction method. On the other hand, the accuracy of distortion correction is higher with bilinear than with nearest neighbor, and higher with bicubic than with bilinear.

Next, column 401 in FIG. 4 shows an example of the required bandwidth of the DRAM 110 for each distortion correction method. As described above, bilinear requires reading pixel values that are 4 times larger than nearest neighbor, and bicubic requires reading pixel values that are 16 times larger than nearest neighbor. If all of these pixel values are read from the DRAM 110 every time, the required bandwidth will be 4 times and 16 times larger, respectively. However, for example, in bilinear, some of the pixel values of the four pre-correction images 501 used in distortion correction of a pixel are likely to be referenced in distortion correction of the adjacent pixel. Therefore, a cache is generally provided for the DRAM 110, and pixel values read from the DRAM 110 once are temporarily stored in the cache for a specified period of time. As a result, when the pixel value of the same pixel is read again within a certain period of time, the DRAM bandwidth is reduced by reading from the cache without accessing the DRAM 110. The amount of bandwidth reduction varies depending on the cache hit rate. In this embodiment, it is assumed that the DRAM 110 is accessed in tile units (area units consisting of an arbitrary number of vertical and horizontal pixels), and the cache hit rate is estimated from the lens distortion aberration and the pixel area referenced by each distortion correction method. The required bandwidth 401 of the DRAM 110 shown in Figure 4 is an example of a value calculated based on such an estimation.

Column 402 shows the bandwidth increase rate of each distortion correction method when the nearest neighbor is used as a reference. Looking at 402, the bandwidth increase rate of bilinear is 12%, and the bandwidth increase rate of bicubic is 40%. Here, the memory access processing content selection unit 113 first refers to the reliability calculated in S204 and obtains the ratio of the area with a reliability of 0 to the entire image. Here, it is assumed that it is obtained as 14%. Although details will be described later, the area with a reliability of 0 is determined as a pixel that does not require distortion correction in a later step. Therefore, the fact that the reliability of the area is 0 is 14%, means that the number of pixels to be corrected for distortion can be reduced by 14%, and therefore the required bandwidth for reading pixel values from the DRAM 110 can be reduced by 14%. In other words, when the ratio of the area with a reliability of 0 is 14%, the required bandwidths for nearest neighbor, bilinear, and bicubic are 4.3 MB/Frame, 4.8 MB/Frame, and 6.0 MB/Frame, respectively.

Next, the bandwidth increase rate of each distortion correction method in FIG. 4 is referenced, and a distortion correction method that can be applied is determined using the reducible bandwidth. In this embodiment, the nearest neighbor's required bandwidth of 5.0 MB/Frame when performing distortion correction on the entire region is set as the maximum allowable bandwidth, and a distortion correction method in which the required bandwidth is equal to or less than the maximum allowable bandwidth (equal to or less than the maximum allowable access bandwidth) is adopted. As described above, when the bandwidth reduction rate is 14%, bilinear is 4.8 MB/Frame and bicubic is 6.0 MB/Frame, so the bilinear distortion correction method, which has the required bandwidth of 5.0 MB/Frame or less and is the most accurate distortion correction method, is selected. For example, when the bandwidth reduction rate is less than 10%, the required bandwidth of bilinear exceeds 5.0 MB/Frame, so the nearest neighbor distortion correction method is selected. Furthermore, if the bandwidth reduction rate is 28% or more, the required bandwidth for bicubic will be 5.0 MB/Frame or less, so the bicubic distortion correction method will be selected.

In this manner, in this embodiment, a distortion correction method is selected based on the DRAM bandwidth required by each distortion correction method and a predetermined maximum allowable bandwidth in areas excluding areas with low reliability as described above.

In S207, the distortion correction necessity determination unit 114 determines whether distortion correction is necessary for each pixel of the distance image generated in S204. In S207, first, as in the case of calculating the reliability described above, the pre-correction coordinate value 504 for the pixel of interest in the corrected image is read from the DRAM 110. Next, the reliability corresponding to the pixel closest to the pre-correction coordinate position 505 corresponding to the read-out pixel of interest is read from the SRAM 112. If the reliability is 0, it is determined that distortion correction is unnecessary and the process proceeds to S209, and if the reliability is 1, it is determined that distortion correction is necessary and the process proceeds to S208.

Next, in S208, the distortion correction unit 115 performs distortion correction on the pixel of interest for which it was determined in S207 that distortion correction is necessary, using the distortion correction method selected in S206. In this embodiment, the bilinear distortion correction method is selected, so distortion correction is performed bilinearly. At this time, the distortion correction unit 115 receives the pre-correction coordinate values 504 read from the DRAM 110 by the distortion correction necessity determination unit 114, and the distortion correction unit 115 does not need to read the pre-correction coordinate values 504 from the DRAM 110.

On the other hand, in S209, the mask unit 116 assigns a predetermined value to pixels determined in S207 not to require distortion correction without performing distortion correction. Since distortion correction is not performed on pixels determined not to require distortion correction, the process of reading pre-correction pixel values from the DRAM 110 for distortion correction is not required, and the DRAM bandwidth can be reduced. Furthermore, the DRAM bandwidth reduced from the allowable DRAM bandwidth, which is 5.0 MB/Frame in this embodiment, can be used for distortion correction in S208.

Next, in S210, the distortion correction unit 115 or the mask unit 116 outputs the distortion-corrected pixel value or the pixel value masked with a predetermined value. In this embodiment, the distortion correction unit 115 and the mask unit 116 are configured to output the distortion-corrected pixel value or the masked pixel value to the outside of the distance measuring device 100, but they may also be stored in a memory inside the distance measuring device 100, for example, in the DRAM 110.

In this way, steps S207 to S210 can generate and output a pixel value for one pixel in the distance image.

In S211, it is determined whether distortion correction or masking has been performed on all pixels in the distance image, and if not, the process returns to S207. By repeating these steps for all pixels, the distance image after distortion correction can be output.

Next, in S212, if image reception has not been stopped, for example based on user input, the process returns to S201 to receive the next image. The above process continues until image reception is stopped.

When two captured images are obtained from one optical system in this way, it is preferable to apply this embodiment to a configuration in which a distance image is first generated from the two captured images and then distortion correction is performed. In this way, by performing distortion correction only on one distance image instead of two captured images, the pixel values read out for distortion correction are halved, thereby reducing the DRAM bandwidth. In addition, since distortion correction processing is not performed on areas with low reliability, the DRAM bandwidth can be further reduced. As a result, the distortion correction method for the area to be corrected can be selected depending on the amount of bandwidth reduction. This allows the reduced bandwidth to be allocated to distortion correction in the area to be corrected, enabling more accurate distortion correction.

In addition, in this embodiment, the reliability has two values, 0 and 1, but it may have three or more values. For example, if the reliability has three values, 0, 1, and 2, distortion correction may not be performed for reliability 0, and distortion correction may be performed using different distortion correction methods for reliability 1 and 2.

(Embodiment 2)
In the first embodiment, a distance image is generated before distortion correction, and then distortion correction is performed on the distance image, whereas in the second embodiment, distortion correction is performed before generating the distance image, and then the distance image is generated. In the second embodiment, a distance measuring device using a stereo camera as an imaging device will be described. When capturing images using multiple separate optical systems arranged at intervals of several centimeters to several tens of centimeters, as in a stereo camera, the amount of distortion of the captured images differs for each optical system, so that a configuration in which distortion correction is performed on the multiple captured images before generating a distance image is preferable. Calculating parallax between captured images with different amounts of distortion results in a large error, so a highly accurate distance image can be obtained by first correcting the distortion of the captured images, then calculating the parallax and generating a distance image.

Figure 6 is a configuration diagram of a distance measuring device according to the second embodiment. Reference numeral 600 denotes a distance measuring device according to this embodiment, which generates a distance image of a subject. The distance measuring device 600 includes two image capturing units 601a and 601b, and a distance image generating unit 602. Note that the processing of the image capturing units 601a and 601b is the same as that of the image capturing unit 101 according to the first embodiment, so details are omitted.

The distance image generating unit 602 receives a pair of captured images consisting of image data a and image data b having a predetermined parallax from the two imaging units. The received captured image data is received by two image receiving units 606a, 606b, and then various image corrections, not including distortion correction, are performed by two image correcting units 607a, 607. The processing of the image receiving units 606a, 606b and the image correcting units 607a, 607 is the same as in the first embodiment, so details are omitted.

The two image correction units 607a, 607 obtain pre-distortion-corrected captured images a and b, which are then passed to the DRAM 610 and the reliability calculation unit 611. The two pre-distortion-corrected captured images a, b passed to the DRAM 610 are referenced during distortion correction. The DRAM 610 is also assumed to have coordinate transformation map information for distortion correction stored in advance. Here, since the amounts of distortion of the images obtained by the two imaging units are different, two coordinate transformation maps corresponding to the amounts of distortion of each captured image are stored.

On the other hand, the reliability calculation unit 611 calculates the reliability for each of the two received pre-distortion-corrected captured images. Note that the reliability calculation unit 611 is the same as the reliability calculation unit 111 in the first embodiment, so details are omitted. Data on each calculated reliability is stored in the SRAM 612.

The subsequent processing by the memory access processing content selection unit 613, the distortion correction necessity determination unit 614, the distortion correction unit 615, and the mask unit 616 is performed on two captured images. The processing performed by these units on each captured image is the same as that performed by the memory access processing content selection unit 113, the distortion correction necessity determination unit 114, the distortion correction unit 115, and the mask unit 116 in the first embodiment, so details are omitted.

Next, a detailed description of each unit will be given with reference to a flowchart. Figure 7 is a flowchart showing the processing flow of the distance image generating unit 602 in the distance measuring device 600 of this embodiment.

The steps other than S710 are arranged in a different order than in the flowchart of FIG. 2, but each step performs the same processing as the corresponding step. S701 and S702 correspond to S201 and S202, S703 to S709 correspond to S205 to S211, S711 and S712 correspond to S203 and S204, and S713 corresponds to S212. In this embodiment, S710 is added in order to perform distortion correction on the pre-distortion-corrected captured image a and the pre-distortion-corrected captured image b.

As in the first embodiment, by not performing distortion correction in areas with low reliability during these processes, it is possible to reduce the number of pixel values read out and the DRAM bandwidth, thereby enabling more accurate distortion correction within the limited maximum allowable bandwidth.

In this way, two distortion-corrected images are obtained by performing distortion correction on the captured images obtained from the two imaging units, and the parallax is obtained from these distortion-corrected images by the parallax calculation unit 617, and a distance image is obtained by the distance calculation unit 618. The processing contents of the parallax calculation unit 617 and the distance calculation unit 618 are the same as those of the parallax calculation unit 108 and the distance calculation unit 109 in the first embodiment, so details are omitted.

In this way, in a distance measuring device that uses multiple separate optical systems such as a stereo camera to acquire multiple captured images and generate a distance image, the amount of distortion in the captured images differs for each optical system, so it is preferable to perform distortion correction before generating the distance image as in this embodiment. In this way, parallax can be calculated between captured images that have been correctly distortion corrected, and a highly accurate distance image can be obtained. In addition, by not performing distortion correction in areas with low reliability, the DRAM bandwidth can be reduced, and the distortion correction method to be applied to the area to be corrected can be selected depending on the amount of bandwidth reduction. This allows the reduced bandwidth to be allocated to distortion correction in the area to be corrected, enabling more accurate distortion correction.

(Embodiment 3)
In the first embodiment, the reliability is calculated for each pixel, but in the third embodiment, one reliability is generated for each set of pixels according to the cache size. In this way, the cache hit rate can be improved and the DRAM bandwidth can be reduced. In addition, the size of the SRAM that stores the reliability can be reduced. In the third embodiment, similarly to the first embodiment, multiple photodiodes are arranged for one ranging pixel, and a method of calculating the distance using a phase difference method is applied. Therefore, a configuration is adopted in which distortion correction is performed after a distance image is generated. However, even in the configuration in which distortion correction is performed before a distance image is generated as in the second embodiment, one reliability can be generated for each set of pixels according to the cache size.

Figure 8 is a configuration diagram of the third embodiment. In the configuration diagram, a cache 817 has been added to the configuration of Figure 1. An example of the structure of the cache 817 will be explained using Figure 9. The cache 817 is configured to have multiple tile-shaped buffers 900 (hereinafter referred to as tile buffers). One tile buffer can hold data for a total of 16 pixels, 4 pixels vertically and 4 pixels horizontally, and cache replacement is performed in tile buffer units. 901 represents one pixel. Note that the size of 4 pixels vertically and 4 pixels horizontally is just an example, and the vertical and horizontal sizes and the number of tile buffers can be set to any number.

Next, the processing flow of this embodiment will be explained using the flow diagram in Figure 2.

Steps S201 to S204 are the same as in embodiment 1, so details are omitted.

Next, in S205, the reliability calculation unit 111 calculates the reliability for each pixel-sized (referred to as the tile buffer size) area of one tile buffer in the cache 817. In this embodiment, one reliability is calculated for an area of 4 pixels vertically and 4 pixels horizontally. There are various methods for calculating the reliability for each tile buffer size area, but in this embodiment, after all the reliability is calculated for each pixel, if there is even one pixel in the tile buffer size area with a reliability of 1, the reliability of that area is set to 1. An image of the reliability generated when the input image is as shown in FIG. 3(a) is shown in FIG. 10. A small square area 1000 in FIG. 10 is one tile buffer size area, and one reliability is generated for each of these areas. In this embodiment, the area surrounded by a thick line such as 1001 is an area with a reliability of 0. Here, by calculating the reliability for each tile buffer size, the amount of data for the reliability is reduced, and the SRAM size required to store the reliability can be reduced.

S206 is the same as in embodiment 1, so details are omitted.

Next, steps S207 to S209 will be explained together with reference to FIG. 11. 1100 is a post-distortion correction image, and first, distortion correction is performed on the focus point 1 (1101). The pre-correction coordinate value 1104 at the focus point 1 is read from the coordinate transformation map 1103. Then, the reliability of the tile buffer size area corresponding to the pre-correction coordinate value 1104 is read. Here, the reliability is assumed to be 1. If the nearest neighbor method is selected as the distortion correction method, an attempt is made to read from the DRAM 110 the pixel value on the pre-distortion correction image 1106 that is closest to the pre-distortion correction coordinate position 1107 corresponding to the pre-correction coordinate value 1104. At this time, if the corresponding pixel value is not stored in the cache 817, the data of the tile buffer size area 1109 including the corresponding pixel value is stored in the tile buffer of the cache 817, and the necessary pixel value is read from the tile buffer of the cache 817.

Next, distortion correction is performed for the focus point 2 (1102). Here, the reliability of the tile buffer size area corresponding to the pre-correction coordinate value 1105 at the focus point 2 is read, but since the focus point 2 is next to the focus point 1, the tile buffer size area corresponding to the focus point 1 and the focus point 2 may be the same. Here, it is assumed that the tile buffer size area corresponding to the focus point 1 and the focus point 2 is the same, and the reliability in the same tile buffer size area 1109 as the focus point 1 is read for the focus point 2, and a reliability of 1 is obtained. Since the reliability is 1, the distortion correction unit 115 attempts to read out the pixel value closest to the pre-correction coordinate position 1108 corresponding to the pre-correction coordinate value 1105 in order to perform distortion correction. Here, the reliability in the same tile buffer size area 1109 as the focus point 1 is referenced, so the pixel value to be read out is also in the same tile buffer size area 1109 as the focus point. Then, since the tile buffer size area 1109 is already in the tile buffer of the cache 817, the pixel values can be read from the cache 817 without accessing the DRAM 110 (a cache hit occurs).

The nearest neighbor distortion correction method was explained above, but bilinear and bicubic also increase the likelihood of a cache hit because pixel values near the pre-correction coordinate position are read together into the cache 817. In this way, by generating reliability for each tile buffer size area of the cache, the cache hit rate increases, and DRAM data can be read efficiently. In addition, by calculating reliability for each tile buffer size, the amount of reliability data is reduced, and the SRAM size required to store the reliability can be reduced.

In FIG. 8, the reliability calculation unit 111 is configured to acquire image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without going through the DRAM 110.

(Embodiment 4)
In the first to third embodiments, the DRAM bandwidth required for distortion correction is reduced by not performing distortion correction on pixels with a reliability of 0, and the reduced bandwidth can be used for distortion correction on pixels with high reliability. In the fourth embodiment, an example will be described in which the reduced bandwidth is used to perform additional processing to increase the added value of the distance measuring device.

Figure 12 is a configuration diagram of a distance measuring device showing the fourth embodiment. In Figure 1 (first embodiment), the distortion correction method was selected by the memory access processing content selection unit 113, and the selection result was passed to the distortion correction unit 115. On the other hand, in Figure 12 (fourth embodiment), the memory access processing content selection unit 1213 selects additional processing other than distortion correction, so there is no need to be connected to the distortion correction unit 115. Instead, the additional processing to be executed is selected based on the DRAM bandwidth required for the preset distortion correction, the DRAM bandwidth required for the additional processing selected by the memory access processing content selection unit 1213, and the proportion of the area with a reliability of 0.

Figure 13(a) shows bandwidth information required for the distortion correction method used. In this embodiment, a bilinear distortion correction method is used, but other distortion correction methods may be used. Next, the percentage of the area stored in the SRAM with a reliability of 0 is calculated. Here, it is assumed that the area with a reliability of 0 is 23%. This makes it possible to reduce the bandwidth by 23% of the DRAM required bandwidth (1301) of 5.6 MB/Frame for bilinear distortion correction, i.e., 1.3 MB/Frame.

Next, information on the additional processes selected by the memory access process content selection unit 1213 is shown in FIG. 13(b). Here, two types of additional processes, object recognition process and object tracking process, are written as the additional processes 1302, and the required DRAM bandwidth 1303 for these processes is 1 MB/Frame and 3 MB/Frame, respectively.

Next, additional processing that can be performed with the reducible bandwidth is selected. In this embodiment, since a bandwidth reduction of 1.3 MB/Frame is possible, only object recognition is selected. If a bandwidth reduction of 4 MB/Frame or more is possible, both object recognition and object tracking are selected.

In this embodiment, there are only two types of candidate additional processes to select, but there may be three or more types. Any algorithm may be used to determine which additional process to select. Any method may be used to provide the distortion correction method and DRAM bandwidth information required for the additional process. For example, a user input unit may be provided so that such information can be obtained from user input, or the information may be stored in advance in a memory such as a DRAM or SRAM. In this embodiment, the bandwidth information is provided in the format shown in Figures 13(a) and (b), but this format is merely an example and another format may be used.

In this way, the object recognition process is selected according to the amount of bandwidth reduction. By combining the object recognition results with the generated distance image, it is possible to link a specific object (person, animal, vehicle, etc.) with information about the distance to that object. In addition, by handling the acquired distance information and recognition results in chronological order across multiple frames, it is possible to calculate the trajectory and speed of the object.

As explained above, by not performing distortion correction in areas with low reliability, DRAM bandwidth can be reduced, and the reduced bandwidth can be used to perform additional processing to increase the added value of the ranging device.

In FIG. 12, the reliability calculation unit 111 is configured to acquire image data via the DRAM 110, but the image data may be acquired directly from the image correction unit 107 without going through the DRAM 110.

(Embodiment 5)
In the first to fourth embodiments described above, the DRAM bandwidth required for distortion correction is reduced by not performing distortion correction on pixels with a reliability of 0 or pixels with a low reliability when the reliability is multi-valued.

In contrast, in the fifth embodiment, the DRAM bandwidth used is reduced by using reliability-based control even in processes that involve DRAM access other than distortion correction, such as enlarging or reducing the display area, cropping out a partial area, and displaying multiple areas on a multi-screen.

Figure 14 is a diagram showing the configuration of a distance measuring device according to embodiment 5. Functions that overlap with the previous embodiment are given the same reference numerals.

Reference numeral 1400 denotes a distance measuring device according to this embodiment, which displays a specified image area and generates a distance image of the subject with distortion correction. Reference numeral 1401 denotes a distance image generating unit, which receives image data from the imaging unit 101 and generates a distance image. The distance image generating unit 1401 can output an image (hereinafter referred to as a display image) that has been subjected to processing such as cropping and enlargement/reduction as well as distortion correction on the distance image so that a specified image area of the distance image is output.

The internal configuration of the distance image generation unit 1401 will be described. The image correction unit 107 performs necessary correction processing on the image data received by the image receiving unit 106. For example, it performs correction processing for uneven brightness due to peripheral light loss that occurs when passing through the lens 103, and processing to correct characteristic variations such as uneven sensitivity for each pixel of the image sensor 104. The parallax calculation unit calculates parallax for each pixel of the image corrected by the image correction unit 107. 109 is a distance calculation unit that converts the parallax data calculated by the parallax calculation unit 108 into a distance image. 110 is a DRAM that stores the distance image generated by the distance calculation unit 109.

1402 is a reliability calculation unit that calculates reliability for each arbitrary pixel unit in the image from the image correction unit 107. 112 is an SRAM that stores the reliability calculated for each pixel in this embodiment. The reliability may be calculated for multiple pixels instead of for each pixel, and the reliability does not necessarily have to be stored in an SRAM but may be stored in other memory.

1403 is a distance image readout determination unit. It references information relating to the cropping position and the degree of magnification/reduction of the distance image corresponding to the display image (hereinafter referred to as display area information) and generates coordinate values of the distance image corresponding to each pixel of the display image. It also determines whether to read the distance image from the DRAM 110 according to the display area information and the reliability of each pixel of the distance image. When performing distortion correction, the distance image readout determination unit 1403, as in the other embodiments described above, references a coordinate transformation map that associates the coordinate values of the images before and after distortion correction and generates coordinate values of the pre-distortion-corrected image corresponding to each pixel of the post-distortion-corrected image.

1404 is an image shape transformation unit, which performs image shape transformation processing on pixels for which the distance image read determination unit 1403 has determined that distance image reading is necessary, and outputs the pixel value after image shape transformation. The image shape transformation processing refers to the coordinate values from the distance image read determination unit 1403 and reads the corresponding pixel value of the distance image from the DRAM 110. The pixel value may be calculated using an interpolation method such as bilinear or bicubic using surrounding pixel values, but in the case of image cropping without scaling, the distance image may be read directly according to the coordinate values. Note that the image shape transformation processing performed by the image shape transformation unit 1404 includes distortion correction in addition to cropping and scaling.

1405 is a mask unit that sets a predetermined value to the pixels of the display image and outputs it when the distance image readout determination unit 1403 determines that reading of the distance image is unnecessary.

The hardware configuration of the distance measuring device according to embodiment 5 has been described above using FIG. 14.

The same configuration can be used to perform distortion correction as in the above-described embodiment. In this case, a coordinate transformation map that associates coordinates before and after distortion correction is stored in advance in the DRAM 110, and coordinate values of pixels in the pre-distortion-corrected image that correspond to each pixel in the post-distortion-corrected image are generated based on the coordinate transformation map. Below, the process flow for performing image shape transformation processing including distortion correction using the coordinate transformation map and display area information is explained with reference to a flowchart.

Figure 15 is an example of a flowchart showing the processing flow of the distance image generating unit 1401 in the distance measuring device 1400 of this embodiment. Functions that overlap with the previous embodiment are given the same reference numerals.

Steps S201 to S204 are the same as in the embodiment described above, so the explanation is omitted.

In S1501, the reliability calculation unit 1402 calculates the reliability for each predetermined pixel unit in the image corrected in S202. In this embodiment, the reliability is calculated in units of one pixel, but the reliability may be calculated in units of any pixel. In this embodiment, the reliability is calculated in one bit, but the reliability may be calculated in any number of bits. The reliability is calculated based on factors that affect the distance measurement accuracy, but in the calculation of distance using the phase difference method in this embodiment, the reliability is calculated based on the amount of change in contrast. The amount of change in contrast can be calculated from the variance of the pixel values of the point of interest and its surrounding pixels. The calculated reliability is associated with the coordinates of the image before distortion correction and stored in an SRAM or the like. The processes of S1501 and S203 to S204 are independent and may be processed in parallel, or S1501 may be performed before the process of S203, and the calculated reliability may be stored in the SRAM in advance.

In S1502, the distance image readout determination unit 1403 references the display area information and generates coordinate values of the distortion-corrected image corresponding to each pixel of the display image.

In S1503, the distance image read determination unit 1403 refers to the coordinate transformation map and reads from the DRAM 110 the coordinate values of the pre-distortion-corrected image that correspond to the coordinate values of the post-distortion-corrected image generated in S1502.

Here, steps S1502 and S1503 will be described with reference to FIG. 16. FIG. 16 is a diagram for explaining a method for acquiring the coordinate values of the pre-distortion-corrected image from the coordinate values of the display image 1600 via the post-distortion-corrected image 1601 in steps S1502 and S1503 of the image shape transformation process in embodiment 5. The display image 1600 is displayed by cutting out two image areas (image cut-out area A 1603 and image cut-out area B 1604) from the post-distortion-corrected image 1601.

The display area information is information that specifies the coordinate values of each cut-out area of the distortion-corrected image 1601 in the display image 1600, and may have coordinate values for each pixel of the display image 1600, or may be information that indicates a rectangular area. For example, if the rectangular area is a rectangle, it may be configured to have the start coordinates (X1, Y1) of the upper left vertex and the end coordinates (X2, Y2) of the lower right vertex of the rectangle.

The distance image read determination unit 1403 first refers to the display area information in S1502 and obtains a post-distortion-corrected coordinate position 1606 corresponding to the pixel of interest 1605 in the display image 1600. Next, the distance image read determination unit 1403 refers to the coordinate transformation map in S1503 and generates a pre-distortion-corrected coordinate position 1607 corresponding to the post-distortion-corrected coordinate position 1606. In this way, the distance image read determination unit 1403 can obtain the pre-distortion-corrected coordinate position 1607 corresponding to the pixel of interest 1605 in the display image 1600, i.e., the coordinate value of the pre-distortion-corrected image.

In S1504, the reliability corresponding to the pixel closest to the read pre-distortion-correction coordinate position 1607 is read from the SRAM 112. If the reliability is 0, it is determined that reading of the distance image is not necessary, and the process proceeds to S1506. If the reliability is 1, it is determined that reading of the distance image is necessary, and the process proceeds to S1505.

Next, in S1505, the image shape transformation unit 1404 reads out the pixel value of the pre-distortion correction coordinate position 1607 on the pre-distortion correction image for which it was determined in S1504 that reading of the distance image is necessary. Then, image shape transformation processing is performed in which the pixel value read out by the image shape transformation unit 1404 is set as the pixel value of the point of interest 1605 on the display image.

On the other hand, in S1506, the image shape transformation unit 1404 does not read the pixel value of the pre-distortion correction coordinate position 1607 on the pre-distortion correction image for which it was determined in S1504 that reading of the distance image is not necessary, and does not perform image shape transformation processing. Instead, the mask unit 1405 performs mask processing to assign a predetermined value to the pixel of the point of interest 1605 on the display image.

In S211, it is determined whether pixel values have been assigned to all pixels in the display image 1600 through image shape transformation processing or mask processing, and if there are any pixels to which no pixel value has been assigned, the process returns to S1502.

By repeating these steps for all pixels of the display image, a display image can be output that has been subjected to distortion correction, image cropping, and enlargement/reduction processing.

In S212, if image reception has not been stopped, for example based on user input, the process returns to S201 to receive the next image and executes the processes from S202 onward.

As described above, according to this embodiment, in addition to distortion correction processing, it is possible to reduce the DRAM bandwidth used by controlling using reliability even in processing that involves DRAM access, such as enlarging or reducing the display area, cutting out a partial area, and displaying multiple areas on a multi-screen.

(Embodiment 6)
In the sixth embodiment, the DRAM bandwidth used is reduced in the process of detecting a predetermined subject from a captured image and outputting a distance image to the subject.

Figure 17 is a diagram showing the configuration of a distance measuring device according to embodiment 6. Functions that overlap with those in the previous embodiment are given the same reference numerals and will not be described.

1702 denotes an object detection processing unit, which receives as input one of a pair of captured images output from the image correction unit 107, performs subject recognition and identification processing, such as character recognition of vehicle numbers and signs, and shape recognition of parts, to obtain object position information. There are various methods for recognition and identification processing. It may be a method of calculating the degree of similarity by comparing the pattern with a previously stored object, or it may be possible to estimate the object using machine learning that finds and estimates correlations from a large amount of data, or a multi-layered algorithm such as a neural network.

The object position information generated by the object detection processing unit 1702 is information relating to coordinates indicating the image area corresponding to the subject to be recognized and identified. 1703 is an SRAM, which in this embodiment stores object position information calculated for each pixel. The object position information may not be for each pixel, but may be information indicating a rectangular area including the subject. For example, if the rectangular area is a rectangle, the start coordinates (X1, Y1) of the upper left vertex of the rectangle and the end coordinates (X2, Y2) of the lower right vertex may be stored for each rectangular area. The object position information does not necessarily have to be stored in an SRAM, but may be stored in other memories.

1704 is a distance image readout determination unit. Depending on the above-mentioned display area information, object position information for the subject to be identified, and the reliability, if there is an image area in the display area that corresponds to the subject to be recognized and identified, and the reliability is 1, it is determined that reading of the display area in the distance image is necessary. If there is no image area in the display area that corresponds to the subject, or if the reliability is 0, it is determined that reading of the distance image is not necessary.

Next, as another example of this embodiment, a configuration is shown in which object detection is performed using the distortion-corrected image, and a distance image of the image area portion corresponding to the detected subject is output. Figure 18 shows another configuration of a distance measuring device according to embodiment 6, in which an image shape transformation unit for distortion correction is added in front of the object detection processing unit, separate from the image shape transformation unit 1404. Functions that overlap with the embodiment shown in Figure 17 above are given the same reference numerals, and descriptions are omitted.

1803 is a DRAM that stores image data of one of a pair of captured images output from the image correction unit 107. In this embodiment, of image data a and image data b having parallax, image data a is used, and the corrected image obtained by correcting image data a by the image correction unit 107 is referred to as image data A.

1804 is an image shape transformation unit that references a coordinate transformation map stored in advance in DRAM 1803 to generate coordinate values of pixels in the pre-correction image that correspond to each pixel in the distortion-corrected image. The image shape transformation unit 1804 corrects the distortion of image data A according to the coordinate transformation map to generate distortion-corrected image data A'.

1802 is an object detection processing unit, which uses image data A' after distortion correction to perform subject recognition and identification processing, such as character recognition of vehicle numbers and signs, and part shape identification, to obtain object position information. The recognition and identification processing methods are as described above. The object position information may be stored in an SRAM (not shown).

The same DRAM may be used for the DRAM 1803 that stores the image data A before distortion correction and the DRAM 110 that stores the distance image. In this case, the same coordinate transformation map can be referenced. Also, coordinate transformation map information in which coordinate values are set in advance to resize the image data to the size handled by the object detection processing unit 1802 may be stored in the DRAM 1803.

1805 is a distance image readout determination unit. It refers to the object position information generated by the object detection processing unit 1802 and the reliability calculated by the reliability calculation unit 1402, and if the image area corresponds to the subject to be recognized and identified and the reliability is 1, it determines that it is necessary to read out the pixel value of the coordinates indicated by the object position information on the distance image.

Figure 19 is a flowchart showing an example of the process from distance image read determination by the distance image read determination unit 1805 in the distance measuring device 1800 of this embodiment to image shape deformation processing of the distance image.

In S1901, the object position information generated by the object detection processing unit 1802 is obtained. The object position information is information indicating the coordinates of the image area corresponding to the object. The object position information may be read from an SRAM (not shown) with 1 bit indicating the presence or absence of an object for each pixel (1 if the object is present, 0 if not), or may be information indicating a rectangular area including the image area corresponding to the object. For example, if the rectangular area is a rectangle, the information has the start coordinates (X1, Y1) of the upper left vertex of the rectangle and the end coordinates (X2, Y2) of the lower right vertex.

In S1902, the coordinate transformation map of the distance image is referenced from DARM110, a pixel of interest is selected from among the pixels corresponding to the coordinates indicated by the object position information, and a pre-distortion-corrected coordinate position corresponding to the selected pixel of interest is generated.

In S1903, the reliability corresponding to the pixel closest to the read pre-distortion correction coordinate position is read from the SRAM 112. If the reliability is 0, it is determined that reading of the distance image is unnecessary, and the process proceeds to S1907. If the reliability is 1, it is determined that reading of the pixel value of the pixel of interest on the distance image is necessary, and the process proceeds to S1906.

Next, in S1906, for a pixel of interest in the distance image that was determined in S1903 to require reading, the image shape transformation unit 1404 reads out the corresponding pixel value in the pre-distortion corrected image and performs image shape transformation processing. On the other hand, in S1905, for a pixel that was determined in S1903 not to require reading of the distance image, the mask unit 1405 performs mask processing to assign a predetermined value to the pixel, without reading out the pre-distortion corrected image and performing image shape transformation processing.

In S1908, the pixel values of the distance image processed in S1906 or S1907 are output. By repeatedly performing the processing flow described above using FIG. 19 for all coordinates indicated by the object position information, a distance image is output for only the subject to be identified.

As described above, according to this embodiment, it is possible to reduce the DRAM bandwidth used in the process of detecting an image area corresponding to a subject within a specified display area and outputting a distance image for only that subject. This is because this embodiment can limit the pixel values of the distance image read from the DRAM by using the object position information and reliability of the subject.

Even if the object position information of the subject is obtained from outside the distance measuring device, the pixel values of the distance image read from the DRAM can be limited by using the object position information of the subject and the reliability of each pixel as described above. In this embodiment, the object position information is generated, but the object position information may be generated by an external device. For example, the captured image may be output from the image transmission unit 105 or the image correction unit 107 to an external device (such as a GPU), and the distance image read determination units 1705 and 1805 may obtain the detection result of the image area corresponding to the subject on the captured image in the external device as object position information.

(Embodiment 7)
In embodiment 7, the DRAM bandwidth used is reduced in the process of synthesizing two distance images generated by two ranging devices, in which the ranging areas overlap, i.e., two distance images generated based on captured images in which the imaging areas overlap, and outputting a single synthetic distance image.

Figure 20 is a diagram showing the configuration of a distance measuring device according to embodiment 7.

2001 is a first imaging unit equipped with a lens and an image sensor, and outputs image data generated by the image sensor.

2002 is a first captured image processing unit, which receives and processes image data from the first capturing unit 2001, and has an image correction unit, a parallax calculation unit, and a distance calculation unit. The first captured image processing unit 2002 generates a distance image (distance image 1) using the captured image data, and writes distance image 1 to the DRAM 2007.

2003 is a first reliability calculation unit, which calculates the reliability of the distance (reliability 1) based on the image data after image correction from the first captured image processing unit. The method of calculating the reliability is as described in the above embodiment.

2004 is a second imaging unit equipped with a lens and an image sensor, which outputs image data generated by the image sensor.

2005 is a second captured image processing unit, which receives and processes image data from the second capturing unit 2004, and has an image correction unit, a parallax calculation unit, and a distance calculation unit. The second captured image processing unit 2005 generates a distance image (distance image 2) using the captured image data, and writes distance image 2 to the DRAM 2007.

2006 is a second reliability calculation unit, which calculates the reliability of the distance (reliability 2) based on the image data after image correction from the second captured image processing unit. The method of calculating the reliability is as described in the above embodiment.

2008 is a distance image readout determination unit. It generates coordinate values of distance image 1 and distance image 2 corresponding to each pixel of the display image. Specifically, it generates coordinate values of each pixel by referring to a coordinate transformation map that associates coordinates of distance images before and after distortion correction corresponding to distance image 1 and distance image 2, which are stored in advance in DRAM 2007. The distance image readout determination unit 2006 obtains the reliability corresponding to the pixel closest to the generated coordinate value from each of the first reliability calculation unit and the second reliability calculation unit.

If there is an overlapping area between distance image 1 and distance image 2, the distance image read determination unit 2008 determines the magnitude of each reliability (high or low reliability) and instructs the distance information synthesis unit 2009 to read the pixel value of the distance image with the higher reliability. If both reliability levels are 0, it is determined that reading of the distance image is unnecessary, and the mask unit 2010 performs masking processing (setting a predetermined value to the pixel of the display image and outputting it). Note that if each reliability level is the same value other than 0, it may be instructed to read the pixel value of either distance image. However, in such a case, the effect of optical distortion can be reduced by reading the pixel value of the distance image whose coordinate value of the pixel whose reliability has been calculated is closer to the optical axis center of the first imaging unit 2001 or the second imaging unit 2004.

The distance information synthesis unit 2009 reads out the distance image generated by either the first imaging unit 2001 and the first captured image processing unit 2002, or the second imaging unit 2004 and the second captured image processing unit 2005, in response to an instruction from the distance image readout determination unit 2008. Note that the distance information synthesis unit 2009 may perform an interpolation process to improve image quality, as in the above-described embodiment.

As described above, a hardware configuration of a distance measuring device according to embodiment 7 has been illustrated as having two image capturing units 2001 and 2004 as shown in FIG. 20, but the same can be applied to a case where three or more image capturing units are used.

As described above, according to this embodiment, when distance images from multiple distance-measuring cameras are synthesized and displayed, the pixel values of the distance images read from the DRAM are limited based on the reliability of each pixel, thereby reducing the DRAM bandwidth used. When synthesizing distance images, it is also possible to perform distortion correction to correct optical distortion, which similarly reduces the DRAM bandwidth used. Note that by performing distortion correction on overlapping areas of multiple distance images to be synthesized only on the distance image with the highest reliability, the DRAM bandwidth used can be reduced compared to when distortion correction is performed on all distance images.

Other Examples
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

In the above embodiment, the distance measurement methods described are a method in which multiple photodiodes are arranged for one distance measurement pixel and the distance is calculated using a phase difference method, and a method using a stereo camera. However, the present invention is not limited to these distance measurement methods, and can also be applied to shape deformation processing when visualizing and displaying distance information obtained by TOF, millimeter wave radar, LiDAR (Light Detection and Ranging), etc.

100 Distance measuring device 101 Imaging unit 102 Distance image generating unit 110 DRAM
111 Reliability calculation unit 115 Distortion correction unit 1404 Image shape transformation unit 1702 Object detection processing unit

Claims (19)

A generating device for generating a distance image representing a distance to a subject,
a generating means for generating the distance image based on a pair of captured images having parallax;
a correction means for correcting distortion in the distance image caused by an optical system of an imaging means that generated the pair of captured images, for an area in the distance image where the reliability is equal to or greater than a predetermined threshold value;
A generating device comprising: A generating device for generating a distance image representing a distance to a subject,
a correction means for correcting distortion in a pair of captured images having parallax, the distortion being caused by an optical system of an imaging means that generated the pair of captured images, for an area in which the reliability is equal to or greater than a predetermined threshold value;
a generating means for generating a distance image based on the pair of captured images whose distortion has been corrected by the correcting means;
A generating device comprising: 3. The generating device according to claim 1, wherein the correction means calculates the reliability based on pixel values of a pixel of interest and pixels surrounding the pixel of interest. The generating device according to claim 1 , wherein the reliability is a change in contrast calculated based on a variance of pixel values of a pixel of interest and pixels surrounding the pixel of interest. The generating device according to claim 1 , wherein the correction means applies different predetermined distortion correction methods depending on the proportion of the area in which the reliability is less than the predetermined threshold. The generating device according to claim 1 , further comprising a masking unit that sets a predetermined value to pixels in an area in which the reliability is less than the predetermined threshold value. A generating device as described in any one of claims 1 to 6, characterized in that it further comprises a selection means for selecting a process that involves access to the storage means that is below a predetermined access bandwidth, based at least on a predetermined access bandwidth to the storage means that stores the distance image and the proportion of areas where the reliability is above a predetermined threshold. the selection means selects one distortion correction method based on a ratio of an area in which the reliability is less than a predetermined threshold and an access bandwidth to the storage means required for each of a plurality of predetermined distortion correction methods;
The generating device according to claim 7 , wherein the correction means applies the distortion correction method selected by the selection means. The generating device described in claim 8, characterized in that the selection means selects one from the plurality of specified additional processes based on the specified access bandwidth, the percentage of the area where the reliability is below a specified threshold, the access bandwidth to the storage means required by the selected distortion correction method, and the access bandwidth to the storage means required by the plurality of specified additional processes. The generating device according to claim 8 or 9, characterized in that the selection means selects one distortion correction method from the plurality of predetermined distortion correction methods in which different pixels are referenced when interpolating pixel values in distortion correction. the storage means includes a buffer capable of storing pixel values of a predetermined number of pixels, and a cache means for replacing pixel values in units of the buffer;
10. The generating device according to claim 7, wherein the correction means calculates a reliability for each pixel corresponding to the buffer, and acquires pixel values for which distortion correction is performed from the cache means. a designation unit for designating a part of the distance image as a cut-out region,
The generating device according to claim 1 , wherein the correction means corrects the distortion based on the cut-out region and a region in which the reliability is equal to or higher than a predetermined threshold value. The generating device according to claim 12, characterized in that the cut-out area is an image area corresponding to a specified object in the captured image. a synthesis unit for synthesizing a plurality of distance images having overlapping areas, the distance images being generated based on a plurality of pairs of captured images having overlapping imaging areas, to generate a synthetic distance image,
the synthesis means uses, as a pixel value of an area of the synthetic distance image corresponding to the overlapped area, a pixel value of the distance image having the highest reliability of the overlapped area among pixel values of the plurality of distance images in the overlapped area.
A generating device according to any one of claims 1 to 13. the correction means corrects distortion of the distance image having the highest reliability in the overlapping region among the regions in each of the plurality of distance images in which the reliability is equal to or greater than a threshold value;
15. The generating device according to claim 14. A method for generating a distance image representing a distance to a subject, comprising the steps of:
a generating step of generating the distance image based on a pair of captured images having parallax;
a correction step of correcting distortion in the distance image caused by an optical system of an imaging means that generated the pair of captured images, for an area in the distance image where the reliability is equal to or greater than a predetermined threshold value;
A method for generating a product comprising the steps of: A method for generating a distance image representing a distance to a subject, comprising the steps of:
a correction step of correcting distortion in a pair of captured images having parallax, the distortion being caused by an optical system of an imaging unit that generated the pair of captured images, for an area in which the reliability is equal to or greater than a predetermined threshold value;
a generating step of generating a distance image based on the pair of captured images whose distortion has been corrected by the correcting step;
A method for generating a product comprising the steps of: On the computer,
A method for generating a distance image representing a distance to a subject, comprising the steps of:
a generating step of generating the distance image based on a pair of captured images having parallax;
a correction step of correcting distortion in the distance image caused by an optical system of an imaging means that generated the pair of captured images, for an area in the distance image where the reliability is equal to or greater than a predetermined threshold value;
A program for executing each step of the method. On the computer,
A method for generating a distance image representing a distance to a subject, comprising the steps of:
a correction step of correcting distortion in a pair of captured images having parallax, the distortion being caused by an optical system of an imaging unit that generated the pair of captured images, for an area in which the reliability is equal to or greater than a predetermined threshold value;
a generating step of generating a distance image based on the pair of captured images whose distortion has been corrected by the correcting step;
A program for executing each step of the method.

Images