JP6477193B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method.

  An imaging apparatus such as a stereo camera has two imaging elements that capture an image of a subject via different lenses, and measures the distance to the subject using the principle of triangulation. For example, using two moving images taken by a stereo camera, the position of the subject in real space is calculated from the parallax of the subject between the two moving images, and the speed of the subject in real space is calculated from the time change of the calculated position. A technique has been proposed (see, for example, Patent Document 1).

  An imaging apparatus using a CMOS (Complementary Metal Oxide Semiconductor) image sensor images a subject by sequentially shifting the exposure time of each row of a pixel array in which a plurality of pixels are arranged in a matrix. For this reason, when a moving subject is picked up by an image pickup apparatus using a CMOS image sensor, a so-called rolling shutter distortion in which the subject image is distorted may occur.

  On the other hand, there has been proposed an imaging device that detects a movement and a moving direction of a subject from a difference in rolling shutter distortion with respect to the subject in two images taken with the scanning order of rows in two imaging elements reversed (for example, for example) , See Patent Document 2).

JP 2008-015815 A JP 2011-205215 A

  When calculating the speed of a moving subject using two moving images captured by a stereo camera, at least two frames (four frames in total) are used in each of the two moving images. That is, in order to calculate the speed of the subject, in a storage device such as a memory, a part of the storage area is occupied by at least four frames.

  In one aspect, an image processing apparatus and an image processing method disclosed in the present invention are intended to reduce the amount of image data used for calculating the speed of a subject.

  An image processing apparatus according to one aspect includes a first image sensor and a second image sensor in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted. Is read out from the first image sensor in an image processing apparatus that calculates the speed of the subject from an image captured by an image capturing apparatus in which the first image sensor and the second image sensor are opposite to each other. And the second subject read out from the second image sensor and the position of the first subject that is the subject on the first image and the second subject that is the subject on the second image A first calculation unit that calculates a parallax indicating a difference from the position of the first, a second calculation unit that calculates distortion information that is a difference in shape between the first subject and the second subject, and parallax and distortion information And a third calculation unit for calculating the speed of the subject. That.

  An image processing method according to another aspect includes a first image sensor and a second image sensor in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted, and the plurality of rows In the image processing method for calculating the speed of a subject from an image captured by an imaging device in which the first imaging device and the second imaging device are opposite to each other, the first read from the first imaging device And the second subject read out from the second image sensor and the position of the first subject that is the subject on the first image and the second subject that is the subject on the second image Parallax indicating a difference from the position of the first subject is calculated, distortion information which is a difference in shape between the first subject and the second subject is calculated, and the velocity of the subject is calculated using the parallax and the distortion information.

  The image processing apparatus and the image processing method disclosed in the present invention can reduce the amount of data of an image used for calculating the speed of a subject.

It is a figure which shows one Embodiment of an image processing apparatus and an image processing method. It is a figure which shows another embodiment of an image processing apparatus and an image processing method. FIG. 3 is a diagram illustrating an example of a timing at which a pixel signal is read from the image sensor illustrated in FIG. 2. FIG. 4 is a diagram illustrating an example of an image captured by the image sensor illustrated in FIG. 2 based on the timing diagram illustrated in FIG. 3. It is a figure which shows an example of the calculation process of the parallax in the parallax calculation part shown in FIG. It is a figure which shows an example of the arithmetic processing in the distortion calculation part shown in FIG. It is a figure which shows an example of the image process in the image processing apparatus shown in FIG. It is a figure which shows another embodiment of an image processing apparatus and an image processing method. It is a figure which shows an example of the image image | photographed based on the timing diagram which the imaging device shown in FIG. 8 showed in FIG. It is a figure which shows an example of the arithmetic processing in the distortion calculation part shown in FIG. It is a figure which shows an example of the image process in the image processing apparatus shown in FIG. It is a figure which shows another embodiment of an image processing apparatus and an image processing method. It is a figure which shows an example of the image image | photographed based on the timing diagram which the imaging device shown in FIG. 12 showed in FIG. It is a figure which shows an example of the arithmetic processing in the distortion calculation part shown in FIG. It is a figure which shows an example of the image process in the image processing apparatus shown in FIG. It is a figure which shows another embodiment of an image processing apparatus and an image processing method. It is a figure which shows an example of the image process in the image processing apparatus shown in FIG. FIG. 18 is a diagram showing a continuation of the image processing shown in FIG. 17. It is a figure which shows another embodiment of an image processing apparatus and an image processing method. It is a figure which shows an example of the image image | photographed based on the timing diagram shown in FIG. 3 by the imaging device shown in FIG. It is a figure which shows an example of the arithmetic processing in the distortion calculation part shown in FIG. It is a figure which shows an example of the image process in the image processing apparatus shown in FIG.

  Hereinafter, embodiments will be described with reference to the drawings.

  FIG. 1 shows an embodiment of an image processing apparatus and an image processing method. The dashed arrows in FIG. 1 indicate the flow of information such as signals. The image processing apparatus 100 illustrated in FIG. 1 is connected to an imaging device CAM such as a stereo camera that simultaneously images the subject OBJ from different positions via a storage unit MEM. The storage unit MEM is a frame memory or the like, and stores an image captured by the imaging device CAM. Note that the storage unit MEM may be disposed in the imaging device CAM, or may be disposed in the image processing device 100. Further, the image processing apparatus 100 and the storage unit MEM, or the storage unit MEM and the imaging device CAM may be connected by wire or may be connected wirelessly.

  The imaging device CAM includes lenses LEN1 and LEN2, an imaging element ISEN1 that images the subject OBJ via the lens LEN1, an imaging element ISEN2 that images the subject OBJ via the lens LEN2, and a drive unit DRV that drives the imaging elements ISEN1 and ISEN2. Have

  The imaging elements ISEN1 and ISEN2 are CMOS image sensors or the like, and have a pixel array in which a plurality of pixels are arranged in a matrix. The drive unit DRV performs control for sequentially shifting the start time of exposure (scanning) of each row of the pixel array for each of the image sensors ISEN1 and ISEN2, and sequentially reads out pixel signals obtained by exposure for each row. To run. The images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2 are output to the storage unit MEM.

  In the imaging device CAM shown in FIG. 1, the horizontally long rectangles included in the timing diagrams shown in the two parentheses are the rows L1,... Of the pixel arrays of the imaging elements ISEN1, ISEN2, from time t0 to time te. .. Indicates the exposure time of Le. As shown in the timing diagram, the exposure start time and the pixel signal read time in each row L1,... Le of the pixel array of each image sensor ISEN1, ISEN2 are shifted by a time Δt.

  For example, the row L1 indicates the top row of each of the image sensors ISEN1 and ISEN2, and the row Le indicates the bottom row of each of the image sensors ISEN1 and ISEN2. In the image sensor ISEN1, the charges accumulated in the pixel array by exposure are read from each row in the order indicated by the direction D1, and in the image sensor ISEN2, the charges accumulated in the pixel array by the exposure in the order indicated by the direction D2. Read from each row. That is, the image sensor ISEN1 performs exposure for each row from the upper row of the pixel array toward the lower row of the pixel array, and the image sensor ISEN2 performs the exposure from the lower row of the pixel array to the upper side of the pixel array. Exposure is performed row by row toward the row. For example, exposure of the first row L1 of the image sensor ISEN1 and the first row Le of the image sensor ISEN2 is started at time t0.

  At time t1, the driving unit DRV reads out pixel signals from the row L1 of the image sensor ISEN1 and the row Le of the image sensor ISEN2. Thereafter, the driving unit DRV reads pixel signals from each row every time exposure of each row of the imaging elements ISEN1 and ISEN2 is completed. Thereby, the image IMG (IMG1, IMG2) is read from the image sensor ISEN (ISEN1, ISEN2) and stored in the storage unit MEM. Note that the images IMG1 and IMG2 may be output to the image processing apparatus 100.

  As described above, the drive unit DRV reverses the order of the rows for performing exposure and the order of the rows for reading pixel signals (that is, the orientations D1 and D2) with the imaging elements ISEN1 and ISEN2. Note that the scanning directions D1 and D2 in the arrangement direction of the rows of the imaging elements ISEN1 and ISEN2 may be set in advance in the drive unit DRV, or may be set in the drive unit DRV under the control of the image processing apparatus 100.

  In each of the imaging elements ISEN1 and ISEN2, when the exposure times of the respective rows of the pixel array are shifted from each other, rolling shutter distortion may occur in the images IMG1 and IMG2 obtained by imaging.

  The image processing apparatus 100 calculates the speed of the subject OBJ included in the images IMG1 and IMG2 using the two images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2. For example, the image processing device 100 may be realized by an arithmetic processing device such as a CPU (Central Processing Unit) that executes an image processing program, or may be realized by dedicated hardware. The image processing apparatus 100 includes a parallax calculation unit 10, a distortion calculation unit 20, and a speed calculation unit 30. The parallax calculation unit 10 obtains the images IMG1 and IMG2 from the storage unit MEM, and calculates a parallax DPF indicating a difference between the position of the subject OBJ on the image IMG1 and the position of the subject OBJ on the image IMG2. It is an example.

  In FIG. 1, two parentheses shown in the image processing apparatus 100 show an example of images IMG1 and IMG2 read from the image sensors ISEN1 and ISEN2, respectively. In the example shown in FIG. 1, the subject OBJ having an elongated rectangular shape moves at a predetermined speed, and therefore appears as the subject OBJ1 on the image IMG1 and as the subject OBJ2 on the image IMG2 due to rolling shutter distortion. appear. In the images IMG1 and IMG2 shown in parentheses, a rectangle indicated by a broken line indicates the original shape of the subject OBJ when there is no rolling shutter distortion.

  When the subject OBJ moves from the left to the right within the angle of view, the subject OBJ1 on the image IMG1 shows a shape shifted to the right side with respect to the original shape as it goes below the angle of view. In addition, the subject OBJ2 on the image IMG2 shows a shape that is shifted to the right side with respect to the original shape as the angle of view increases. The distortion of each of the subjects OBJ1 and OBJ2 is also referred to as shear strain in the following description because the subject OBJ appears to have received forces opposite to each other above and below the subject OBJ. The deviation between the position of the subject OBJ1 on the image IMG1 and the position of the subject OBJ2 on the image IMG2 is caused by imaging the subject OBJ through the lenses LEN1 and LEN2 arranged at different positions.

  For example, the parallax calculation unit 10 performs image processing such as template matching on the image IMG2 with the image IMG1 as a reference, and indicates the difference between the position of the subject OBJ1 on the image IMG1 and the position of the subject OBJ2 on the image IMG2. DPF is calculated. Then, the parallax calculation unit 10 outputs the calculated parallax DPF to the distortion calculation unit 20 and the speed calculation unit 30.

  The distortion calculation unit 20 is an example of a second calculation unit that compares the image IMG1 and the image IMG2 in order to calculate the speed of the subject OBJ. For example, the distortion calculation unit 20 calculates the distortion information INF indicating the difference in the shapes of the subjects OBJ1 and OBJ2 having different distortions by reversing the scanning directions D1 and D2 by the imaging elements ISEN1 and ISEN2. . The distortion calculation unit 20 uses the parallax DPF calculated by the parallax calculation unit 10 and executes a process of superimposing the subjects OBJ1 and OBJ2. For example, the distortion calculation unit 20 aligns the position of the subject OBJ2 on the image IMG2 with the position of the subject OBJ1 on the image IMG1. Then, the distortion calculation unit 20 performs deformation processing such as shear deformation that deforms the object OBJ2 in opposite directions in the direction along the row on the upper side and the lower side of the object OBJ2 so as to match the shape of the object OBJ1. . For example, the strain calculation unit 20 performs deformation processing a plurality of times while changing the deformation amount of the shear deformation, and a SAD (Sum of Absolute Difference) value indicating the degree of coincidence between the shape of the subject OBJ2 and the shape of the subject OBJ1. Is calculated. The distortion calculation unit 20 calculates, as distortion information INF, the amount of deformation due to shear deformation of the subject OBJ2 when the SAD value has the minimum value (the shape of the subject OBJ2 is matched with the shape of the subject OBJ1). For example, the distortion calculation unit 20 calculates the amount of change (slope indicated by the subject OBJ1) of the shear distortion (rolling shutter distortion) indicated by the subject OBJ1 on the image IMG1 as the distortion information INF. The distortion calculation unit 20 outputs the calculated distortion information INF to the speed calculation unit 30.

  The speed calculation unit 30 is an example of a third calculation unit that calculates the speed VL of the subject OBJ in real space using the parallax DPF calculated by the parallax calculation unit 10 and the distortion information INF calculated by the distortion calculation unit 20. is there. For example, the speed calculation unit 30 moves the subject OBJ on the image sensor ISEN1 based on the number of rows of pixels in the image sensor ISEN1 (the number of pixels in the scanning direction D1) and the slope of the shear distortion indicated by the subject OBJ1 on the image IMG1. The number of pixels (distance) obtained is calculated. Next, the speed calculation unit 30 divides the calculated number of pixels by the readout time from time t1 when the image sensor ISEN1 reads out the pixel signal to time te, and the speed at which the subject OBJ moves on the image sensor ISEN1 (image sensor) The speed of the subject OBJ on ISEN1 is calculated. Then, the speed calculation unit 30 uses the parallax DPF, the distance between the optical axes of the lens LEN1 and the lens LEN2, and the focal length of the lens LEN1, for example, from the speed of the subject OBJ on the image sensor ISEN1 to determine the subject in real space. OBJ speed VL is calculated. The speed calculation unit 30 outputs the calculated speed VL to an external output device such as a monitor. Note that the speed calculation unit 30 may store the calculated speed VL and the images IMG1 and IMG2 in the storage unit MEM.

  As described above, in the embodiment illustrated in FIG. 1, the reading of the pixel signals from the imaging elements ISEN1 and ISEN2 is executed with the scanning directions D1 and D2 in the row arrangement direction opposite to each other. As a result, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 show different shapes (rolling shutter distortion). Therefore, the image processing apparatus 100 performs deformation processing such as shear deformation on the subject OBJ2 on the image IMG2, and calculates the distortion information INF by matching the shape of the subject OBJ2 with the shape of the subject OBJ1. The image processing apparatus 100 uses the deformation amount of the shear deformation included in the distortion information INF and the number of pixels in the image sensor ISEN1 (the number of pixels in the scanning direction D1) as the pixel on which the subject OBJ has moved on the image sensor ISEN1. Calculate the number. The image processing apparatus 100 uses the division value (the speed of the subject OBJ on the image sensor ISEN1) obtained by dividing the calculated number of pixels by the readout time of the pixel signal, the parallax DPF, and the like to obtain the speed VL of the subject OBJ in real space. calculate.

  Thereby, the image processing apparatus 100 acquires two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM from the storage unit MEM, and thus in the real space. The speed VL of the subject OBJ can be calculated. That is, the image processing apparatus 100 can reduce the data amount of the image stored in the storage unit MEM, compared to a case where at least four frames captured by a stereo camera having two imaging elements having the same scanning direction are used. .

  Further, the image processing apparatus 100 can calculate the speed VL of the subject OBJ in the real space using the two images IMG1 and IMG2 obtained by one imaging with the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM. That is, the image processing apparatus 100 does not have to cause the imaging device CAM to capture images twice or more in order to calculate the speed VL of the subject OBJ in real space.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance between the imaging device and the subject OBJ from two frames taken by one imaging device, the speed of the subject OBJ in real space is the same as or similar to that of the image processing device 100. It is difficult to calculate.

  FIG. 2 shows another embodiment of the image processing apparatus and the image processing method. The image processing apparatus 100A illustrated in FIG. 2 is connected to the imaging apparatus CAM2. The imaging device CAM2 is the same as or similar to the imaging device CAM shown in FIG. The same or similar elements as those described in FIG. 1 are denoted by the same or similar reference numerals, and detailed description thereof will be omitted. The dashed arrows in FIG. 2 indicate the flow of information such as signals. The imaging elements ISEN1 and ISEN2 and the drive unit DRV of the imaging device CAM2 are connected to a bus BUS in the image processing device 100A.

  The drive unit DRV, for example, reverses the scanning direction in the row arrangement direction when the pixel signals are sequentially read out for each row based on the control of the image processing apparatus 100A by the imaging elements ISEN1 and ISEN2. The imaging elements ISEN1 and ISEN2 are driven. The drive unit DRV reads out pixel signals using a common clock for the image sensor ISEN1 and ISEN2. Note that the scanning direction in the arrangement direction of the rows of the imaging elements ISEN1 and ISEN2 may be set in advance in the drive unit DRV. The operation of the drive unit DRV will be described with reference to FIG.

  The images IMG1 and IMG2 read from the imaging elements ISEN1 and ISEN2 with the scanning direction of the imaging element ISEN1 in the array direction and the scanning direction of the imaging element ISEN2 in the array direction are reversed. Is output to 100A. In addition, the drive unit DRV outputs read time information TINF indicating a read start time and an end time to the image processing apparatus 100A.

  The image processing apparatus 100A includes a control unit CNTL, a parallax calculation unit 10a, a distortion calculation unit 20a, a speed calculation unit 30a, a recording unit REC, a display unit DIP, memories MEM1, MEM2, and a bus BUS. The control unit CNTL, the parallax calculation unit 10a, the distortion calculation unit 20a, the speed calculation unit 30a, the recording unit REC, the display unit DIP, and the memories MEM1 and MEM2 are connected to the bus BUS.

  The memory MEM1 is a frame memory or the like that stores images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2. The memory MEM2 is a RAM (Random Access Memory) that stores parallax DPF, distortion information INF, readout time information TINF, and the like.

  The control unit CNTL is an arithmetic processing device such as a CPU (Central Processing Unit) and controls each element of the image processing device 100A and the imaging device CAM2. For example, the control unit CNTL causes the drive unit DRV to execute drive processing for reversing the scanning directions in the arrangement direction of the rows of the imaging elements ISEN1 and ISEN2. The operation of the control unit CNTL will be described with reference to FIG.

  The parallax calculation unit 10a receives the images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2 from the memory MEM1 via the bus BUS, and the difference between the position of the subject OBJ on the image IMG1 and the position of the subject OBJ on the image IMG2 Is calculated. That is, the parallax calculation unit 10a is an example of a first calculation unit. The parallax calculation unit 10a outputs the calculated parallax DPF to the memory MEM2 via the bus BUS. The operation of the parallax calculation unit 10a will be described with reference to FIGS.

  The distortion calculation unit 20a receives the images IMG1 and IMG2 from the memory MEM1 via the bus BUS. Then, the distortion calculation unit 20a calculates distortion information INF indicating the difference in shape between the subject OBJ1 of the image IMG1 and the subject OBJ2 of the image IMG2 that are generated because the scanning directions of the imaging elements ISEN1 and ISEN2 are opposite to each other. That is, the distortion calculation unit 20a is an example of a second calculation unit. The distortion calculation unit 20a outputs the calculated distortion information INF to the memory MEM2. The operation of the distortion calculation unit 20a will be described with reference to FIG.

  The speed calculation unit 30a is an example of a third calculation unit that receives the parallax DFP, distortion information INF, readout time information TINF, and the like from the memory MEM2 via the bus BUS and calculates the speed VL of the subject OBJ in real space. The speed calculation unit 30a outputs the calculated speed VL to the recording unit REC and the display unit DIP. The operation of the speed calculation unit 30a will be described with reference to FIG.

  The recording unit REC is a memory or the like that records the images IMG1, IMG2, and the speed VL. The display unit DIP is a liquid crystal display that displays the images IMG1, IMG2, the speed VL, and the like.

  FIG. 3 shows an example of the timing at which pixel signals are read from the image sensors ISEN1 and ISEN2 shown in FIG. In the timing chart shown in FIG. 3, the read timing of the images IMG1 and IMG2 is shown. As shown in the timing diagram, a plurality of pixels arranged in the pixel arrays of the image sensors ISEN1 and ISEN2 are arranged in n pixel areas AR (AR1, AR2,... ARn) every k rows in the row arrangement direction. Divided. In the timing chart, L1, L1 + k, L1 + 2k,... Le indicate rows of the image sensors ISEN1 and ISEN2, and the horizontal axis of the timing chart indicates time. Rows L1, L1 + k, L1 + 2k,... Le− (k−1) are rows located at the upper boundary of each pixel area AR. Symbol VD in the timing diagram corresponds to the period of the vertical synchronization signal (frame period). In FIG. 3, the boundary lines of each row are not shown for easy viewing of the drawing.

  The timing chart of the image sensor ISEN1 is the same as or similar to the timing chart of the image sensor ISEN1 shown in FIG. For example, the imaging element ISEN1 starts exposure of the row L1 of the pixel area AR1 at time t0, and reads out the pixel signal of the row L1 at time t1. The imaging element ISEN1 sequentially executes the control to shift the exposure (scanning) start time, for example, by Δt shown in FIG. 1, toward the row L1 + (k−1) of the pixel area AR1 (broken line arrow in FIG. 3). ) Perform exposure line by line. The imaging element ISEN1 starts exposure of the row L1 + k of the pixel area AR2 at a time shifted by Δt from the time of starting exposure of the row L1 + (k−1) of the pixel area AR1. That is, the imaging element ISEN1 executes control for sequentially shifting the start time of exposure (scanning) in the same or similar manner as the timing chart of the imaging element ISEN1 shown in FIG. 1, and moves toward the row Le of the pixel area ARn ( (Dotted arrows in FIG. 3) The exposure is executed for each row. The image sensor ISEN1 reads the pixel signal of the row Le at time te.

  On the other hand, in the timing chart of the image sensor ISEN2, the image sensor ISEN2 starts exposure of the row L1 + (k−1) of the pixel area AR1 at time t0 and reads out the pixel signal of the row L1 + (k−1) at time t1. . The image sensor ISEN2 executes control for sequentially shifting the exposure start time, and performs exposure for each row toward the row L1 of the pixel region AR1 (broken arrow in FIG. 3). The image sensor ISEN2 is shifted by Δt from the time when exposure of the row L1 in the pixel area AR1 is started (time when the image sensor ISEN1 starts exposure of the row L + k of the pixel area AR2), and the row L1 + (2k -1) exposure is started. Then, the image sensor ISEN2 executes control to sequentially shift the exposure start time, and performs exposure for each row toward the row L1 + k of the pixel area AR2 (broken arrow in FIG. 3). The image sensor ISEN2 repeatedly performs such exposure processing up to the pixel area ARn. The image sensor ISEN2 reads out the pixel signal of the row Le− (k−1) of the pixel area ARn at time te. When the image sensor ISEN2 performs exposure according to the timing chart shown in FIG. 3, the image processing apparatus 100A obtains, for each pixel area AR, a row in which the readout times between the image sensor ISEN1 and the image sensor ISEN2 match. Can do.

  FIG. 4 shows an example of images IMG1 and IMG2 taken by the imaging elements ISEN1 and ISEN2 shown in FIG. 2 based on the timing chart shown in FIG. In FIG. 4, the subject OBJ moves from left to right within the angle of view. Further, in FIG. 4, the number of pixel areas AR is assumed to be three (n = 3) for easy viewing of the drawing. In this case, the subject OBJ1 on the image IMG1 shifts to the right with respect to the original shape indicated by the broken-line rectangle as it goes below the angle of view, as in FIG. On the other hand, the subject OBJ2 (OBJ2a, OBJ2b, OBJ2c) on the image IMG2 is divided into three because the pixel area AR is three. The shape of the subject OBJ2 on the image IMG2 is on the right side of the original shape indicated by the dashed rectangle as it goes from the pixel area AR1 toward the pixel area AR3 and toward the angle of view in each pixel area AR. Shift. As described with reference to FIG. 1, the subject OBJ1 on the image IMG1 and the subject OBJ2 on the image IMG2 are displaced in the left-right direction because the subject OBJ is imaged through the lenses LEN1 and LEN2 arranged at different positions. Caused by.

  FIG. 5 illustrates an example of a parallax DPF calculation process in the parallax calculation unit 10a illustrated in FIG. The images IMG1 and IMG2 shown in FIG. 5A are the same as or similar to those in FIG. FIG. 5B shows image areas including the subjects OBJ1 and OBJ2b in the images IMG1 and IMG2 shown in FIG.

  For example, the parallax calculation unit 10a calculates the parallax DPF between the subject OBJ1 of the image IMG1 and the subject OBJ2 of the image IMG2 for each pixel region AR of the image IMG2 in k rows in the row direction (lateral direction in FIG. 5). Divide into In FIG. 5A, the parallax calculation unit 10a divides each pixel area AR of the images IMG1 and IMG2 into five. That is, the image IMG1 is divided into image areas MBa (MBa11-MBa15, MBa21-MBa25, MBa31-MBa35). The image IMG2 is divided into image areas MBb (MBb11-MBb15, MBb21-MBb25, MBb31-MBb35). The image areas MBa and MBb are examples of the first image area and are also referred to as macro blocks.

  For example, the parallax calculation unit 10a smoothes the pixel values of the images IMG1 and IMG2 for each of the macro blocks MBa and MBb. The parallax calculation unit 10a generates a template between the macroblock MBb to be processed in the smoothed image IMG2 and each macroblock MBa in the pixel area AR that includes the macroblock MBb to be processed in the smoothed image IMG1. Image processing such as matching is executed. For example, the parallax calculation unit 10a calculates a SAD (Sum of Absolute Difference) value indicating the degree of matching between the macro block MBb to be processed and each macro block MBa. The parallax calculation unit 10a extracts the macroblock MBa of the image IMG1 indicating the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). In FIG. 5A, when the macro block MBb24 in the pixel area AR2 indicated by shading in the image IMG2 is a processing target, the parallax calculation unit 10a performs the macro block MBa22 in the pixel area AR2 of the image IMG1 indicated by shading. To calculate the minimum SAD value. That is, the subject OBJ1 of the macroblock MBa22 shown in FIG. 5A corresponds to the subject OBJ2b of the macroblock MBb24.

  Then, the parallax calculation unit 10a calculates the parallax DPF of the subject OBJ from the difference between the position of the processing target macroblock MBb and the position of the macroblock MBa corresponding to the processing target macroblock MBb. The parallax calculation unit 10a outputs the parallax DPF calculated for each macroblock MBb of the image IMG2 to the memory MEM2.

  In addition, since the parallax calculation unit 10a smoothes the images IMG1 and IMG2 for each of the macro blocks MBa and MBb, the accuracy of the parallax DPF is about the size of the macro blocks MBa and MBb. Therefore, the parallax calculation unit 10a may execute processing for increasing the accuracy of the parallax DPF. For example, as shown in FIG. 5B, the disparity calculating unit 10a is shaded including the central row LMBC among the processing target macroblock MBb and the macroblock MBa corresponding to the processing target macroblock MBb. The image area having the width of the indicated m rows is extracted. Note that m rows is smaller than the number k of macroblocks MBa and MBb. In FIG. 5B, the parallax calculation unit 10a executes image processing such as template matching in the image region having a width of m rows between the extracted macro block MBb24 to be processed and the corresponding macro block MBa22.

  The parallax calculation unit 10a calculates a position where the SAD value has a minimum value in an image area having a width of m rows, and corrects the parallax DPF at the calculated position. Since the position is calculated in an image area having a width of m rows smaller than the number k of rows of the macro blocks MBa and MBb, the corrected parallax DPF has an accuracy in pixel units. In addition, since the scanning directions of the imaging elements ISEN1 and ISEN2 are opposite to each other for each pixel area AR, the readout time of the pixel signal of the row LMBC located at the center between the macroblock MBb of the image IMG2 and the macroblock MBa of the image IMG1 is Match or nearly match each other. That is, the shapes of the subjects OBJ1 and OBJ2 in the image area having the width of m rows match each other as compared with the shapes of the subjects OBJ1 and OBJ2 in the sizes of the macro blocks MBa and MBb. As a result, the corrected parallax DPF is improved with accuracy in pixel units.

  Further, as illustrated in FIG. 7, the parallax calculation unit 10a may recalculate the parallax using the distortion information INF calculated by the distortion calculation unit 20a.

  FIG. 6 shows an example of calculation processing in the distortion calculation unit 20a shown in FIG. In the process shown in FIG. 6, the distortion calculation unit 20a uses the macroblock MBa22 of the image IMG1 and the macroblock MBb24 of the image IMG2 shown in FIG. 5B, and shows the distortion indicating the difference in shape between the subjects OBJ1 and OBJ2. Information INF is calculated. Note that the distortion calculation unit 20a performs the same or similar calculation process as that of the macro block MBb24 on the macro block MBb other than the macro block MBb24 of the image IMG2.

  The distortion calculation unit 20a performs a process of superimposing the subjects OBJ1 and OBJ2 on each other using the parallax DPF calculated by the parallax calculation unit 10a for each macroblock MBb of the image IMG2. For example, the distortion calculation unit 20a matches the position of the subject OBJ2b in the macroblock MBb24 and the position of the subject OBJ1 in the macroblock MBa22 using the parallax DPF in the processing target macroblock MBb24. Then, the distortion calculation unit 20a deforms the deformation of the shear deformation (the upper and lower sides of the subject OBJ2b are deformed in opposite directions as indicated by dotted arrows in FIG. 6 so as to match the shape of the subject OBJ1 of the macroblock MBa22. Process) is executed on the macroblock MBb24. For example, the strain calculation unit 20a performs a shear deformation process on the macro block MBb 24 to be processed using Expression (1) including an affine transformation matrix, and obtains a macro block MBc indicated by a broken broken line.

(X, y) indicates the pixel position of the macro block MBb24 to be processed, and (xa, ya) indicates the pixel position of the macro block MBc subjected to shear deformation. The variable α is a deformation amount of the shear deformation, and indicates the inclination of the shear distortion (rolling shutter distortion) indicated by the subject OBJ1 of the macro block MBa22. Hereinafter, the variable α is also referred to as a shear rate. The variable Tx indicates a position difference (that is, parallax) between the macroblock MBb24 and the macroblock MBa22.

  For example, the strain calculation unit 20a performs image processing such as template matching between the macroblock MBc and the macroblock MBa22 that have undergone shear deformation by changing the shear rate α and the variable Tx, and calculates the SAD value. Note that the variable Tx is defined by the equation (2) using the shear rate α and the number of rows of the macroblock MBb (number of pixels in the row arrangement direction) k, so that the transformation process using the equation (1) is performed. The number of variables to be changed can be one of the shear rates α.

Tx = −α × k / 2 (2)
Thereby, the distortion calculation unit 20a can reduce the calculation amount of the deformation process. Further, the distortion calculation unit 20a can make the centers of the shear-deformed macroblock MBc and the macroblock MBa22 coincide with each other even if the macroblock MBb24 is subjected to the deformation process by using Expression (2).

  The distortion calculation unit 20a determines that the subject OBJ2b subjected to shear deformation when the SAD value indicates the minimum value most closely matches the object OBJ1, and calculates the shear rate α when the SAD value indicates the minimum value as the distortion information INF. .

  The speed calculation unit 30a calculates the speed VL of the subject OBJ in real space using the parallax DPF calculated by the parallax calculation unit 10a, the distortion information INF calculated by the distortion calculation unit 20a, and the readout time information TINF. For example, the number of pixels (distance) ΔPX that the subject OBJ has moved on the image sensor ISEN1 is the number of rows (number of pixels in the row arrangement direction) k in the macroblock MBa and the shear rate α indicated by the subject OBJ1 of the macroblock MBa. And is represented by Formula (3). The speed calculation unit 30a calculates the number of pixels ΔPX from Expression (3).

ΔPX = α × k / 2 (3)
Next, the speed calculation unit 30a calculates the readout time ΔtR for the image sensor ISEN1 to read out the pixel signal in the macro block MBa from Expression (4).

ΔtR = (te−t1) × k / (Le−L1 + 1) (4)
Variables t1 and te indicate the start time and end time of reading of the pixel signal shown in FIG. 3 included in the read time information TINF. The variable k indicates the number of rows of the macroblock MBa (the number of pixels in the row arrangement direction). The variables L1 and Le respectively indicate the first row and the last row of the image sensor ISEN1 shown in FIG. 3, and (Le−L1 + 1) indicates the total number of pixel arrays of the image sensor ISEN1 (total number of rows in the array direction). Number of pixels).

  The speed calculation unit 30a calculates the speed of the subject OBJ moving on the image sensor ISEN1 by dividing the number of pixels ΔPX moved on the image sensor ISEN1 by the readout time ΔtR. Then, the speed calculation unit 30a uses the speed of the subject OBJ moving on the imaging element ISEN1, the parallax DPF, the distance between the optical axes of the lens LEN1 and the lens LEN2, and the focal length of the lens LEN1, and uses the subject in real space. OBJ speed VL is calculated. For example, the distance DIS between the imaging device CAM2 and the subject OBJ is expressed by Expression (5) using the parallax DPF, the distance B between the optical axes of the lens LEN1 and the lens LEN2, and the focal length f of the lens LEN1. .

DIS = B × f / DPF (5)
The speed calculation unit 30a calculates the speed VL of the subject OBJ in the real space using Expression (6).
VL = V0 * DIS / f = V0 * (B * f / DPF) / f = V0 * B / DPF (6)
A variable V0 indicates the speed of the subject OBJ that moves on the imaging element ISEN1. The speed calculation unit 30a outputs the calculated speed VL to the recording unit REC and the display unit DIP together with the images IMG1 and IMG2.

  FIG. 7 shows an example of image processing in the image processing apparatus 100A shown in FIG. The processing shown in FIG. 7 is realized by the control unit CNTL executing an image processing program. Note that the processing illustrated in FIG. 7 may be executed by hardware provided in the image processing apparatus 100A. In this case, the parallax calculation unit 10a, the distortion calculation unit 20a, and the speed calculation unit 30a illustrated in FIG. 2 are realized by a circuit arranged in the image processing apparatus 100A.

  In step S100, the parallax calculation unit 10a acquires images IMG1 and IMG2 captured by the imaging device CAM2 from the memory MEM1.

  In step S110, the parallax calculation unit 10a calculates the parallax DPF from the position difference between the macro block MBb to be processed among the macro blocks MBb of the image IMG2 and the macro block MBa of the image IMG1 corresponding to the macro block MBb to be processed. calculate. For example, the parallax calculation unit 10a smoothes the pixel values of the images IMG1 and IMG2 for each of the macro blocks MBa and MBb. The parallax calculation unit 10a selects the macro block MBb to be processed from the smoothed image IMG2. The parallax calculation unit 10a performs image processing such as template matching between the selected processing target macroblock MBb and each macroblock MBa in the pixel area AR in which the processing target macroblock MBb is included in the smoothed image IMG1. To calculate the SAD value. The parallax calculation unit 10a extracts the macroblock MBa of the image IMG1 indicating the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). Then, the parallax calculation unit 10a calculates the parallax DPF of the subject OBJ from the difference between the position of the macro block MBb to be processed and the position of the extracted macro block MBa. The parallax calculation unit 10a outputs the parallax DPF calculated for each macroblock MBb to the memory MEM2.

  In step S120, the parallax calculation unit 10a corrects the parallax DPF calculated in step S110. For example, the parallax calculation unit 10a performs image processing such as template matching on the image area having a width of m rows including the central row LMBC among the macroblocks MBa and MBb used in step S110, and calculates an SAD value. . The parallax calculation unit 10a calculates a position where the SAD value has a minimum value in an image area having a width of m rows, and corrects the parallax DPF at the calculated position. The parallax calculation unit 10a outputs the corrected parallax DPF to the memory MEM2. Note that the process of step S120 for correcting the parallax DPF may be omitted.

  In step S130, the distortion calculation unit 20a uses the parallax DPF corrected in step S120 to align the position of the macro block MBb to be processed with the position of the macro block MBa of the corresponding image IMG1.

  In step S140, the distortion calculation unit 20a calculates a distortion information INF by performing a deformation process on the processing target macroblock MBb of the image IMG2. For example, the distortion calculation unit 20a changes the shear rate α in Expression (1) so as to match the shape of the subject OBJ2 with the shape of the subject OBJ1 of the macroblock MBa extracted in step S110. A deformation process of shear deformation is executed on MBb. The distortion calculation unit 20a performs image processing such as template matching between the macro block MBb to be processed that has undergone shear deformation and the macro block MBa extracted in step S110, and calculates an SAD value. When the SAD value indicates the minimum value, the distortion calculation unit 20a determines that the shape of the subject OBJ2 that has undergone shear deformation most closely matches the shape of the subject OBJ1. Then, the strain calculation unit 20a calculates the shear rate α when the SAD value shows the minimum value as the strain information INF, and outputs the calculated strain information INF to the memory MEM2.

  In step S150, the strain calculation unit 20a determines whether or not the shear rate α of the strain information INF calculated in step S140 is zero. For example, when the shear rate α is 0 (ie, there is no rolling shutter distortion of the subject OBJ), the distortion calculation unit 20a determines that the subject OBJ is not moving (ie, the speed VL in real space is 0). In this case, the process of the image processing apparatus 100A moves to step S180. On the other hand, when the distortion calculation unit 20a determines that the shear rate α is not 0 (that is, there is a rolling shutter distortion of the subject OBJ), the process of the image processing apparatus 100A proceeds to step S160.

  In step S160, the parallax calculation unit 10a recalculates the parallax DPF using the distortion information INF calculated in step S140. In step S120, the parallax calculation unit 10a corrects the parallax DPF using an image region having a width of m rows including the central row LMBC among the macroblocks MBa and MBb. However, the shapes of the subjects OBJ1 and OBJ2 in the image area having a width of m rows are still different from each other. For this reason, the parallax DPF corrected in step S120 includes an error from the parallax caused by imaging the subject OBJ through the lenses LEN1 and LEN2 arranged at different positions. Therefore, for example, the parallax calculation unit 10a performs shear deformation on the processing target macroblock MBb using the shear rate α calculated in step S140. The parallax calculation unit 10a performs template matching between the shear-deformed macroblock MBb to be processed and the macroblock MBa corresponding to the macroblock MBb to be processed, and calculates an SAD value. When the SAD value indicates the minimum value, the parallax calculation unit 10a calculates the parallax DPF of the subject OBJ based on the difference in position between the shear-deformed processing target macroblock MBb and the macroblock MBa corresponding to the processing target macroblock MBb. Recalculate. The parallax calculation unit 10a outputs the recalculated parallax DPF to the memory MEM2.

  In step S170, the speed calculation unit 30a calculates the speed VL of the subject in the real space from the equations (3) to (6) using the parallax DPF recalculated in step S160 and the distortion information INF calculated in step S140. To do.

  In step S180, the control unit CNTL determines whether or not there is a next macroblock MBb in the image IMG2. When the control unit CNTL determines that there is a macroblock MBb to be processed next, the processing of the image processing apparatus 100A proceeds to step S110. On the other hand, when the control unit CNTL determines that there is no macro block MBb to be processed next, the processing of the image processing apparatus 100A ends.

  As described above, in the embodiment shown in FIGS. 2 to 7, the reading of the pixel signals from the image sensors ISEN1 and ISEN2 is executed with the scanning directions in the row arrangement direction reversed for each pixel area AR. Accordingly, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 have different shapes (rolling shutter distortion) for each pixel area AR. Therefore, the image processing apparatus 100A executes deformation processing such as shear deformation on the subject OBJ2 on the image IMG2 for each macroblock MBb, and adjusts the shape of the subject OBJ2 to the shape of the subject OBJ1, thereby distortion information INF. Is calculated. The image processing apparatus 100A calculates the number of pixels ΔPX that the subject OBJ has moved on the image sensor ISEN1 from the shear rate α included in the distortion information INF and the number k of rows of the macroblock MBa. The image processing apparatus 100A calculates the speed VL of the subject OBJ in the real space using a division value (speed V0) obtained by dividing the calculated number of pixels ΔPX by the pixel signal readout time ΔtR, the parallax DPF, and the like.

  Thereby, the image processing apparatus 100A can calculate the speed VL of the subject OBJ in the real space from two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. . That is, the image processing apparatus 100A can reduce the data amount of the image stored in the memory MEM1 as compared with the case where at least four frames captured by a stereo camera having two imaging elements having the same scanning direction are used.

  Further, the image processing apparatus 100A can calculate the speed VL of the subject OBJ in the real space using the two images IMG1 and IMG2 obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. In other words, the image processing apparatus 100A does not have to cause the imaging device CAM2 to capture images twice or more in order to calculate the speed VL of the subject OBJ in the real space. Then, the image processing apparatus 100A can reduce the amount of calculation for calculating the speed of the subject OBJ in the real space compared to the conventional case.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance DIS between the imaging device and the subject OBJ from the two frames taken by one imaging device, the speed of the subject OBJ in real space is the same or similar to that of the image processing device 100A. It is difficult to calculate with.

  FIG. 8 shows another embodiment of the image processing apparatus and the image processing method. The same or similar elements as those described in FIG. 2 are denoted by the same or similar reference numerals, and detailed description thereof will be omitted. The dashed arrows in FIG. 8 indicate the flow of information such as signals.

  The image processing apparatus 100B includes a control unit CNTL, a parallax calculation unit 10b, a distortion calculation unit 20b, a speed calculation unit 30b, a recording unit REC, a display unit DIP, memories MEM1, MEM2, and a bus BUS. The control unit CNTL, the parallax calculation unit 10b, the distortion calculation unit 20b, the speed calculation unit 30b, the recording unit REC, the display unit DIP, and the memories MEM1 and MEM2 are connected to the bus BUS.

  The control unit CNTL is an arithmetic processing device such as a CPU, and controls each element of the image processing device 100B and the imaging device CAM2. For example, as shown in the timing diagram of FIG. 3, the control unit CNTL performs a driving process for reversing the scanning directions in the arrangement direction of the rows of the imaging elements ISEN1 and ISEN2 for each pixel region AR in the driving unit DRV. Let it run.

The parallax calculation unit 10b receives the images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2 from the memory MEM1 via the bus BUS. Similar to the parallax calculation unit 10a illustrated in FIG. 2, the parallax calculation unit 10b generates a parallax DPF indicating a difference between the position of the subject OBJ1 on the image IMG1 and the position of the subject OBJ2 on the image IMG2 as a macroblock of the image IMG2. Calculated for each MBb. The parallax calculation unit 10b is an example of a first calculation unit. The operation of the parallax calculation unit 10b will be described with reference to FIG. 9. The distortion calculation unit 20b receives the images IMG1 and IMG2 from the memory MEM1 via the bus BUS. The distortion calculation unit 20b generates distortion information INF indicating the difference in shape between the subject OBJ1 of the image IMG1 and the subject OBJ2 of the image IMG2 generated by the imaging elements ISEN1 and ISEN2 because the scanning directions are opposite to each other. Calculated for each block MBb. The distortion calculation unit 20b is an example of a second calculation unit. The distortion calculation unit 20b outputs the calculated distortion information INF to the memory MEM2. The operation of the distortion calculation unit 20b will be described with reference to FIG.

  The speed calculation unit 30b is an example of a third calculation unit that receives parallax DPF, distortion information INF, readout time information TINF, and the like from the memory MEM2 via the bus BUS and calculates the speed VL of the subject OBJ in real space. The speed calculation unit 30b outputs the calculated speed VL to the recording unit REC and the display unit DIP. The operation of the speed calculation unit 30b will be described with reference to FIG.

  FIG. 9 shows an example of images IMG1 and IMG2 taken by the image pickup devices ISEN1 and ISEN2 shown in FIG. 8 based on the timing chart shown in FIG. In FIG. 9, the subject OBJ moves, for example, from the top to the bottom (the scanning direction of the image sensor ISEN1) within the angle of view. Also, in FIG. 9, in order to make the drawing easier to see, the number of pixel areas AR is three (n = 3) as in FIG. Furthermore, in the images IMG1 and IMG2, as in FIG. 5, the parallax calculation unit 10b divides each pixel area AR into k columns in the row direction (horizontal direction in FIG. 9). That is, the image IMG1 is divided into image areas MBa (MBa11-MBa15, MBa21-MBa25, MBa31-MBa35). The image IMG2 is divided into image areas MBb (MBb11-MBb15, MBb21-MBb25, MBb31-MBb35).

  The direction of scanning in the pixel area AR of the image sensor ISEN1 is the same as the direction in which the subject OBJ moves. For this reason, the relative speed difference between the scanning speed in the arrangement direction of the row from which the image sensor ISEN1 reads out the pixel signal and the speed of the subject OBJ moving on the image sensor ISEN1 becomes small. Thereby, the subject OBJ1 on the image IMG1 shows a shape extending in a downward direction with respect to the original shape indicated by a shaded rectangle. On the other hand, the relative speed difference between the scanning speed of the image sensor ISEN2 in the row arrangement direction and the speed of the subject OBJ moving on the image sensor ISEN2 is determined by the scanning direction in the pixel area AR of the image sensor ISEN2 and the subject OBJ moving. Since the directions to do are opposite to each other, it becomes larger. Therefore, the subject OBJ2 on the image IMG2 shows a shape contracted in the vertical direction with respect to the original shape indicated by the shaded rectangle. As described with reference to FIG. 1, the subject OBJ1 on the image IMG1 and the subject OBJ2 on the image IMG2 are displaced in the left-right direction because the subject OBJ is imaged through the lenses LEN1 and LEN2 arranged at different positions. Caused by.

  For example, the parallax calculation unit 10b smoothes the pixel values of the images IMG1 and IMG2 for each of the macroblocks MBa and MBb, similarly to the parallax calculation unit 10a illustrated in FIG. The parallax calculation unit 10b generates a template between the macroblock MBb to be processed in the smoothed image IMG2 and each macroblock MBa in the pixel area AR that includes the macroblock MBb to be processed in the smoothed image IMG1. Image processing such as matching is executed. Then, the parallax calculation unit 10b calculates a SAD value indicating the degree of coincidence between the processing target macroblock MBb and each macroblock MBa. The parallax calculation unit 10b extracts the macroblock MBa of the image IMG1 indicating the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). In FIG. 9, when the macro block MBb14 in the pixel area AR1 of the image IMG2 is the processing target, the parallax calculation unit 10b calculates the minimum SAD value in the macro block MBa12 in the pixel area AR1 of the image IMG1. That is, the subject OBJ1 of the macroblock MBa12 shown in FIG. 9 corresponds to the subject OBJ2 of the macroblock MBb14.

  Then, the parallax calculation unit 10b calculates the parallax DPF of the subject OBJ from the difference between the position of the processing target macroblock MBb and the position of the macroblock MBa corresponding to the processing target macroblock MBb. The parallax calculation unit 10b outputs the parallax DPF calculated for each macroblock MBb of the image IMG2 to the memory MEM2.

  FIG. 10 shows an example of calculation processing in the distortion calculation unit 20b shown in FIG. In the example shown in FIG. 10, the distortion calculation unit 20b uses the macroblock MBb14 of the image IMG2 and the macroblock MBa12 of the image IMG1 corresponding to the macroblock MBb14 shown in FIG. 9, and the difference in the shapes of the subjects OBJ1 and OBJ2 Is calculated. Note that the distortion calculation unit 20b performs the same or similar calculation process as that of the macroblock MBb14 on the macroblock MBb other than the macroblock MBb14 of the image IMG2.

  The distortion calculation unit 20b executes a process of superimposing the subjects OBJ1 and OBJ2 on each other using the parallax DPF calculated for each macroblock MBb of the image IMG2 by the parallax calculation unit 10b. For example, the distortion calculation unit 20b matches the position of the subject OBJ2 in the macroblock MBb14 and the position of the subject OBJ1 in the macroblock MBa12 using the parallax DPF in the processing target macroblock MBb14. Then, the distortion calculation unit 20b performs a deformation process (hereinafter also referred to as expansion / contraction deformation) for enlarging (or reducing) the subject OBJ2 in the direction of the arrow illustrated in FIG. 10 so as to match the shape of the subject OBJ1 of the macroblock MBa12. The macro block MBb14 is executed. For example, the distortion calculation unit 20b performs an enlargement / reduction process on the macroblock MBb14 to be processed using Expression (7) including an affine transformation matrix, and obtains an enlarged / reduced macroblock MBd indicated by a broken line.

(X, y) indicates the pixel position of the macro block MBb14 to be processed, and (xa, ya) indicates the pixel position of the enlarged or reduced macro block MBd. The variable β is a deformation amount due to the enlargement / reduction process, and is also referred to as an enlargement / reduction ratio. The variable Ty indicates the difference between the center position in the row arrangement direction of the subject OBJ1 and the center in the row arrangement direction of the subject OBJ2.

  For example, the distortion calculation unit 20b performs image processing such as template matching between the macroblock MBd and the macroblock MBa12 that are scaled by changing the scaling ratio β and the variable Ty, and calculates the SAD value. When the SAD value indicates the minimum value, the distortion calculation unit 20b determines that the enlarged / reduced subject OBJ2 most closely matches the subject OBJ1, and the distortion calculation unit 20b determines the expansion / contraction rate β when the SAD value indicates the minimum value. Calculated as distortion information INF.

  The speed calculation unit 30b calculates the speed VL of the subject OBJ in real space using the parallax DPF calculated by the parallax calculation unit 10b, the distortion information INF calculated by the distortion calculation unit 20b, and the readout time information TINF. For example, as shown in FIG. 9, since the relative speed difference between the scanning speed in the row arrangement direction of the image sensor ISEN1 and the speed of the object OBJ moving on the image sensor ISEN1 is small, the object on the image IMG1 OBJ1 indicates a shape in which the subject OBJ extends downward. On the other hand, since the relative speed difference between the scanning speed of the image sensor ISEN2 in the row arrangement direction and the speed of the subject OBJ moving on the image sensor ISEN2 increases, the subject OBJ2 on the image IMG2 moves the subject OBJ in the vertical direction. Shows the shrunk shape. In other words, FIG. 9 shows that the sizes of the subjects OBJ1 and OBJ2 in the row arrangement direction are the scanning speed in the row arrangement direction of the imaging elements ISEN1 and ISEN2, and the speed of the subject OBJ moving on the imaging elements ISEN1 and ISEN2. It is inversely proportional to the speed difference. When the ratio of the sizes of the subjects OBJ1 and OBJ2 in the row arrangement direction is the enlargement / reduction rate β, the enlargement / reduction rate β is determined by the scanning speed in the row arrangement direction of the imaging elements ISEN1 and ISEN2 and the subject OBJ on the imaging elements ISEN1 and ISEN2. The speed is expressed by the equation (8).

  Variables L1 and L2 indicate the size (number of pixels) of the rows of the subjects OBJ1 and OBJ2 in the arrangement direction. The variable Vsr indicates the scanning speed in the row arrangement direction of the imaging elements ISEN1 and ISEN2. A variable V0 indicates the speed of the subject OBJ that moves on the imaging elements ISEN1 and ISEN2. Then, Expression (9) for calculating the speed V0 of the subject OBJ moving on the imaging elements ISEN1 and ISEN2 is derived from Expression (8).

  Further, as shown in Expression (4), it takes a readout time ΔtR for the image sensors ISEN1 and ISEN2 to read out pixel signals from the macroblocks MBa and MBb having k rows. From this, the scanning speed Vsr in the row arrangement direction in the imaging elements ISEN1 and ISEN2 is expressed as in Expression (10).

Vsr = k / ΔtR = (Le−L1 + 1) / (te−t1) (10)
The speed calculation unit 30b calculates the speed V0 of the subject OBJ moving on the image sensor ISEN1 from the expressions (9) and (10) using the enlargement / reduction ratio β and the readout time information TINF. The speed calculation unit 30b uses the speed V0 of the subject OBJ moving on the image sensor ISEN1, the parallax DPF, the distance B between the optical axes of the lens LEN1 and the lens LEN2, and the focal length f of the lens LEN1, and the expression (6) To calculate the speed VL of the subject OBJ in real space. Then, the speed calculation unit 30b outputs the calculated speed VL to the recording unit REC and the display unit DIP together with the images IMG1 and IMG2.

  FIG. 11 shows an example of image processing in the image processing apparatus 100B shown in FIG. The processing shown in FIG. 11 is realized by the control unit CNTL executing an image processing program. Note that among the operations of the steps shown in FIG. 11, those showing the same or similar processing as the steps shown in FIG. 7 are given the same step numbers, and detailed descriptions thereof are omitted.

  Further, the process illustrated in FIG. 11 may be executed by hardware provided in the image processing apparatus 100B. In this case, the parallax calculation unit 10b, the distortion calculation unit 20b, and the speed calculation unit 30b illustrated in FIG. 8 are realized by circuits arranged in the image processing apparatus 100B.

  The image processing apparatus 100B executes the process of step S140a after executing the processes of step S100, step S110, and step S130 shown in FIG.

  In step S140a, the distortion calculation unit 20b performs a deformation process on the processing target macroblock MBb of the image IMG2 to calculate distortion information INF. For example, the distortion calculation unit 20b changes the enlargement / contraction ratio β and the variable Ty in Expression (7) so that the shape of the subject OBJ2 matches the shape of the subject OBJ1 of the macroblock MBa extracted in step S110. The enlargement / reduction processing is executed on the macro block MBb. The distortion calculation unit 20b performs image processing such as template matching between the macroblock MBb to be processed that has been enlarged / reduced and the macroblock MBa extracted in step S110, and calculates an SAD value. When the SAD value indicates the minimum value, the distortion calculation unit 20b determines that the shape of the scaled subject OBJ2 most closely matches the shape of the subject OBJ1. Then, the distortion calculation unit 20b calculates the expansion / contraction rate β when the SAD value shows the minimum value as the distortion information INF, and outputs the calculated distortion information INF to the memory MEM2.

  In step S150a, the distortion calculation unit 20b determines whether the expansion / contraction rate β of the distortion information INF calculated in step S140a is 1. For example, when the enlargement / reduction ratio β is 1 (that is, there is no rolling shutter distortion of the subject OBJ), the distortion calculation unit 20b determines that the subject OBJ is not moving (that is, the speed VL in real space is 0). In this case, the process of the image processing apparatus 100B moves to step S180. On the other hand, when the distortion calculation unit 20b determines that the enlargement / reduction ratio β is not 1 (that is, there is a rolling shutter distortion of the subject OBJ), the processing of the image processing apparatus 100B proceeds to step S160a.

  In step S160a, the parallax calculation unit 10b recalculates the parallax DPF using the distortion information INF calculated in step S140a. For example, the parallax calculation unit 10b performs an enlargement / reduction process on the processing target macroblock MBb using the enlargement / reduction ratio β calculated in step S140a. The parallax calculation unit 10b matches the shape of the subject OBJ2 of the macro block MBb to be scaled up and down with the shape of the subject OBJ1 of the macro block MBa corresponding to the macro block MBb to be processed. The parallax calculation unit 10b performs template matching between the enlarged / reduced macro block MBb to be processed and the macro block MBa corresponding to the macro block MBb to be processed, and calculates an SAD value. When the SAD value indicates the minimum value, the parallax calculation unit 10b re-analyzes the parallax DPF of the subject OBJ from the position difference between the enlarged / reduced macro block MBb and the macro block MBa corresponding to the macro block MBb to be processed. calculate. The parallax calculation unit 10b outputs the recalculated parallax DPF to the memory MEM2.

  In step S170a, the speed calculation unit 30b uses the parallax DPF recalculated in step S160a, the distortion information INF calculated in step S140a, and the like from the equations (6), (9), and (10). The speed VL of the subject OBJ at is calculated.

  The image processing apparatus 100B executes the process of step S180 after executing the process of step S170a.

  As described above, in the embodiment shown in FIGS. 8 to 11, the reading of the pixel signals from the image sensors ISEN1 and ISEN2 is executed with the scanning directions in the row arrangement direction reversed for each pixel area AR. Accordingly, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 have different shapes (rolling shutter distortion) for each pixel area AR. Therefore, the image processing apparatus 100B calculates a distortion information INF by executing a deformation process for scaling up and down the subject OBJ2 on the image IMG2 for each macroblock MBb and matching the shape of the subject OBJ2 with the shape of the subject OBJ1. To do. The image processing apparatus 100B calculates the speed VL of the subject OBJ in the real space using the parallax DPF, the scaling factor β of the distortion information INF, the readout time information TINF, and the like.

  Thereby, the image processing apparatus 100B can calculate the speed VL of the subject OBJ in the real space from two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. . That is, the image processing apparatus 100B can reduce the data amount of the image stored in the memory MEM1 as compared to the case where at least four frames captured by a stereo camera having two imaging elements having the same scanning direction are used.

  Further, the image processing apparatus 100B can calculate the speed VL of the subject OBJ in the real space using the two images IMG1 and IMG2 obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. That is, the image processing apparatus 100B does not have to cause the imaging device CAM2 to capture images twice or more in order to calculate the speed VL of the subject OBJ in the real space. Then, the image processing apparatus 100B can reduce the amount of calculation for calculating the speed of the subject OBJ in the real space compared to the conventional case.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance DIS between the imaging device and the subject OBJ from the two frames taken by one imaging device, the speed of the subject OBJ in the real space is the same or similar to that of the image processing device 100B. It is difficult to calculate with.

  FIG. 12 shows another embodiment of the image processing apparatus and the image processing method. The same or similar elements as those described in FIG. 8 are denoted by the same or similar reference numerals, and detailed description thereof will be omitted. The dashed arrows in FIG. 12 indicate the flow of information such as signals.

  The image processing apparatus 100C includes a control unit CNTL, a parallax calculation unit 10c, a distortion calculation unit 20c, a speed calculation unit 30c, a recording unit REC, a display unit DIP, memories MEM1, MEM2, and a bus BUS. The control unit CNTL, the parallax calculation unit 10c, the distortion calculation unit 20c, the speed calculation unit 30c, the recording unit REC, the display unit DIP, and the memories MEM1 and MEM2 are connected to the bus BUS.

The parallax calculation unit 10c receives images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2 from the memory MEM1 via the bus BUS. Similar to the parallax calculation unit 10b illustrated in FIG. 8, the parallax calculation unit 10c generates a parallax DPF indicating the difference between the position of the subject OBJ1 on the image IMG1 and the position of the subject OBJ2 on the image IMG2 as a macroblock of the image IMG2. Calculated for each MBb. The parallax calculation unit 10c is an example of a first calculation unit. The operation of the parallax calculation unit 10c will be described with reference to FIG. 13. The distortion calculation unit 20c receives the images IMG1 and IMG2 from the memory MEM1 via the bus BUS. The distortion calculation unit 20c generates distortion information INF indicating the difference in shape between the subject OBJ1 of the image IMG1 and the subject OBJ2 of the image IMG2 generated by the imaging elements ISEN1 and ISEN2 because the scanning directions are opposite to each other. Calculated for each block MBb. The distortion calculation unit 20c is an example of a second calculation unit. The distortion calculation unit 20c outputs the calculated distortion information INF to the memory MEM2. The operation of the distortion calculation unit 20c will be described with reference to FIG.

  The speed calculation unit 30c is an example of a third calculation unit that receives parallax DPF, distortion information INF, readout time information TINF, and the like from the memory MEM2 via the bus BUS and calculates the speed VL of the subject OBJ in real space. The speed calculation unit 30c outputs the calculated speed VL to the recording unit REC and the display unit DIP. The operation of the speed calculation unit 30c will be described with reference to FIG.

  FIG. 13 shows an example of images IMG1 and IMG2 taken by the image pickup devices ISEN1 and ISEN2 shown in FIG. 12 based on the timing chart shown in FIG. In FIG. 13, the subject OBJ moves, for example, from the top to the bottom (the scanning direction of the image sensor ISEN1) within the angle of view. Further, in FIG. 13, in order to make the drawing easier to see, the number of pixel regions AR is three (n = 3) as in FIG. Furthermore, in the images IMG1 and IMG2, as in FIG. 9, the parallax calculation unit 10c divides each pixel area AR into k columns in the row direction (horizontal direction in FIG. 13). That is, the image IMG1 is divided into image areas MBa (MBa11-MBa15, MBa21-MBa25, MBa31-MBa35). The image IMG2 is divided into image areas MBb (MBb11-MBb15, MBb21-MBb25, MBb31-MBb35).

  Since the scanning direction in the pixel area AR of the image sensor ISEN1 and the moving direction of the subject OBJ are the same, the scanning speed Vsr in the row arrangement direction of the image sensor ISEN1 and the speed V0 of the subject OBJ moving on the image sensor ISEN1 The relative speed difference becomes smaller. Thereby, for example, the subject OBJ1 (OBJ1a, OBJ1b) on the image IMG1 has a shape extending across the pixel areas AR1, AR2 in the downward direction with respect to the original shape indicated by the shaded rectangle. On the other hand, since the scanning direction in the pixel area AR of the image sensor ISEN2 and the moving direction of the subject OBJ are opposite to each other, the scanning speed Vsr in the row arrangement direction of the image sensor ISEN2 and the subject OBJ moving on the image sensor ISEN2 The relative speed difference from the speed V0 is increased. For this reason, for example, the subject OBJ2 (OBJ2a, OBJ2b) on the image IMG2 has a shape contracted in the vertical direction with respect to the original shape indicated by the shaded rectangles as the pixel regions AR1, AR2. As described with reference to FIG. 1, the subject OBJ1 on the image IMG1 and the subject OBJ2 on the image IMG2 are displaced in the left-right direction because the subject OBJ is imaged through the lenses LEN1 and LEN2 arranged at different positions. Caused by.

  For example, the parallax calculation unit 10c smoothes the pixel values of the images IMG1 and IMG2 for each of the macroblocks MBa and MBb, similarly to the parallax calculation unit 10b illustrated in FIG. The parallax calculation unit 10c generates a template between the macroblock MBb to be processed in the smoothed image IMG2 and each macroblock MBa in the pixel area AR that includes the macroblock MBb to be processed in the smoothed image IMG1. Image processing such as matching is executed. Then, the parallax calculation unit 10c calculates an SAD value indicating the degree of matching between the macro block MBb to be processed and each macro block MBa. The parallax calculation unit 10b extracts the macroblock MBa of the image IMG1 indicating the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). In FIG. 13, when the macro block MBb14 in the pixel area AR1 of the image IMG2 is the processing target, the parallax calculation unit 10c calculates the minimum SAD value in the macro block MBa12 in the pixel area AR1 of the image IMG1. That is, the subject OBJ1a of the macroblock MBa12 shown in FIG. 13 corresponds to the subject OBJ2a of the macroblock MBb14.

  In addition, when the macro block MBb24 in the pixel area AR2 of the image IMG2 is the processing target, the parallax calculation unit 10c calculates the minimum SAD value in the macro block MBa22 in the pixel area AR2 of the image IMG1. That is, the subject OBJ1b of the macroblock MBa22 shown in FIG. 13 corresponds to the subject OBJ2b of the macroblock MBb24. Then, the parallax calculation unit 10c calculates the parallax DPF of the subject OBJ from the difference between the position of the processing target macroblock MBb and the position of the macroblock MBa corresponding to the processing target macroblock MBb. The parallax calculation unit 10c outputs the parallax DPF calculated for each macroblock MBb of the image IMG2 to the memory MEM2.

  Since the parallax calculation unit 10c smoothes the images IMG1 and IMG2 for each of the macro blocks MBa and MBb, the accuracy of the parallax DPF is about the size of the macro blocks MBa and MBb. Therefore, the parallax calculation unit 10c may execute processing for increasing the accuracy of the parallax DPF. For example, in the macro block MBa corresponding to the processing target macro block MBb and the processing target macro block MBb, the parallax calculation unit 10c detects the edges of the subjects OBJ1 and OBJ2 from the luminance change in the row direction (the horizontal direction in FIG. 13). To detect. The parallax calculation unit 10c calculates the difference between the positions of the detected edges of the subjects OBJ1 and OBJ2, and corrects the parallax DPF with the calculated difference between the positions. By correcting the parallax DPF based on the positions of the edges of the subjects OBJ1 and OBJ2 in the macroblocks MBa and MBb, the corrected parallax DPF has accuracy in units of pixels.

  As shown in FIG. 13, since the subject OBJ1 extends over the pixel areas AR1 and AR2, the macro blocks MBa12 and MBa22 of the image IMG1 lack a part of the subject OBJ1. Therefore, for example, even if the distortion calculation unit 20b illustrated in FIG. 8 performs the enlargement / reduction processing on each of the macroblocks MBb14 and MBb24 that are the processing target of the image IMG2 using Expression (7), the shapes of the subjects OBJ1 and OBJ2 are changed. It becomes difficult to match or almost match each other. That is, it becomes difficult for the distortion calculation unit 20b illustrated in FIG. 8 to calculate the minimum SAD value and to calculate the distortion information INF.

  FIG. 14 shows an example of calculation processing in the distortion calculation unit 20c shown in FIG. FIG. 14A shows the subject OBJ2a of the macro block MBb14 of the image IMG2 shown in FIG. FIG. 14B shows the subject OBJ1a of the macroblock MBa12 of the image IMG1 shown in FIG. That is, in the example shown in FIG. 14, the distortion calculation unit 20c uses the macroblock MBb14 to be processed and the macroblock MBa12 corresponding to the macroblock MBb14 to obtain distortion information INF indicating the difference in shape between the subjects OBJ1 and OBJ2. calculate. Note that the distortion calculation unit 20c performs the same or similar calculation process as that of the macroblock MBb14 on the macroblock MBb other than the macroblock MBb14 of the image IMG2.

  For example, the distortion calculation unit 20c detects the edges of the subjects OBJ1 and OBJ2 from the luminance change in the row arrangement direction based on the luminance distributions of the macroblock MBa12 of the image IMG1 and the macroblock MBb14 of the image IMG2. For example, in the macro block MBb14 of the image IMG2 shown in FIG. 14A, since the subject OBJ2a exists between the rows EG2a and EG2b, the luminance of the image IMG2 changes more than a predetermined value at the positions of the rows EG2a and EG2b. . That is, the distortion calculation unit 20c detects the edge of the subject OBJ2a from the change in luminance greater than a predetermined value at the positions of the rows EG2a and EG2b. Further, in the macro block MBa12 of the image IMG1 shown in FIG. 14B, since the subject OBJ1a exists in the row below the row EG1, the luminance of the image IMG1 changes more than a predetermined value at the position of the row EG1. . The distortion calculation unit 20c detects the edge of the subject OBJ1a from a change in luminance greater than a predetermined value at the position of the row EG1. The reason why the distortion calculation unit 20c does not detect two edges of the subject OBJ1a from the macroblock MBa12 of the image IMG1 is that the subject OBJ1 straddles the pixel areas AR1 and AR2, as shown in FIG.

  The distortion calculation unit 20c extracts edges corresponding to each other among the detected edge of the subject OBJ1a (row EG1) and the edge of the subject OBJ2a (rows EG2a and EG2b) based on the change in luminance in the row arrangement direction. . For example, as shown in FIG. 14, the subjects OBJ1a and OBJ2a exist below the row EG1 of the subject OBJ1a and the row EG2a of the subject OBJ2a, respectively. Therefore, the luminance in the row arrangement direction at the positions of the row EG1 and the row EG2a shows the same or similar changes. The distortion calculation unit 20c extracts the row EG1 of the subject OBJ1a and the row EG2a of the subject OBJ2a as corresponding edges. The distortion calculation unit 20c calculates, as distortion information INF, a line EG1 indicating the position of the edge of the subject OBJ1a, a line EG2a and EG2b indicating the position of the edge of the subject OBJ2a, and an edge corresponding to each other between the subjects OBJ1a and OBJ2a. The distortion calculation unit 20c outputs the calculated distortion information INF to the memory MEM2.

  The speed calculation unit 30c calculates the speed VL of the subject OBJ in real space using the parallax DPF calculated by the parallax calculation unit 10c, the distortion information INF calculated by the distortion calculation unit 20c, and the readout time information TINF. For example, in FIG. 14, since the edge of the row EG1 of the subject OBJ1a corresponds to the edge of the row EG2a of the subject OBJ2a, the relationship of Expressions (11) to (14) is established.

Y0 + V0 × T1 = Vsr × T1 (11)
Y0 + V0 × T1 = EG1 (12)
Y0 + V0 × T2 = k−Vsr × T2 (13)
Y0 + V0 × T2 = EG2a (14)
The variable Y0 indicates the position of the row of the subject OBJ in the arrangement direction at time t0 when the image sensors ISEN1 and ISEN2 start exposure. The variable T1 indicates the time at which the row from which the image sensor ISEN1 reads the pixel signal in the macro block MBa12 is the row EG1. The variable T2 indicates the time when the row from which the image sensor ISEN2 reads the pixel signal of the macroblock MBb14 is the row EG2a. The variable k indicates the number of rows of the macroblocks MBa12 and MBb14 (the number of pixels in the row arrangement direction).

  Expressions (11) and (12) indicate that the edge of the subject OBJ1a on the image sensor ISEN1 is located in the row EG1 of the macroblock MBa12 at time T1. Expressions (13) and (14) indicate that the edge of the subject OBJ2a on the image sensor ISEN2 is located in the row EG2a of the macroblock MBb14 at time T2. From Expression (11) -Expression (14), Expression (15) for calculating the speed V0 of the subject OBJ moving on the image sensor ISEN1 is derived. Note that rows EG1 and EG2a in Expression (15) indicate edges corresponding to each other between the subject OBJ2 of the processing target macroblock MBb and the subject OBJ1 of the macroblock MBa corresponding to the processing target macroblock MBb. Is substituted.

  The speed calculation unit 30c uses the distortion information INF and the readout time information TINF to calculate the speed V0 of the subject OBJ that moves on the image sensor ISEN1 from Expressions (10) and (15). The speed calculation unit 30c uses the speed V0, the parallax DPF, the distance B between the optical axes of the lens LEN1 and the lens LEN2, and the focal length f of the lens LEN1 using the speed V0 of the subject OBJ that moves on the image sensor ISEN1 and the focal length f of the lens LEN1. To calculate the speed VL of the subject OBJ in real space. Then, the speed calculation unit 30b outputs the calculated speed VL to the recording unit REC and the display unit DIP together with the images IMG1 and IMG2.

  FIG. 15 shows an example of image processing in the image processing apparatus 100C shown in FIG. The processing shown in FIG. 15 is realized by the control unit CNTL executing an image processing program. Note that among the operations of the steps shown in FIG. 15, those showing the same or similar processing as the steps shown in FIG. 11 are given the same step numbers, and detailed descriptions thereof are omitted.

  Further, the process illustrated in FIG. 15 may be executed by hardware provided in the image processing apparatus 100C. In this case, the parallax calculation unit 10c, the distortion calculation unit 20c, and the speed calculation unit 30c illustrated in FIG. 12 are realized by a circuit arranged in the image processing apparatus 100C.

  The image processing apparatus 100C executes the processing of step S120a after executing the processing of step S100 and step S110 shown in FIG.

  In step S120a, the parallax calculation unit 10c corrects the parallax DPF calculated in step S110. For example, the parallax calculation unit 10c detects the edges of the subjects OBJ1 and OBJ2 from the change in luminance in the row direction in the processing target macroblock MBb and the macroblock MBa corresponding to the processing target macroblock MBb. The parallax calculation unit 10c calculates the difference between the positions of the detected edges of the subjects OBJ1 and OBJ2, and corrects the parallax DPF with the calculated difference between the positions. The parallax calculation unit 10c outputs the corrected parallax DPF to the memory MEM2. Note that the process of step S120a for correcting the parallax DPF may be omitted.

  In step S145, the distortion calculation unit 20c detects the edges of the subjects OBJ1 and OBJ2 from the macroblock MBb to be processed of the image IMG2 and the macroblock MBa of the image IMG1 corresponding to the macroblock MBb to be processed, and distortion information INF. Is calculated. For example, the distortion calculation unit 20c detects the edge of the subject OBJ2 from the change in luminance in the row arrangement direction based on the luminance distribution of the processing target macroblock MBb of the image IMG2. The distortion calculation unit 20c detects the edge of the subject OBJ1 from the change in luminance in the row arrangement direction based on the luminance distribution of the macroblock MBa of the image IMG1 corresponding to the processing target macroblock MBb. Then, the distortion calculation unit 20c extracts edges corresponding to each other from the detected edge of the subject OBJ1 and the edge of the subject OBJ2 based on the luminance change in the row arrangement direction. The distortion calculation unit 20c calculates the detected edges of the objects OBJ1 and OBJ2 and the edges corresponding to each other between the extracted objects OBJ1 and OBJ2 as distortion information INF. The distortion calculation unit 20c outputs the calculated distortion information INF to the memory MEM2.

  In step S155, the distortion calculation unit 20c determines whether or not corresponding edges are extracted between the subjects OBJ1 and OBJ2 in step S145. For example, when the corresponding edges are not extracted between the subjects OBJ1 and OBJ2, the distortion calculation unit 20c determines that it is difficult to calculate the speed VL of the subject OBJ in the real space. In this case, the process of the image processing apparatus 100C moves to step S180. On the other hand, when the distortion calculation unit 20c extracts corresponding edges between the subjects OBJ1 and OBJ2, the process of the image processing apparatus 100C proceeds to step S170b.

  In step S170b, the speed calculation unit 30c uses the parallax DPF corrected in step S120a, the distortion information INF calculated in step S145, and the like in the real space from Expression (6), Expression (10), and Expression (15). The speed VL of the subject OBJ is calculated.

  The image processing apparatus 100C performs the process of step S180 after executing the process of step S170b.

  As described above, in the embodiment shown in FIGS. 12 to 15, the reading of the pixel signals from the imaging elements ISEN1 and ISEN2 is executed with the scanning directions in the row arrangement direction reversed for each pixel area AR. Accordingly, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 have different shapes (rolling shutter distortion) for each pixel area AR. Therefore, the image processing apparatus 100C detects the edges of the subjects OBJ1 and OBJ2 based on the luminance distribution in the macroblocks MBa and MBb, and extracts the corresponding edges between the subjects OBJ1a and OBJ2, thereby obtaining the distortion information INF. calculate. The image processing apparatus 100C calculates the speed VL of the subject OBJ in the real space using the parallax DPF, the distortion information INF, the readout time information TINF, and the like.

  Thereby, the image processing apparatus 100C can calculate the speed VL of the subject OBJ in the real space from two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. . That is, the image processing apparatus 100C does not have to cause the imaging device CAM2 to capture images twice or more in order to calculate the speed VL of the subject OBJ in the real space. Then, the image processing apparatus 100C can reduce the amount of calculation for calculating the speed of the subject OBJ in the real space as compared with the conventional case.

  Further, the image processing apparatus 100C detects the edges of the subjects OBJ1 and OBJ2 based on the luminance distribution in the macro blocks MBa and MBb, and extracts corresponding edges between the subjects OBJ1a and OBJ2. Thereby, the image processing apparatus 100C can calculate the speed VL of the subject OBJ in the real space even when the subjects OBJ1 and OBJ2 straddle the plurality of pixel regions AR.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance DIS between the imaging device and the subject OBJ from the two frames taken by one imaging device, the speed of the subject OBJ in real space is the same as or similar to that of the image processing device 100C. It is difficult to calculate with.

  FIG. 16 shows another embodiment of the image processing apparatus and the image processing method. The same or similar elements as those described in FIGS. 8 and 12 are denoted by the same or similar reference numerals, and detailed description thereof will be omitted. The dashed arrows in FIG. 16 indicate the flow of information such as signals.

  The image processing apparatus 100D includes a control unit CNTL, a calculation unit CAL, a recording unit REC, a display unit DIP, memories MEM1, MEM2, and a bus BUS. The control unit CNTL, the calculation unit CAL, the recording unit REC, the display unit DIP, and the memories MEM1 and MEM2 are connected to the bus BUS.

  The calculation unit CAL includes the parallax calculation unit 10b, the distortion calculation unit 20b, and the speed calculation unit 30b illustrated in FIG. 8, and the parallax calculation unit 10c, the distortion calculation unit 20c, and the speed calculation unit 30c illustrated in FIG. That is, the image processing apparatus 100D has the functions of the image processing apparatus 100B shown in FIG. 8 and the functions of the image processing apparatus 100C shown in FIG.

  17 and 18 show an example of image processing in the image processing apparatus 100D shown in FIG. The processing illustrated in FIGS. 17 and 18 is realized by the control unit CNTL executing an image processing program. Note that the processing illustrated in FIGS. 17 and 18 may be executed by hardware provided in the image processing apparatus 100D. In this case, the calculation unit CAL illustrated in FIG. 16 is realized by a circuit arranged in the image processing apparatus 100D.

  In step S200, the parallax calculation unit 10c acquires images IMG1 and IMG2 captured by the imaging device CAM2 from the memory MEM1.

  In step S210, the disparity calculating unit 10c calculates the disparity DPF from the difference in position between the macro block MBb to be processed in the macro block MBb of the image IMG2 and the macro block MBa of the image IMG1 corresponding to the macro block MBb to be processed. calculate. For example, the parallax calculation unit 10c smoothes the pixel values of the images IMG1 and IMG2 for each of the macroblocks MBa and MBb. The parallax calculation unit 10c selects the macro block MBb to be processed from the smoothed image IMG2. The parallax calculation unit 10c performs image processing such as template matching between the selected processing target macroblock MBb and each macroblock MBa of the pixel area AR in which the processing target macroblock MBb is included in the smoothed image IMG1. To calculate the SAD value. The parallax calculation unit 10c extracts the macroblock MBa of the image IMG1 indicating the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). Then, the parallax calculation unit 10a calculates the parallax DPF of the subject OBJ from the difference between the position of the macro block MBb to be processed and the position of the extracted macro block MBa. The parallax calculation unit 10a outputs the parallax DPF calculated for each macroblock MBb to the memory MEM2.

  In step S220, the parallax calculation unit 10c corrects the parallax DPF calculated in step S210. For example, the parallax calculation unit 10c detects the edges of the subjects OBJ1 and OBJ2 from the change in luminance in the row direction in the processing target macroblock MBb and the macroblock MBa corresponding to the processing target macroblock MBb. The parallax calculation unit 10c calculates the difference between the positions of the detected edges of the subjects OBJ1 and OBJ2, and corrects the parallax DPF with the calculated difference between the positions. The parallax calculation unit 10c outputs the corrected parallax DPF to the memory MEM2.

  In step S230, the distortion calculation unit 20c detects the edges of the subjects OBJ1 and OBJ2 in the row arrangement direction based on the luminance distribution in the processing target macroblock MBb and the macroblock MBa corresponding to the processing target macroblock MBb. To do.

  In step S240, the distortion calculation unit 20c determines whether or not the numbers of edges of the subjects OBJ1 and OBJ2 detected in step S230 match each other. When the number of edges matches each other, the distortion calculation unit 20c determines that the subjects OBJ1 and OBJ2 are within one macroblock, for example, as illustrated in FIG. In this case, the process of the image processing apparatus 100D moves to step S280. On the other hand, when the number of edges does not match each other, the distortion calculation unit 20c determines that at least one of the subjects OBJ1 and OBJ2 straddles a plurality of macroblocks, for example, as illustrated in FIG. In this case, the process of the image processing apparatus 100D moves to step S250.

  In step S250, the distortion calculation unit 20c calculates distortion information INF using the edges of the subjects OBJ1 and OBJ2 detected in step S230. For example, the distortion calculation unit 20c, based on the change in luminance in the row arrangement direction between the macro block MBb to be processed and the corresponding macro block MBa, out of the edges of the subject OBJ1 and the subject OBJ2 detected in step S230. Extract corresponding edges. The distortion calculation unit 20c calculates the detected edges of the objects OBJ1 and OBJ2 and the edges corresponding to each other between the extracted objects OBJ1 and OBJ2 as distortion information INF. The distortion calculation unit 20c outputs the calculated distortion information INF to the memory MEM2.

  In step S260, the distortion calculation unit 20c determines whether or not corresponding edges are extracted between the subjects OBJ1 and OBJ2 in step S250. For example, when the corresponding edges are not extracted between the subjects OBJ1 and OBJ2, the distortion calculation unit 20c determines that it is difficult to calculate the speed VL of the subject OBJ in the real space. In this case, the process of the image processing apparatus 100D moves to step S370. On the other hand, when the distortion calculation unit 20c extracts corresponding edges between the subjects OBJ1 and OBJ2, the process of the image processing apparatus 100D proceeds to step S270.

  In step S270, the speed calculation unit 30c uses the parallax DPF corrected in step S220, the distortion information INF calculated in step S250, and the like in the real space from the equations (6), (10), and (15). The speed VL of the subject OBJ is calculated.

  In step S280, the distortion calculation unit 20b uses the parallax DPF corrected in step S220 to set the position of the processing target macroblock MBb of the image IMG2 to the macroblock MBa of the image IMG1 corresponding to the processing target macroblock MBb. Adjust to the position of.

  In step S290, the distortion calculation unit 20b performs a deformation process on the processing target macroblock MBb of the image IMG2 to calculate distortion information INF. For example, the distortion calculation unit 20b changes the enlargement / reduction ratio β and the variable Ty in Expression (7) so that the shape of the subject OBJ2 matches the shape of the subject OBJ1 of the macroblock MBa extracted in step S210. The enlargement / reduction processing is executed on the macro block MBb. The distortion calculation unit 20b performs image processing such as template matching between the macroblock MBb to be processed that has been enlarged / reduced and the macroblock MBa extracted in step S210, and calculates an SAD value. When the SAD value indicates the minimum value, the distortion calculation unit 20b determines that the shape of the scaled subject OBJ2 most closely matches the shape of the subject OBJ1. Then, the distortion calculation unit 20b calculates the expansion / contraction rate β when the SAD value shows the minimum value as the distortion information INF, and outputs the calculated distortion information INF to the memory MEM2.

  In step S300 illustrated in FIG. 18, the distortion calculation unit 20b determines whether the expansion / contraction rate β of the distortion information INF calculated in step S290 is 1. For example, when the enlargement / reduction ratio β is 1 (that is, there is no rolling shutter distortion of the subject OBJ), the distortion calculation unit 20b determines that the subject OBJ is not moving (that is, the speed VL in real space is 0). In this case, the process of the image processing apparatus 100D moves to step S330. On the other hand, when the distortion calculation unit 20b determines that the enlargement / reduction ratio β is not 1 (that is, there is a rolling shutter distortion of the subject OBJ), the processing of the image processing device 100D proceeds to step S310.

  In step S310, the parallax calculation unit 10b recalculates the parallax DPF using the distortion information INF calculated in step S290. For example, the parallax calculation unit 10b performs an enlargement / reduction process on the macro block MBb to be processed using the enlargement / reduction ratio β calculated in step S290. The parallax calculation unit 10b matches the shape of the subject OBJ2 of the macro block MBb to be scaled up and down with the shape of the subject OBJ1 of the macro block MBa corresponding to the macro block MBb to be processed. The parallax calculation unit 10b performs template matching between the enlarged / reduced macro block MBb to be processed and the macro block MBa corresponding to the macro block MBb to be processed, and calculates an SAD value. When the SAD value indicates the minimum value, the parallax calculation unit 10b re-analyzes the parallax DPF of the subject OBJ from the position difference between the enlarged / reduced macro block MBb and the macro block MBa corresponding to the macro block MBb to be processed. calculate. The parallax calculation unit 10b outputs the recalculated parallax DPF to the memory MEM2.

  In step S320, the speed calculation unit 30b uses the parallax DPF recalculated in step S310, the distortion information INF calculated in step S290, and the like to calculate the real space from the equations (6), (9), and (10). The speed VL of the subject OBJ at is calculated.

  In step S330, the distortion calculation unit 20c calculates distortion information INF using the edges of the subjects OBJ1 and OBJ2 detected in step S230. For example, the distortion calculation unit 20c, based on the change in luminance in the row arrangement direction between the macro block MBb to be processed and the corresponding macro block MBa, out of the edges of the subject OBJ1 and the subject OBJ2 detected in step S230. Extract corresponding edges. The distortion calculation unit 20c calculates the detected edges of the objects OBJ1 and OBJ2 and the edges corresponding to each other between the extracted objects OBJ1 and OBJ2 as distortion information INF. The distortion calculation unit 20c outputs the calculated distortion information INF to the memory MEM2.

  In step S340, the distortion calculation unit 20c determines whether or not corresponding edges are extracted between the subjects OBJ1 and OBJ2 in step S330. For example, when the corresponding edges are not extracted between the subjects OBJ1 and OBJ2, the distortion calculation unit 20c determines that it is difficult to calculate the speed VL of the subject OBJ in the real space. In this case, the process of the image processing apparatus 100D moves to step S360. On the other hand, when the distortion calculation unit 20c extracts corresponding edges between the subjects OBJ1 and OBJ2, the process of the image processing apparatus 100D proceeds to step S350.

  In step S350, the speed calculation unit 30c calculates the speed VL of the subject OBJ in the real space from Expression (6), Expression (10), and Expression (15). The speed calculation unit 30c calculates the speed VL of the subject OBJ in real space, the parallax DPF corrected in step S220 or the parallax DPF recalculated in step S310, and the distortion information calculated in step S330. INF or the like is used.

  In step S360, the calculation unit CAL calculates the average value of the speeds VL calculated in steps S320 and S350 as the speed VL of the subject OBJ in the real space. The calculation unit CAL outputs the calculated speed VL of the subject OBJ in the real space to the memory MEM2. When the process of step S350 is skipped, the calculation unit CAL sets the speed VL calculated in step S320 as the speed VL of the subject OBJ in real space.

  In step S370, the control unit CNTL determines whether or not there is a next macroblock MBb in the image IMG2. When the control unit CNTL determines that there is a macroblock MBb to be processed next, the processing of the image processing apparatus 100D proceeds to step S210. On the other hand, when the control unit CNTL determines that there is no macro block MBb to be processed next, the processing of the image processing apparatus 100D ends.

  As described above, in the embodiments shown in FIGS. 16 to 18, the reading of the pixel signals from the imaging elements ISEN <b> 1 and ISEN <b> 2 is performed with the scanning directions in the row arrangement direction reversed for each pixel area AR. Accordingly, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 have different shapes (rolling shutter distortion) for each pixel area AR. Therefore, the image processing apparatus 100D detects the edges of the subjects OBJ1 and OBJ2 based on the luminance distribution in the macroblocks MBa and MBb, and calculates the speed VL of the subject OBJ in the real space according to the number of detected edges. Switch.

  That is, the image processing device 100D performs the same or similar processing as the image processing of the image processing device 100B shown in FIG. 8 and the image processing device 100C shown in FIG. 12 when the numbers of edges of the subjects OBJ1 and OBJ2 match each other. To calculate the speed VL of the subject OBJ. Then, the image processing apparatus 100D sets the average value of the speeds VL calculated from the respective image processes as the speed VL of the subject OBJ in the real space. On the other hand, when the numbers of edges of the subjects OBJ1 and OBJ2 do not match each other, the image processing device 100D executes the same or similar processing as the image processing in the image processing device 100C shown in FIG. The speed VL is calculated.

  Thereby, the image processing apparatus 100D can reliably calculate the speed VL of the subject OBJ in the real space according to the states of the subjects OBJ1 and OBJ2 in the macro blocks MBa and MBb.

  Further, the image processing apparatus 100D can calculate the speed VL of the subject OBJ in real space from two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. That is, the image processing apparatus 100D can reduce the data amount of the image stored in the memory MEM1 as compared to the case where at least four frames captured by a stereo camera having two imaging elements having the same scanning direction are used.

  Further, the image processing apparatus 100D can calculate the speed VL of the subject OBJ in the real space using the two images IMG1 and IMG2 obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. In other words, the image processing apparatus 100D does not have to cause the imaging apparatus CAM2 to capture images twice or more in order to calculate the speed VL of the subject OBJ in the real space. Then, the image processing apparatus 100D can reduce the amount of calculation for calculating the speed of the subject OBJ in the real space compared to the conventional case.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance DIS between the imaging device and the subject OBJ from the two frames taken by one imaging device, the speed of the subject OBJ in real space is the same or similar to that of the image processing device 100D. It is difficult to calculate with.

  FIG. 19 shows another embodiment of the image processing apparatus and the image processing method. The same or similar elements as those described in FIGS. 2 and 8 are denoted by the same or similar reference numerals, and detailed description thereof will be omitted. The dashed arrows in FIG. 19 indicate the flow of information such as signals.

  The image processing apparatus 100E includes a control unit CNTL, a parallax calculation unit 10d, a distortion calculation unit 20d, a speed calculation unit 30d, a recording unit REC, a display unit DIP, memories MEM1, MEM2, and a bus BUS. The control unit CNTL, the parallax calculation unit 10d, the distortion calculation unit 20d, the speed calculation unit 30d, the recording unit REC, the display unit DIP, and the memories MEM1 and MEM2 are connected to the bus BUS.

  The parallax calculation unit 10d receives images IMG1 and IMG2 captured by the imaging elements ISEN1 and ISEN2 from the memory MEM1 via the bus BUS. The parallax calculation unit 10d calculates a parallax DPF indicating the difference between the position of the subject OBJ1 on the image IMG1 and the position of the subject OBJ2 on the image IMG2 for each macroblock MBb of the image IMG2. The parallax calculation unit 10d is an example of a first calculation unit. The operation of the parallax calculation unit 10d will be described with reference to FIG.

  The distortion calculation unit 20d receives the images IMG1 and IMG2 from the memory MEM1 via the bus BUS. The distortion calculation unit 20d generates distortion information INF indicating the difference in shape between the subject OBJ1 of the image IMG1 and the subject OBJ2 of the image IMG2 generated by the imaging elements ISEN1 and ISEN2 because the scanning directions are opposite to each other. Calculated for each block MBb. The distortion calculation unit 20d is an example of a second calculation unit. The distortion calculation unit 20d outputs the calculated distortion information INF to the memory MEM2. The operation of the distortion calculation unit 20d will be described with reference to FIG.

  The speed calculation unit 30d is an example of a third calculation unit that receives parallax DPF, distortion information INF, readout time information TINF, and the like from the memory MEM2 via the bus BUS and calculates the speed VL of the subject OBJ in real space. The speed calculation unit 30d outputs the calculated speed VL to the recording unit REC and the display unit DIP. The operation of the speed calculation unit 30d will be described with reference to FIG.

  FIG. 20 shows an example of images IMG1 and IMG2 taken by the imaging elements ISEN1 and ISEN2 shown in FIG. 19 based on the timing chart shown in FIG. In FIG. 20, the subject OBJ moves, for example, in an oblique direction OD (that is, the row arrangement direction OD1 and the row direction OD2) within the angle of view. Hereinafter, the row arrangement direction OD1 is also referred to as a column direction OD1. Further, in FIG. 20, in order to make the drawing easier to see, the number of pixel regions AR is three (n = 3) as in FIG. Furthermore, in the images IMG1 and IMG2, as in FIG. 5, the parallax calculation unit 10d divides each pixel area AR into k columns in the row direction (horizontal direction in FIG. 20). That is, the image IMG1 is divided into image areas MBa (MBa11-MBa15, MBa21-MBa25, MBa31-MBa35). The image IMG2 is divided into image areas MBb (MBb11-MBb15, MBb21-MBb25, MBb31-MBb35).

  In FIG. 20, since the subject OBJ moves in the row direction OD2, the subject OBJ1 on the image IMG1 shifts to the right with respect to the original shape indicated by the dashed rectangle as it goes below the angle of view as in FIG. . On the other hand, the subject OBJ2 on the image IMG2 shifts to the right side with respect to the original shape indicated by a broken-line rectangle as it goes above the angle of view. In FIG. 20, since the subject OBJ moves in the column direction OD1, the subject OBJ1 on the image IMG1 has a shape extending downward with respect to the original shape indicated by a broken-line rectangle as in FIG. On the other hand, the subject OBJ2 on the image IMG2 shows a shape contracted in the vertical direction with respect to the original shape indicated by a broken-line rectangle. As described with reference to FIG. 1, the subject OBJ1 on the image IMG1 and the subject OBJ2 on the image IMG2 are displaced in the left-right direction because the subject OBJ is imaged through the lenses LEN1 and LEN2 arranged at different positions. Caused by.

  For example, the parallax calculation unit 10d smoothes the pixel values of the images IMG1 and IMG2 for each macroblock MBa and MBb, for example. The parallax calculation unit 10d generates a template between the macroblock MBb to be processed in the smoothed image IMG2 and each macroblock MBa in the pixel area AR that includes the macroblock MBb to be processed in the smoothed image IMG1. Image processing such as matching is executed. Then, the parallax calculation unit 10d calculates an SAD value indicating the degree of coincidence between the processing target macroblock MBb and each macroblock MBa. The parallax calculation unit 10d extracts the macroblock MBa of the image IMG1 that indicates the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2). In FIG. 20, when the macro block MBb14 in the pixel area AR1 in the image IMG2 is the processing target, the parallax calculation unit 10d calculates the minimum SAD value in the macro block MBa12 in the pixel area AR1 in the image IMG1. The parallax calculation unit 10d extracts the macroblock MBa12. That is, the subject OBJ1 of the macroblock MBa12 shown in FIG. 20 corresponds to the subject OBJ2 of the macroblock MBb14.

  FIG. 21 shows an example of calculation processing in the distortion calculation unit 20d shown in FIG. In the example illustrated in FIG. 21, the distortion calculation unit 20d uses the macroblock MBb14 of the image IMG2 illustrated in FIG. 20 and the macroblock MBa12 of the image IMG1 corresponding to the macroblock MBb14, and the difference in the shapes of the subjects OBJ1 and OBJ2 Is calculated. Note that the distortion calculation unit 20d performs the same or similar calculation process as that of the macroblock MBb14 on other macroblocks MBb other than the macroblock MBb14.

  For example, the distortion calculation unit 20d executes a deformation process in which the macroblock MBb14 is subjected to shear deformation and expansion / contraction so that the shaded subject OBJ2 matches the shape of the subject OBJ1 of the macroblock MBa12. For example, the distortion calculation unit 20d performs shear deformation and expansion / contraction deformation processing on the macroblock MBb14 using Expression (16) including an affine transformation matrix, and obtains the macroblock MBe indicated by the broken line subjected to the deformation processing.

(X, y) indicates the pixel position of the macroblock MBb14, and (xa, ya) indicates the pixel position of the transformed macroblock MBe. That is, Expression (16) is a combination of Expression (1) indicating shear deformation and Expression (7) indicating expansion / contraction processing.

  The strain calculation unit 20d executes, for example, image processing such as template matching between the macroblock MBe and the macroblock MBa12 that are deformed by changing the shear rate α, the expansion / contraction rate β, and the variables Tx and Ty, and the SAD value is calculated. calculate. When the SAD value indicates the minimum value, the distortion calculation unit 20d determines that the shape of the subject OBJ2 subjected to the deformation process most closely matches the shape of the subject OBJ1. Then, the strain calculation unit 20d calculates, as strain information INF, the shear rate α and the expansion / contraction rate β when the SAD value shows the minimum value. Also, the parallax calculation unit 10d sets the variable Tx when the SAD value indicates the minimum value as the parallax DPF.

  Note that the strain calculation unit 20d may replace the variable Tx in Expression (16) with the shear rate α using Expression (2) to reduce the number of variables to be changed in the deformation process in Expression (16). In this case, the distortion calculation unit 20d can reduce the amount of calculation in the deformation process.

  Alternatively, similarly to the distortion calculation unit 20c illustrated in FIG. 12, the distortion calculation unit 20d may detect the edges of the subjects OBJ1 and OBJ2 in advance from changes in luminance in the arrangement direction of the rows of the macroblocks MBa and MBb. . In this case, the distortion calculation unit 20d calculates the expansion / contraction ratio β from the ratio of the distances between the edges in the row arrangement direction of the detected subjects OBJ1 and OBJ2, thereby changing the variable to be changed by the deformation process using Expression (16). And the amount of calculation in the deformation process can be reduced.

  The speed calculation unit 30d calculates the speed VL of the subject OBJ in real space using the parallax DPF calculated by the parallax calculation unit 10d, the distortion information INF calculated by the distortion calculation unit 20d, and the readout time information TINF. For example, the speed calculation unit 30d calculates the speed VL of the subject OBJ by dividing it into components of the column direction OD1 and the row direction OD2. That is, the speed calculation unit 30d calculates the component in the column direction OD1 of the speed VL from the equations (3) to (6) using the parallax DPF, the shear rate α, and the readout time information TINF. Further, the speed calculation unit 30d calculates a component in the row direction OD2 of the speed VL from the expressions (6), (9), and (10) using the parallax DPF, the expansion / contraction rate β, and the readout time information TINF. Then, the speed calculation unit 30d performs vector synthesis on the calculated components in the column direction OD1 and the row direction OD2, and calculates the speed VL of the subject OBJ in the real space.

  Note that when the distortion calculation unit 20d detects the edges of the subjects OBJ1 and OBJ2, the speed calculation unit 30d uses the column of the speeds VL of the subject OBJ from Equation (15) as in the case of the speed calculation unit 30c illustrated in FIG. A component in the direction OD1 may be calculated. In this case, the image processing apparatus 100D can reduce the amount of calculation for calculating the enlargement / reduction ratio β in Expression (16).

  FIG. 22 shows an example of image processing in the image processing apparatus 100E shown in FIG. The processing shown in FIG. 22 is realized by the control unit CNTL executing an image processing program. Further, the processing illustrated in FIG. 22 may be executed by hardware provided in the image processing apparatus 100E. In this case, the parallax calculation unit 10d, the distortion calculation unit 20d, and the speed calculation unit 30d illustrated in FIG. 19 are realized by a circuit arranged in the image processing apparatus 100E.

  In step S400, the parallax calculation unit 10d acquires images IMG1 and IMG2 captured by the imaging device CAM2 from the memory MEM1.

  In step S410, the parallax calculation unit 10d extracts the macro block MBa of the image IMG1 corresponding to the processing target macro block MBb of the image IMG2. For example, the parallax calculation unit 10d smoothes the pixel values of the images IMG1 and IMG2 for each of the macroblocks MBa and MBb. The parallax calculation unit 10d selects the macro block MBb to be processed from the smoothed image IMG2. The parallax calculation unit 10a performs image processing such as template matching between the selected processing target macroblock MBb and each macroblock MBa in the pixel area AR in which the processing target macroblock MBb is included in the smoothed image IMG1. To calculate the SAD value. The parallax calculation unit 10d extracts the macroblock MBa of the image IMG1 that indicates the minimum SAD value (that is, the image region that most closely matches the processing target macroblock MBb of the image IMG2).

  In step S420, the distortion calculation unit 20d calculates a distortion information INF by performing a deformation process on the processing target macroblock MBb of the image IMG2. For example, the distortion calculation unit 20d changes the shear rate α and the expansion / contraction rate β in Equation (16) so that the shape of the subject OBJ2 matches the shape of the subject OBJ1 of the macroblock MBa extracted in step S410. The transformation process is executed on the macro block MBb to be processed. The distortion calculation unit 20d performs image processing such as template matching between the macro block MBb to be processed and the macro block MBa extracted in step S410, and calculates the SAD value. When the SAD value indicates the minimum value, the distortion calculation unit 20d determines that the shape of the subject OBJ2 subjected to the deformation process most closely matches the shape of the subject OBJ1. Then, the strain calculation unit 20d calculates, as strain information INF, the shear rate α and the expansion / contraction rate β when the SAD value shows the minimum value. The distortion calculation unit 20d outputs the calculated distortion information INF to the memory MEM2. Further, the parallax calculation unit 10d uses the variable Tx when the SAD value indicates the minimum value as the parallax DPF, and outputs the parallax DPF to the memory MEM2.

  In step S430, the speed calculation unit 30d uses the parallax DPF, the distortion information INF, and the readout time information TINF calculated in step S420 to calculate the speed VL of the subject OBJ1 in the real space from Expression (3) -Expression (6). Of these, the component in the column direction OD1 is calculated.

  In step S440, the speed calculation unit 30d uses the parallax DPF, the distortion information INF, and the readout time information TINF calculated in step S420, and the subject in the real space from the expressions (6), (9), and (10). Of the speed VL of OBJ1, the component in the row direction OD2 is calculated.

  In step S450, the speed calculation unit 30d performs vector synthesis of the component in the column direction OD1 calculated in step S430 and the component in the row direction OD2 calculated in step S440, and calculates the speed VL of the subject OBJ in real space.

  In step S460, control unit CNTL determines whether or not next macroblock MBb is present in image IMG2. When the control unit CNTL determines that there is a macroblock MBb to be processed next, the processing of the image processing device 100E proceeds to step S410. On the other hand, when the control unit CNTL determines that there is no macro block MBb to be processed next, the processing of the image processing apparatus 100E ends.

  As described above, in the embodiment shown in FIGS. 19 to 22, the reading of the pixel signals from the imaging elements ISEN1 and ISEN2 is executed with the scanning directions in the row arrangement direction reversed for each pixel area AR. Accordingly, the subject OBJ1 on the image IMG1 photographed by the image sensor ISEN1 and the subject OBJ2 on the image IMG2 photographed by the image sensor ISEN2 have different shapes (rolling shutter distortion) for each pixel area AR. Accordingly, the image processing apparatus 100E executes shear deformation and enlargement / reduction processing for each macroblock MBb on the subject OBJ2 on the image IMG2, and matches the shape of the subject OBJ2 to the shape of the subject OBJ1, thereby disparity DPF And distortion information INF is calculated. The image processing apparatus 100E calculates the speed VL of the subject OBJ in the real space using the calculated parallax DPF, distortion information INF, and readout time information TINF.

  Thereby, the image processing apparatus 100E can calculate the speed VL of the subject OBJ in the real space from two still images (images IMG1 and IMG2) obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. . That is, the image processing apparatus 100E can reduce the amount of image data stored in the memory MEM1 as compared to the case where at least four frames captured by a stereo camera having two imaging elements having the same scanning direction are used.

  Further, the image processing apparatus 100E can calculate the speed VL of the subject OBJ in the real space by using the two images IMG1 and IMG2 obtained by one imaging by the imaging elements ISEN1 and ISEN2 in the imaging apparatus CAM2. That is, the image processing apparatus 100E does not have to cause the imaging device CAM2 to capture images twice or more in order to calculate the speed VL of the subject OBJ in the real space. Then, the image processing apparatus 100E can reduce the amount of calculation for calculating the speed of the subject OBJ in the real space compared to the conventional case.

  Note that the speed of the subject OBJ may be calculated even when two frames of a moving image captured by one imaging device having one imaging element are used. However, since it is difficult to calculate the distance DIS between the imaging device and the subject OBJ from the two frames taken by one imaging device, the speed of the subject OBJ in real space is the same as or similar to that of the image processing device 100E. It is difficult to calculate with.

  From the above detailed description, features and advantages of the embodiments will become apparent. This is intended to cover the features and advantages of the embodiments described above without departing from the spirit and scope of the claims. Also, any improvement and modification should be readily conceivable by those having ordinary knowledge in the art. Therefore, there is no intention to limit the scope of the inventive embodiments to those described above, and appropriate modifications and equivalents included in the scope disclosed in the embodiments can be used.

Regarding the above embodiment, the following additional notes are disclosed.
(Appendix 1)
The first imaging device includes a first imaging device and a second imaging device in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted, and the readout of the plurality of rows is the first imaging device. In an image processing device that calculates the speed of a subject from an image captured by an imaging device that is opposite to each other by an element and the second imaging device,
Using the first image read from the first image sensor and the second image read from the second image sensor, the first subject that is the subject on the first image A first calculator that calculates a parallax indicating a difference between a position and a position of a second subject that is a subject on the second image;
A second calculation unit that calculates distortion information that is a difference in shape between the first subject and the second subject;
An image processing apparatus comprising: a third calculation unit that calculates the speed of the subject using the parallax and the distortion information.
(Appendix 2)
In the image processing apparatus according to attachment 1,
The plurality of pixels included in the first image sensor and the plurality of pixels included in the second image sensor are divided into a plurality of pixel regions in the row arrangement direction, and the pixel signal is transmitted for each pixel region. An image processing apparatus comprising: a control unit that causes the imaging apparatus to execute a driving process in which the first imaging element and the second imaging element reverse scanning directions that are sequentially read out each time. .
(Appendix 3)
In the image processing device according to attachment 2,
The first calculation unit includes a second image area on the second image obtained by dividing each pixel area in a row direction, and an area on the first image corresponding to the second image area. Calculating the difference in position from the first image region for each second image region as the parallax;
The second calculation unit performs a deformation process for deforming the second subject so that the shape of the second subject matches the shape of the first subject, thereby obtaining the distortion information for each second image region. Calculate
The third calculation unit uses the parallax and the distortion information to calculate the speed of the subject for each second image region.
(Appendix 4)
In the image processing device according to attachment 3,
The second calculation unit executes shear deformation on the second subject as the deformation processing, and calculates a deformation amount of the second subject due to the shear deformation as the distortion information. apparatus.
(Appendix 5)
In the image processing apparatus according to appendix 3 or appendix 4,
The second calculation unit performs an enlargement / reduction process for enlarging or reducing the second subject as the deformation process, and calculates a deformation amount of the second object by the enlargement / reduction process as the distortion information. An image processing apparatus.
(Appendix 6)
The first imaging device includes a first imaging device and a second imaging device in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted, and the readout of the plurality of rows is the first imaging device. In an image processing method for calculating a speed of a subject from an image captured by an imaging device that is opposite to each other by an element and the second imaging element,
Using the first image read from the first image sensor and the second image read from the second image sensor, the first subject that is the subject on the first image Calculating a parallax indicating a difference between a position and a position of a second subject which is a subject on the second image;
Calculating distortion information which is a difference in shape between the first subject and the second subject;
An image processing method, wherein the speed of the subject is calculated using the parallax and the distortion information.
(Appendix 7)
In the image processing method according to attachment 6,
The plurality of pixels included in the first image sensor and the plurality of pixels included in the second image sensor are divided into a plurality of pixel regions in the row arrangement direction, and the pixel signal is transmitted for each pixel region. An image processing method, comprising: causing the imaging apparatus to execute a driving process in which the first imaging element and the second imaging element have opposite scanning directions sequentially read each time.
(Appendix 8)
In the image processing method according to attachment 7,
The process of calculating the parallax is performed on the first image corresponding to the second image area on the second image and the second image area obtained by dividing each pixel area in the row direction. Calculating the difference in position from the first image region for each second image region as the parallax;
The process of calculating the distortion information is performed by performing a deformation process for deforming the second subject so that the shape of the second subject matches the shape of the first subject. To
The process for calculating the speed of the subject uses the parallax and the distortion information, and calculates the speed of the subject for each second image region.
(Appendix 9)
In the image processing method according to attachment 8,
The processing for calculating the strain information is performed by performing shear deformation on the second subject as the deformation processing, and calculating a deformation amount of the second subject due to the shear deformation as the strain information. Processing method.
(Appendix 10)
In the image processing apparatus according to appendix 8 or appendix 9,
The process of calculating the distortion information includes performing an enlargement / reduction process for enlarging or reducing the second subject as the deformation process, and calculating a deformation amount of the second subject by the enlargement / reduction process as the distortion information. A featured image processing apparatus.

10, 10a, 10b, 10c, 10d ... parallax calculation unit; 20, 20a, 20b, 20c, 20d ... distortion calculation unit; 30, 30a, 30b, 30c, 30d ... speed calculation unit; 100, 100A, 100B, 100C, 100D, 100E: Image processing device; AR1-ARn: Pixel region; CAM, CAM2: Imaging device; BUS ... Bus; CAL ... Calculation unit; CNTL ... Control unit; DIP ... Display unit; DPF ... Parallax; IMG1, IMG2 ... image; INF ... distortion information; ISEN1, ISEN2 ... imaging element; LEN1, LEN2 ... lens; MBa11-MBa15, MBa21-MBa25, MBa31-MBa35, MBb11-MBb15, MBb21-MBb25, MBb31-MBb35, MBc, MBd, MBe ... Macroblock; MEM Storage unit; MEM1, MEM2 ... memory; OBJ, OBJ1, OBJ1a, OBJ1b, OBJ2, OBJ2a, OBJ2b, OBJ2c ... subject; REC ... recording unit; TINF ... read time information

Claims (6)

The first imaging device includes a first imaging device and a second imaging device in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted, and the readout of the plurality of rows is the first imaging device. In an image processing device that calculates the speed of a subject from an image captured by an imaging device that is opposite to each other by an element and the second imaging device,
Using the first image read from the first image sensor and the second image read from the second image sensor, the first subject that is the subject on the first image A first calculator that calculates a parallax indicating a difference between a position and a position of a second subject that is a subject on the second image;
A second calculation unit that calculates distortion information that is a difference in shape between the first subject and the second subject;
An image processing apparatus comprising: a third calculation unit that calculates the speed of the subject using the parallax and the distortion information. The image processing apparatus according to claim 1.
The plurality of pixels included in the first image sensor and the plurality of pixels included in the second image sensor are divided into a plurality of pixel regions in the row arrangement direction, and the pixel signal is transmitted for each pixel region. An image processing apparatus comprising: a control unit that causes the imaging apparatus to execute a driving process in which the first imaging element and the second imaging element reverse scanning directions that are sequentially read out each time. . The image processing apparatus according to claim 2,
The first calculation unit includes a second image area on the second image obtained by dividing each pixel area in a row direction, and an area on the first image corresponding to the second image area. Calculating the difference in position from the first image region for each second image region as the parallax;
The second calculation unit performs a deformation process for deforming the second subject so that the shape of the second subject matches the shape of the first subject, thereby obtaining the distortion information for each second image region. Calculate
The third calculation unit uses the parallax and the distortion information to calculate the speed of the subject for each second image region. The image processing apparatus according to claim 3.
The second calculation unit executes shear deformation on the second subject as the deformation processing, and calculates a deformation amount of the second subject due to the shear deformation as the distortion information. apparatus. The image processing apparatus according to claim 3 or 4,
The second calculation unit performs an enlargement / reduction process for enlarging or reducing the second subject as the deformation process, and calculates a deformation amount of the second object by the enlargement / reduction process as the distortion information. An image processing apparatus. The first imaging device includes a first imaging device and a second imaging device in which readout of a plurality of rows of a pixel array including a plurality of pixels arranged in a matrix is sequentially shifted, and the readout of the plurality of rows is the first imaging device. In an image processing method for calculating a speed of a subject from an image captured by an imaging device that is opposite to each other by an element and the second imaging element,
Using the first image read from the first image sensor and the second image read from the second image sensor, the first subject that is the subject on the first image Calculating a parallax indicating a difference between a position and a position of a second subject which is a subject on the second image;
Calculating distortion information which is a difference in shape between the first subject and the second subject;
An image processing method, wherein the speed of the subject is calculated using the parallax and the distortion information.

Images