JP4082410B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method and an image processing apparatus that restore an object as a three-dimensional solid object by stereo matching.

  In recent years, with the development of image processing technology, there is a method for obtaining a three-dimensional model of an object by image processing based on a plurality of images obtained by photographing the object from different directions. As such an image processing method, for example, there is a three-dimensional model generation method disclosed in Japanese Patent Laid-Open No. 2000-99760 (page 3, FIG. 2). In this three-dimensional model generation method, a stereo image is obtained by observing an object from a plurality of different directions, a three-dimensional outline of the object is generated based on the stereo image, and a three-dimensional region (texture region or shading region) is generated. ) Is generated. Based on the 3D contour information and the 3D region information, if there is something that matches the existing 3D model, these information is updated. Register as a dimensional model. If the entire three-dimensional model of the object is not completed, the entire model is generated by moving the observation direction.

  In the three-dimensional model generation method, it is necessary to ensure the continuity of the contour line when restoring the three-dimensional contour line. For this reason, in the conventional model generation method, a net-like matching area is set for an object, and the continuity of the contour line is obtained by fitting the above-described net-like matching area to the object surface using a genetic algorithm. Yes.

  When the continuity of the contour line is obtained by the above-described method, there is a problem that it is difficult to smoothly restore a curved surface or the like because only the net-like matching region is fitted. In theory, since this method can be used only when the parallax is small, there is a problem that this method cannot be used in a stereo camera that captures a close distance, such as the eyes of a robot. Furthermore, since many repeated calculations are required, there is a problem that processing is slow.

  An object of the present invention is to provide an image processing method and apparatus capable of smoothly restoring a curved surface or the like when an object is restored. It is another object of the present invention to provide an image processing method and apparatus that can be used for a stereo camera that captures a close distance, has a small amount of calculation processing, and requires a short processing time.

  An image processing method according to the present invention that solves the above-mentioned problems is the same image is taken by the standard image photographing means and the reference image photographing means, and is photographed by the standard image and the reference image photographing means that are photographed by the standard image photographing means. The shape of the object is restored by stereo matching using the reference image. Then, the image processing method of the present invention sets a plurality of windows associated with each of the base image and the reference image based on a predetermined correlation, and deforms the plurality of windows in the reference image, The separation or overlap between adjacent windows in the reference image is eliminated, and the shape of the object is restored by performing stereo matching between the reference image window and the reference image window.

  A plurality of windows are set for each of the reference image and the reference image based on a predetermined correlation. The plurality of windows can be formed in an aligned state in the standard image, but is usually set in a messy state in a reference image associated with the aligned windows. Therefore, in the present invention, a plurality of windows in the reference image are deformed to eliminate separation or overlap between adjacent windows. For this reason, when restoring the object, the curved surface or the like can be restored smoothly. Further, since there is no inconvenience that the parallax is limited to a small parallax, stereo matching can be performed even with a stereo camera that captures a close distance.

  Here, when deforming adjacent windows, it is preferable to move the corners of the windows toward the corners of the adjacent windows.

  In this way, when the window is deformed, the opposing corners of adjacent windows are moved toward their middle part to expand or contract the window, thereby suitably expanding or contracting the window with a small amount of computation. It can be carried out. As a result, the image processing time can be shortened.

  Alternatively, a new window group is formed by a line connecting the centroid (or center) of the window on the reference screen, and a new line on the reference image is formed by a line connecting the centroid (or center) of the window on the reference image. A deformed window group corresponding to a different window group may be formed.

  In this way, a new window is formed on the reference screen and a reference window corresponding to the reference screen by a line connecting the centroids (or centers) of adjacent windows, and the reference image windows are not separated and overlapped. By adopting the deformed state, the window can be expanded and contracted with a smaller amount of calculation, and the processing time can be shortened.

Alternatively, when deforming the adjacent window, within the boundary region including each corner of the adjacent window, the unified joint point of the adjacent windows is obtained based on the correlation between the adjacent windows , and this unified joint point The window may be expanded or contracted by moving the corner of the window on the basis of. By doing in this way, the junction point of an adjacent window can be calculated | required more correctly.

  Further, each window set on the standard image and each window on the reference image corresponding to each window on the standard screen are divided into the same number to form divided windows, respectively. The divided windows on the reference image and the divided windows on the reference image are associated with each other based on a predetermined correlation, and the divided windows on the reference image are deformed to eliminate separation or overlap between the divided windows on the reference image. May be.

  In this way, by dividing the window on the standard image and the window on the reference image to form divided windows and associating them, the window can be miniaturized. As a result, the shape of the object can be restored more accurately.

  The image processing apparatus of the present invention includes a window setting unit that sets a plurality of windows for a standard image and a reference image of the same object captured by the standard image capturing unit and the reference image capturing unit, respectively, and a standard An adjacent window on the reference image is created by associating the reference image with the reference image based on the relationship between the arrangement order of the windows on the image and the arrangement order of the windows on the reference image, and deforming each window on the reference image A stereo matching unit is provided that restores the shape of the object by performing stereo matching between the window of the reference image and the window of the reference image after eliminating the separation or overlap between them.

  Here, the stereo matching unit may perform a process of moving the corners of the windows toward the corners of the adjacent windows on the reference image.

  In addition, the window deforming unit may perform a process of deforming the window with reference to a line connecting the centroids of adjacent windows.

  Alternatively, the stereo matching unit forms a new window group by a line connecting the centroids (or centers) of the windows on the reference screen, and a new line on the reference image by a line connecting the centroids of the windows on the reference image. You may make it perform the process which forms the window group after a deformation | transformation corresponding to a window group.

  Further, there is provided window dividing means for dividing each window set on the standard image and each window on the reference image corresponding to each window on the quasi-screen into the same number to form divided windows. Further, it may be provided. At this time, the window matching means associates the divided window in the base image divided by the window dividing means with the divided window in the reference image based on a predetermined correlation, and refers to the deformed window in the reference image by deforming it. The separation or overlap between the divided windows in the image is eliminated.

FIG. 1 is a block diagram of an image processing apparatus according to the present invention.
FIG. 2 is a flowchart illustrating a process of image processing by the image processing apparatus.
FIG. 3 is a flowchart showing details of a process of performing window deformation in the image processing process according to the present embodiment.
FIG. 4A is a diagram illustrating a reference image obtained by photographing an object.
FIG. 4B is a diagram illustrating a reference image obtained by photographing an object.
FIG. 5A is an explanatory diagram of a reference drawing for explaining a procedure for deforming a window.
FIG. 5B is an explanatory diagram of a reference screen following FIG. 5A.
FIG. 5C is an explanatory diagram of a reference screen following FIG. 5B.
FIG. 6A is a diagram illustrating a reference image for explaining a process of deforming the window, and is a diagram illustrating the middle of the deformation of the window.
FIG. 6B is a diagram showing a reference image for explaining the process of deforming the window, and is a diagram showing the end of the deformation of the window.
FIG. 7 is a flowchart showing a process of subdividing the deformed window.
FIG. 8A is a diagram for explaining a process of dividing a window and shows a reference image.
FIG. 8B is a diagram for explaining the process of dividing the window, and shows a reference image.
FIG. 9A is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image before division.
FIG. 9B is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image before division.
FIG. 9C is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image after division.
FIG. 9D is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image after division.
FIG. 9E is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image after further transformation after division.
FIG. 9F is a diagram showing a screen at the time of window division transformation, and is a diagram showing a reference image further transformed after division.
FIG. 10A is a diagram illustrating a process of deforming a divided window, and is a diagram illustrating a reference image.
FIG. 10B is a diagram illustrating a process of deforming the divided windows, and is a diagram illustrating a reference image before deformation.
FIG. 10C is a diagram illustrating a process of deforming the divided windows, and is a diagram illustrating the reference image after the modification.
FIG. 11A is a diagram illustrating a process of deforming a window in the second embodiment, and is a diagram illustrating a reference image.
FIG. 11B is a diagram illustrating a process of deforming a window in the second embodiment, and is a diagram illustrating a reference image before division.
FIG. 11C is a diagram illustrating a process of deforming a window in the second embodiment, and is a diagram illustrating a reference image after division.
FIG. 11D is a diagram illustrating a process of deforming a window in the second embodiment, and is a diagram illustrating a range in which a corresponding image is searched for in a reference image.
FIG. 12A is a diagram illustrating a state where the window is deformed, and is a diagram illustrating a reference image.
FIG. 12B is a diagram illustrating a state where the window is deformed, and is a diagram illustrating a reference image.
FIG. 13A is a diagram for explaining a process of deforming a divided window, and is a diagram showing a reference image.
FIG. 13B is a diagram for explaining a process of deforming the divided window, and is a diagram illustrating a reference image before division.
FIG. 13C is a diagram for explaining a process of deforming the divided window, and is a diagram illustrating the reference image after division.
FIG. 14 is an explanatory diagram for explaining a search range of a small window used when a window is deformed.
FIG. 15A is a diagram illustrating a state where the window is deformed, and is a diagram illustrating a reference image.
FIG. 15B is a diagram illustrating a state where the window is deformed, and is a diagram illustrating a reference image.
FIG. 16A is a diagram illustrating a process of deforming a window in the third embodiment, and is a diagram illustrating a reference image.
FIG. 16B is a diagram illustrating a process of deforming the window in the third embodiment, and is a diagram illustrating a reference image before deformation.
FIG. 16C is a diagram illustrating a process of deforming a window in the third embodiment, and is a diagram illustrating a modified reference image.
FIG. 17A is a diagram for explaining window division in the third embodiment, and is a diagram of a reference image.
FIG. 17B is a diagram illustrating window division according to the third embodiment, and is a diagram of a reference image.
FIG. 18A is a diagram for explaining a range for searching for a junction point of divided windows, and a diagram for explaining the search range.
FIG. 18B is a diagram for explaining a range for searching for junction points of divided windows, and is a diagram for explaining points that become junction points.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a block diagram of an image processing apparatus according to the first embodiment of the present invention. As shown in FIG. 1, a standard image camera 2, a reference image camera 3, and a monitor 4 are connected to the image processing apparatus 1 according to the present embodiment. In addition, the image processing apparatus 1 includes a preprocessing unit 11, a window generation unit 12, and a stereo matching unit 13. The window generation unit 12 is provided with a window deformation unit (not shown) which is a window deformation unit of the present invention. The image processing device 1, the cameras 2 and 3, and the monitor 4 are attached to a vehicle (not shown), for example. Specifically, for example, the image processing apparatus 1 is installed at an appropriate position in the passenger compartment. Further, the standard image camera 2 and the reference image camera 3 are provided at positions near each other on the side surface of the vehicle, for example. Furthermore, the monitor 4 is attached to, for example, an instrument panel where the driver's field of view reaches, for example, in the passenger compartment.

  The reference image camera 2 and the reference image camera 3 are both CCD cameras, and take the same object as an analog image. Analog images taken by the reference image camera 2 and the reference image camera 3 are output to the preprocessing unit 11 in the image processing apparatus 1. The analog image output to the image processing apparatus 1 is subjected to predetermined image processing by the image processing apparatus 1 and then the monitor 4 as a projection image composed of a digital image (an image obtained by displaying a three-dimensional object on a two-dimensional plane). Is output. The image processing procedure in the image processing apparatus 1 will be described later. The monitor 4 displays the projection image stereo-matched and output from the image processing apparatus 1 on the screen.

  Next, the procedure of image processing by the image processing apparatus 1 will be described. FIG. 2 is a flowchart illustrating a process of image processing by the image processing apparatus. Here, the overall flow of image processing will be described.

  In order to perform image processing in the image processing apparatus 1, the same object is photographed by the standard image camera 2 and the reference image camera 3. The reference image obtained by photographing the object with the reference image camera 2 is output to the preprocessing unit 11 in the image processing apparatus 1. Similarly, the reference image obtained by photographing the object with the reference image camera 3 is output to the preprocessing unit 11 in the image processing apparatus 1. Thus, image processing by the image processing apparatus 1 is started.

  When image processing is started, the pre-processing unit 11 performs camera calibration on the standard image and the reference image as pre-processing (S1), and then parallelizes the standard image and the reference image (S2). . The preprocessed image is output from the preprocessing unit 11 to the window generation unit 12.

  The window generation unit 12 performs region division that divides each of the base image and the reference image into a plurality of castles (S3). Subsequently, each area divided in the standard image and the reference image is set as a window (S4). When the window is set with the reference image and the reference image, each image is output to the stereo matching unit 13, and correlation (window matching) is performed by performing a correlation operation between the window in the reference image and the window in the reference image. (S5). In this correlation calculation, the correlation is determined such that the correlation is performed when a certain threshold value such as color, brightness, brightness, or the like in each window is exceeded.

  Subsequently, the arrangement order of the windows in the reference image is corrected (S6). The correction of the arrangement order is to detect the miscorresponding window that is different from the arrangement order of multiple windows in the standard image as a result of the association among the plurality of windows in the reference image, and correct the arrangement order of the miscorresponding windows. Do. After this correction, the correction of the arrangement order of the windows in the reference image is completed by re-searching a window whose miscorrespondence has not been corrected yet (S6). After the window arrangement order is corrected in this way, the window is deformed to fill the gaps between the windows in the reference image (S7). Details of the deformation of this window will be described later.

  When the deformation of the window is completed, it is determined whether or not all the windows are in one plane (S8). As a result, when all the windows are not one plane, the window (corresponding window) deviated from the same plane is subdivided (S9). Details of this subdivision will be described later. After the relevant window is subdivided, correlation calculation is performed again (S10). Then, the process returns to step S7, the window is deformed, and it is determined again whether or not all the windows are one plane (S8). After repeating this process, if step S8 is affirmed, the three-dimensional coordinates of the four corners of each window are calculated (S11).

  In the stereo matching unit 13, solid primitive fitting (representing a solid as a combination of basic solids called primitives such as a sphere, a rectangular parallelepiped, and a cone) is performed based on the three-dimensional coordinates of the four corners of each window (S12). ). And a projection image is produced | generated (S13) and an image process is complete | finished. The projection image generated in this way is output to the monitor 4 and displayed.

  In the above image processing step, the present embodiment is characterized by window deformation (S7) in the reference image. Hereinafter, this characteristic process will be further described.

  FIG. 3 is a flowchart showing details of a process of performing window deformation in the image processing process according to the present embodiment.

  As shown in FIG. 3, when the window deformation process is started, first, windows corresponding to a plurality of windows set as the standard image are displayed on the reference image (S21). As shown in FIG. 4A, a plurality of windows W1, W1,... Surrounding the automobile image V1 are displayed in the reference image. In the reference image, as shown in FIG. 4B, a plurality of windows W2, W2,... Surrounding the vehicle image V2 are displayed. The windows W1 displayed on the reference image are quadrangular (square) and all have the same shape. Further, the window W2 displayed on the reference image is the one before the deformation, and each is a quadrangle (square) having the same shape as the window W1 displayed on the standard image. The plurality of windows W2, W2,... On the reference image shown in FIG. 4B are corrected in the arrangement order in step S6, and correspond to the plurality of windows W1, W1,... On the reference image shown in FIG. They are in order.

  The windows W2, W2,... Displayed on the reference image in this way are separated from each other or overlapped by the adjacent windows W2, W2. Since it is difficult to restore the image V2 as a three-dimensional object as it is, the windows W2, W2,... Displayed in the reference image are appropriately expanded or contracted (deformed) in order to eliminate the separation and overlap (S22). .

  Here, the procedure for expanding and contracting the window will be described. Assume that the reference image includes windows WA, WB, and WC shown in FIG. 5A, for example. Here, since the windows WA and WB are overlapped and the WB and WC are separated from each other, the windows WA and WB are contracted so that the windows WA and WB are not overlapped, and a portion separated between the windows WB and WC is formed. Both windows WB and WC are expanded so as to disappear. As for the windows WA and WB, the windows WA and WB are contracted toward the midpoints C1 and C2 where the windows WA and WB overlap. Thus, the ends of the windows WA and WB are joined at the midpoints C1 and C2, respectively, and the windows WA and WB are made continuous. On the other hand, for the windows WB and WC, both the windows WB and WC are expanded toward the middle points C3 and C4 of the separated portion between the windows WB and WC. Thus, the ends of the windows WB and WC are joined at the midpoints C3 and C4, respectively, and the windows WB and WC are made continuous.

  FIG. 6A shows a state in which the windows W2, W2,... Of the reference image shown in FIG. As shown in FIG. 6A, the ends of the windows W2, W2,... Are joined by expanding and contracting the windows W2, W2,. In this way, it is possible to fill a gap between the windows W2, W2,.

  By the way, by expanding and contracting the windows W2, W2,... Of the reference image, the gap between the windows W2, W2 can be filled, but the end portions of the windows W2, W2 adjacent to each other in the column direction (height direction in FIG. 6B). Since the (corner portion) is not continuous and has a staircase shape, there is a high possibility that the correspondence between the reference image windows W1, W1,... Shown in FIG. Therefore, the positional deviation of the window edge portion occurring in the column direction is adjusted (S23).

  A procedure for adjusting the positional deviation of the end will be described with reference to FIGS. 5B and 5C. The windows WA1 and WB1 in the first row are connected by a straight line portion connecting the points F1 and F2. The windows WA2 and WB2 in the second row are connected by a straight line connecting points F3 and F4, and the windows WA3 and WB3 in the third row are connected by a straight line connecting points F5 and F6. Here, the points F2 and F3 are at the same height position in the row direction, but are shifted in the column direction and are at different positions. Similarly, the points F4 and F5 are at the same height position in the row direction, but are shifted and different in the column direction. In order to correct this shift, the points F2, F3 are moved toward the middle point E1 of the points F2, F3 to deform the windows WA1, WB1, WA2, WB2. Similarly, the points F4 and F5 are moved toward the middle point E2 of the points F4 and F5 to deform the windows WA2, WB2, WA3 and WB3. The state after deformation is shown in FIG. 5C.

  The windows WA1 to WA3 and WB1 to WB3 thus deformed are all trapezoidal. And the corner located in the center of the four windows overlaps at one point. In this manner, when the windows W2, W2,... Shown in FIG. 6A are deformed, the result is as shown in FIG. 6B. The windows W1, W1 in the reference image shown in FIG. Matches ... Therefore, it is possible to clarify the correspondence between the windows W1, W1... In the standard image and the windows W2, W2. In this way, the window deformation is completed.

  By deforming the windows in this way, the gaps between the windows can be filled, so that the image of the object can be restored smoothly. In addition, since the amount of calculation processing at this time is small, the processing time can be shortened accordingly.

  Next, the procedure for re-dividing the window in step S9 shown in FIG. 2 will be described with reference to FIG. When it is determined in step S8 that all the windows are not one plane, subdivision of windows that are not one plane is performed. When performing re-division, as shown in FIG. 8A, each window in the reference image is divided into four (S31). When each window of the reference image is divided into four, it is divided into a 2 × 2 grid by two straight lines connecting the midpoint of each side of the window and the midpoint of the side opposite to that side.

  After the division of the window in the standard image, the reference image window is also divided into four, and a search range is set corresponding to the division window of the standard image (S32). Here, a divided window having the same shape as the square divided window of the standard image is generated on the reference image. This point will be described with reference to FIGS. 8A and 8B.

  As shown in FIG. 8A, the window on the reference image is also divided into four as shown in FIG. 8B in correspondence with the window divided into four on the standard image. When dividing the reference image window, first, a divided window having the same size as the divided window of the standard image is set as the reference image window. The display position of the divided window when four divided windows are set for each window on the reference image will be described. A trapezoidal window is displayed in the reference image as shown in FIG. 8B.

Regarding the pre-division window on the reference image, lengths L _AB and L _CD are defined as shown in FIG. 8B. L _AB and L _CD are the lengths of the horizontal sides of each rectangle when the pre-division window is equally divided up and down to form a rectangle having the largest area in each of the upper and lower squares. In the case of FIG. 8B, the length L _AB is defined by the upper rectangle, and the length L _CD is defined by the lower rectangle. Further, as shown in FIG. 8A, the length along the X direction of the divided window of the reference image is x0, and the length along the Y direction is y0. Under these conditions, ranges that can be taken by the respective centroids (centers) in the four divided windows are set as X _A , X _B , X _C , and X _D based on the following equations (1) to (4).

X _A = L _AB / 2-1 (1)
X _B = X _A (2)
X _C = L _CD / 2-1 (3)
X _D = X _C (4)

  Within the range thus set, based on predetermined correlations such as color, brightness, and brightness, the windows having the highest correlation with the divided windows in the reference image are set (arranged) (S33). When four divided windows are set for each window of the reference image, each divided window is deformed so that these divided windows fit within the window before the division (S34). The procedure for deforming the divided window is the same as the procedure in step S7. For example, in FIG. 8B, the divided windows D1 and D2 are separated from each other. For this reason, both the divided windows D1, D2 are expanded toward the middle point of both the divided windows D1, D2, and the gap between the divided windows D1, D2 is filled. Further, the divided windows D3 and D4 overlap each other. For this reason, both the divided windows D3 and D4 are contracted toward the middle point of both the divided windows D3 and D4 so as to eliminate the overlapping portion of the divided windows D3 and D4. Further, the periphery of the divided windows D1 to D4 is expanded / contracted according to the shape of the window W before the division.

  When the deformation of each window is completed in this way, it is determined whether or not the window has a desired size (S35). As a result, if it is not the desired size, the process returns to step S31, and the same processing is performed again to divide the window. on the other hand. If it is determined that the desired size is reached, the process is terminated.

  FIG. 9 shows a state where the window division process has progressed. In FIG. 9A (standard image) and FIG. 9B (reference image), the window is not divided, and each of the windows W1, W1,..., W2, W2,. The range that one covers becomes wider. Next, as shown in FIG. 9C (reference image) and FIG. 9D (reference image), the regions of the windows W1, W1,..., W2, W2,. Therefore, the accuracy can be improved when performing stereo matching. Further, as shown in FIG. 9E (standard image) and FIG. 9F (reference image), the range of each window (divided window) is further narrowed by dividing the window once again to form a divided window. Therefore, the accuracy of stereo matching can be further improved.

  Next, a second embodiment of the present invention will be described.

  As the image processing apparatus of the present embodiment, the same image processing apparatus as that shown in FIG. 1 in the first embodiment can be used. Further, the image processing procedure goes through an image processing process shown in FIG. Among these, the processes for mainly performing the window deformation in step S7 and the window division in step S9 shown in FIG. 2 are different. Hereinafter, this embodiment will be described focusing on this difference.

  10A to 10C are explanatory diagrams illustrating the relationship between the image before deformation and the image after deformation according to the present embodiment. FIG. 10A shows a standard image, FIG. 10B shows a reference image before window deformation, and FIG. 10C shows a reference image after window deformation. In the present embodiment, when a window is expanded / contracted and deformed, the deformation is performed based on a point connecting the centroids of the plurality of windows.

  In the reference image shown in FIG. 10A, nine square windows W1, W1,... Are displayed, and four square windows (deformed windows and deformed windows) are connected by connecting the centroids G1 to G9 of the windows W1, W1,. Called). Specifically, one square deformation window is formed by connecting four centroids G1, G2, G5, and G4. Similarly, one deformation window is formed by the centroids G2, G3, G6, and G5, one square is formed by the centroids G4, G5, G8, and G7, and one square is formed by the centroids G5, G6, G9, and G8.

  On the other hand, as shown in FIG. 10B, in the reference image before deformation, windows W2, W2 associated with the windows W1, W1,... In the standard image are displayed based on a predetermined correlation. These windows W2, W2,... Correspond to the arrangement order of the windows W1, W1,... Shown in FIG. 10A, but are overlapped or separated from each other. In order to eliminate such overlapping and separated states, the window is deformed. In this embodiment, as shown in FIG. 10C, the center of gravity G11 to G19 of each window before deformation is connected to form four trapezoidal deformation windows.

  Specifically, a deformation window surrounded by four centroids G11, G12, G15, and G14 in the reference image corresponds to a deformation window surrounded by four centroids G1, G2, G5, and G4 in the standard image. Similarly, the deformation window surrounded by the centroids G12, G13, G16, and 15 in the reference image corresponds to the deformation window surrounded by the centroids G2, G3, G6, and G5 in the standard image. The deformation window surrounded by the centroids G14, G15, G18, and G17 in the reference image corresponds to the deformation window surrounded by the centroids G4, G5, G8, and G7 in the standard image. The deformation window surrounded by the centroids G15, G16, G19, and G18 in the reference image corresponds to the deformation window surrounded by the centroids G5, G6, G8, and G9 in the standard image.

  For window matching, a deformation window formed by G1 to G9 on the standard screen and a deformation window formed by G11 to G19 on the reference screen are used. In this way, by deforming the window in the reference image, there can be no gap between the windows in the reference image. Therefore, the correspondence between the window of the standard image and the reference image can be clarified. . As a result, the object can be restored more accurately. Further, the window deformation method according to the present embodiment is only performed by appropriately connecting the centroids of the windows before the deformation in the reference image to form the windows. For this reason, it is possible to reduce the amount of calculation processing, and as a result, it is possible to perform window deformation and the like in a short time.

  Next, the procedure for re-dividing the window in step S9 shown in FIG. 2 in this embodiment will be described with reference to FIGS. 11A to 11D.

  In the window division of this embodiment, as shown in FIG. 11A, the modified windows W1, W1,... In the reference image are divided into four grids of 2 × 2 to form divided windows D5, D5. . Further, small windows S1, S1,... Arranged in the vertical direction in the figure are also set with respect to the windows W1, W1,. On the other hand, as shown in FIG. 11B, for the deformation windows W3, W3,... In the reference image, the height positions are matched with the small windows S1, S1,. Small windows S2, S2,... Are set respectively. These small windows S2, S2,... Are set by association based on a predetermined correlation such as color, luminance, brightness, etc., with respect to each small window S1, S1,. At this time, with respect to the small windows S2, S2,... Of the reference image, the search range for searching for a position having a high correlation with the small windows S1, S1,. Is kept within the range where it does not reverse.

  When the small windows S2, S2,... Are set in the reference image in this way, the centroids of the small windows S2, S2,. Thereafter, when the centroids are connected by lines, the deformation windows W3, W3... Are divided by the lines connecting the centroids. 12A (standard image) and FIG. 12B (reference image), the divided windows D61 to D64 formed by dividing these modified windows W3, W3... Are divided into the divided windows D51 to D54 of the standard image. It will be corresponding. By forming the divided windows in this way, the area of each window (divided windows D5 and D6: see FIGS. 11A and 11C) can be narrowed, so that the accuracy is improved when performing stereo matching accordingly. Can be made.

  Moreover, the division | segmentation of the deformation | transformation window in this embodiment can also be performed in the procedure shown to FIG. 13A-FIG. 13C demonstrated below. In this division procedure, as shown in FIG. 13A, small windows S3, S3... Are set so as to straddle the contour lines of the areas of windows W1, W1,. On the other hand, as shown in FIG. 13B, in the reference image, the association with the small windows S3, S3... Of the standard image shown in FIG. And small windows S4, S4... Corresponding to each are set.

  When setting the small windows S4, S4,... On the reference image, as shown in FIG. 14, the search is performed so that the arrangement order of adjacent small windows is not reversed and the search range is expanded as much as possible. The search range of each small window is set with the ranges overlapped. This search range will be further described. When attention is paid to the top vertices P1 and P2 of the upper right deformation window W3 among the four deformation windows W3, W3,... The small window search range X11 is set within a predetermined range centered on the right apex P2.

  Subsequently, the search range X12 of the second small window from the right is set to a predetermined range centered on the midpoint of the vertices P1 and P2. At this time, the search range X11 of the rightmost small window and the search range X12 of the second small window from the right are set to slightly overlap. Further, the search range X13 of the third small window from the right is set to a predetermined range centered on the left vertex P1. At this time, the search range X12 of the second small window from the right and the search range X13 of the third small window from the right are set to slightly overlap. By setting the search ranges X11 to X13 for each small window in this way, it is possible to search for a small window in a wide range without changing the arrangement order of the small windows. It should be noted that the search for small windows in other rows can be performed by the same procedure.

  When the small windows are set in this way, as shown in FIG. 13C, the divided windows D7, D7... Are set by connecting the centroids of the small windows S4, S4. FIG. 15A (standard image) and FIG. 15B (reference image) show the state in which the divided window is displayed on the reference image. As shown in FIGS. 15A and 15B, divided windows D71, D72,... Formed by dividing these modified windows W3, W3,... Correspond to the divided windows D51, D52,. By forming the divided windows in this way, the area of each window can be narrowed, so that the accuracy can be improved when performing stereo matching.

  Next, a third embodiment of the present invention will be described.

  As the image processing apparatus of the present embodiment, the same image processing apparatus as that shown in FIG. 1 in the first embodiment can be used. Further, the image processing procedure goes through an image processing process shown in FIG. Among these, the processes for mainly performing the window deformation in step S7 and the window division in step S9 shown in FIG. 2 are different. Hereinafter, this embodiment will be described focusing on this difference.

  FIG. 16A to FIG. 16C are explanatory diagrams illustrating the relationship between an image before deformation and an image after deformation according to the present embodiment. FIG. 16A shows a standard image, FIG. 16B shows a reference image before window deformation, and FIG. 16C shows a reference image after window deformation. In the standard image shown in FIG. 16A, nine square windows W11 to W19 are displayed, and in the reference image before window deformation shown in FIG. 16B, nine windows W21 to W29 are displayed.

  The windows W11 to W19 in the reference image are arranged in an orderly grid pattern. On the other hand, the nine windows W21 to W29 in the reference image are set based on a predetermined correlation such as color, luminance, and brightness with respect to the windows W11 to W19 in the standard image, and the arrangement order is matched. , The arrangement position is cluttered.

  In the present embodiment, in order to expand / shrink the windows W21 to W29 so that there are no gaps between the windows W21 to W29 in the reference image, the boundary areas (R1 to R4) including the corners of the four adjacent windows are included. Thus, the junction point of the four windows is obtained. The joint points of the four windows are set based on a predetermined correlation such as color, brightness, and brightness with the corresponding joint points in the reference image.

  More specifically, in obtaining the junction points of the windows W21, W22, W25, and W24 shown in FIG. 16B, the boundary points R1 including the corners of the windows W21, W22, W25, and W24 are shown in FIG. 16A. A position having a high correlation with the junction point J1 of the windows W11, W12, W15, and W14 is searched. When a position point having the highest correlation with the junction point J1 in the standard image is searched from the boundary region R1, this point is set as the junction point J11 of the windows W21, W22, W25, and W24 in the reference image. Similarly, the junction point J12 of the windows W22, W23, W26, and W25 is joined from the boundary region R2, and the junction point J13 of the windows W24, W25, W28, and W27 is joined from the boundary region R3 to the junctions of the windows W25, W26, W29, and W28. The point J14 is searched from the boundary region R4. Further, the upper sides of the windows W21 and W22 are contracted so that the upper sides of the windows W21 and W22 are joined at the midpoint between the ends of both sides. Further, the upper sides of the windows W22 and W23 are extended so as to be joined at the midpoint between the ends of the windows W22 and W23. Similarly, the lower sides of the windows W27 and W28 and the lower sides of the windows W28 and W29 are extended so as to be joined to both ends at the midpoint.

  By expanding / shrinking the windows W21 to W29 in the reference image, the gaps between the windows W21 to W29 can be eliminated as shown in the reference image after the deformation. By deforming the windows in this way, the gaps between the windows can be filled, so that the image of the object can be restored smoothly. In addition, since the amount of calculation processing at this time is small, the processing time can be shortened accordingly.

  Next, the procedure for re-dividing the window in step S9 shown in FIG. 2 in this embodiment will be described with reference to FIGS. 17A, 17B, 18A, and 18B. FIG. 17A (standard image) and FIG. 17B (reference image) are views for explaining window division according to the present embodiment. 18A and 18B are diagrams for explaining a range for searching for a junction point of divided windows, FIG. 18A is a diagram showing a search range, and FIG. 18B is a diagram showing points that become junction points. 18A and 18B show enlarged portions near the windows W21 and W22 in the reference image shown in FIG. 17B.

  When dividing the modified window according to the present embodiment, as shown in FIG. 17A, the windows W11 to W19 in the reference image are each divided into four grids to form divided windows. On the other hand, as shown in FIG. 17B, the divided windows are set in the reference images in the windows W21 to S29 based on the correlation by matching the height position with the divided window of the corresponding row on the standard image. These divided windows are set by association based on a predetermined correlation such as color, brightness, and brightness for each divided window in the reference image.

  The setting of the divided windows will be described by paying attention to the windows W11 and W12 in the standard image and the windows W21 and W22 in the reference image. The windows W11 and W12 in the standard image have eight divided windows D81 to D88 each having a square shape. Is set. On the other hand, as shown in FIG. 18A, eight divided windows D91 to D98 corresponding to the divided windows D81 to D88 of the standard screen are set in the reference image, but they are separated or overlapped. In order to join the divided windows from these states, a point having the highest correlation with the joining point J21 of the divided windows D81, D82, D86, and D85 in the base image shown in FIG. 17A is a boundary in the reference image shown in FIG. 18A. Search from region R21. The correlation here can be determined based on, for example, color, brightness, brightness, and the like. Thus, for example, the junction point J31 is finally determined from the candidate points of the junction points J31, J31... Shown in FIG.

  When the junction point J31 having the highest correlation with the junction point J21 in the reference image is searched from the boundary region R21, the junction points of the divided windows D91, D92, D96, D95 are set, and the lower sides of the divided windows D91, D92 and The upper sides of the divided windows D96 and D95 are extended. Then, by joining the corners of the divided windows D91, D92, D96, and D95 at the points searched from the boundary region R21, the separation between the divided windows D91, D92, D96, and D96 can be eliminated. .

  Further, the divided windows D92, D93, D97, and D96 in the reference image are all overlapped. A point having the highest correlation with the junction point J22 of the divided windows D82, D83, D87, and D86 in the standard image corresponding to these is searched from the boundary region R22 in the reference image. The correlation here can be determined based on, for example, color, brightness, brightness, and the like. Thus, for example, the junction point J32 is finally determined from the candidate points of the junction points J32, J32... Shown in FIG.

  When the joint point J32 having the highest correlation with the joint point J22 in the reference image is searched from the boundary region R22, joint points of the divided windows D92, D93, D97, D96 are set, and the lower sides of the divided windows D92, D93 and The upper sides of the divided windows D97 and D96 are contracted, respectively. Then, the divided windows D92, D93, D97, D96 are continuous by joining the corners of the divided windows D92, D93, D97, D96 at the points searched from the boundary region R22. Thus, the overlap between the divided windows D92, D93, D96, and D97 can be eliminated.

  In addition, in the windows W21 to W29 shown in FIG. 17B, the separation and overlap between the windows W21 to W29 can be eliminated by dividing and deforming the windows by the same procedure. In this way, the object can be restored more accurately by dividing the deformation window. Further, the object can be smoothly restored by deforming the divided windows so as to eliminate the gap between the divided windows.

  As described above, according to the present invention, it is possible to provide an image processing method and apparatus capable of smoothly restoring a curved surface or the like when restoring an object. Further, it is possible to provide an image processing method and apparatus that can be used for a stereo camera that captures a close distance, and that requires a small amount of calculation processing and a short processing time.

Claims (16)

The reference image capturing unit captures the target object to acquire the reference image, and the reference image capturing unit captures the same object to acquire the reference image.
A plurality of windows associated with each of the reference image and the reference image based on a predetermined correlation are set.
By deforming the plurality of windows in the reference image, the separation or overlap between adjacent windows in the reference image is eliminated, and stereo matching between the window of the reference image and the window of the reference image is performed, An image processing method, wherein the shape of the object is restored. The image processing method according to claim 1, wherein when the adjacent windows on the reference image are deformed, the corners of the windows are moved toward opposite corners of the adjacent windows. In deforming the adjacent window on the reference image, a new window group is formed by a line connecting the centroids of the windows on the reference screen, and a line connecting the centroids of the windows on the reference image is used. The image processing method according to claim 1, wherein a deformed window group corresponding to a new window group on the reference image is formed. A new window group is formed by a line connecting the centers of the windows on the reference screen, and
The image processing method according to claim 1 , wherein a deformed window group corresponding to a new window group on the reference image is formed by a line connecting the centers of adjacent windows on the reference image. In deforming the adjacent window on the reference image, a unified joint point of the adjacent windows is obtained based on a correlation between the adjacent windows within a boundary region including each corner of the adjacent window. The image processing method according to claim 1, wherein the window is deformed based on the unified joint point. Each window set on the standard image and each window on the reference image corresponding to each window on the standard screen are divided into the same number to form divided windows, respectively.
Associating the divided window in the standard image with the divided window in the reference image based on a predetermined correlation,
The image processing method according to claim 1 , wherein the separation or overlap between the divided windows in the reference image is eliminated by deforming the divided windows in the reference image. After deforming the window on the reference image, each window on the reference image and the reference image is divided into a plurality of areas, and each window on the reference image after the division is deformed to change between adjacent windows. The image processing method according to claim 1 , wherein separation and / or overlap are eliminated and stereo matching is performed based on the divided window of the base image and the divided window of the reference image after deformation. The image processing method according to claim 7, wherein the division of each window of the standard image and the reference image and the deformation of each window of the reference image are repeated a plurality of times. Window setting means for setting a plurality of windows for the standard image and the reference image of the same object respectively photographed by the standard image photographing means and the reference image photographing means;
Based on the relationship between the arrangement order of the plurality of windows set in the reference image and the arrangement order of the plurality of windows set in the reference image, the reference image is associated with the reference image, and the reference image The window of the reference image and the window of the reference image are subjected to stereo matching by eliminating the separation or overlapping between adjacent windows on the reference image by deforming each window of the reference image. An image processing apparatus comprising stereo matching means for restoring a shape. The image processing apparatus according to claim 9, wherein the stereo matching unit performs a process of moving the corners of the windows toward the corners of the adjacent windows on the reference image. The stereo matching unit forms a new window group by a line connecting the centroids of the windows on the reference screen, and a new window group on the reference image by a line connecting the centroids of the windows on the reference image. The image processing apparatus according to claim 9, wherein a process for forming a deformed window group corresponding to the step is performed. The stereo matching unit forms a new window group by a line connecting the centers of the windows on the reference screen, and a new window group on the reference image by a line connecting the centers of the windows on the reference image. The image processing apparatus according to claim 9, wherein a process for forming a deformed window group corresponding to the step is performed. The stereo matching means determines a unified joint point of the adjacent windows based on a correlation between the adjacent windows within a boundary region including each corner of the adjacent window on the reference image, and the unified joint The image processing apparatus according to claim 9, wherein the window is deformed based on a point. Window dividing means for dividing each window set on the reference image and each window on the reference image corresponding to each window on the reference screen into the same number to form divided windows. In addition,
The window matching means associates the divided window in the base image divided by the window dividing means with the divided window in the reference image based on a predetermined correlation, and deforms the divided window in the reference image. The image processing apparatus according to claim 9, wherein separation or overlap between the divided windows in the reference image is eliminated. A window dividing unit that divides each window on the reference image and the reference image into a plurality of regions after deforming the window on the reference image;
The stereo matching means deforms each window on the reference image after the division by the window dividing means to eliminate the separation and / or overlap between adjacent windows, and the divided window of the reference image and the deformed window The image processing apparatus according to claim 9, wherein stereo matching is performed based on a reference image division window. The image processing apparatus according to claim 15, wherein the division of each window of the reference image and the reference image by the window dividing unit and the deformation of each window of the reference image by the window matching unit are repeated a plurality of times.

Images