JP6070262B2 - Image display device - Google Patents

Description

  The present invention relates to a technique for converting a captured image obtained by an in-vehicle camera into a bird's-eye view image and displaying the image on an in-vehicle monitor for the purpose of driving assistance, parking assistance, and the like.

  There is a device using a vehicle-mounted camera, a so-called back eye camera, for the purpose of parking assistance when the vehicle moves backward. In recent years, as such a device, there is a device that creates an overhead image by converting an image captured by a back eye camera from the rear of the host vehicle, and displays the created overhead image on a display unit. Here, the bird's-eye view image is an image obtained by looking down the target parking frame and the rear end of the host vehicle from above the host vehicle with a virtual camera.

  Patent Documents 1 and 2 disclose such a device. In the devices disclosed in Patent Documents 1 and 2, a bird's-eye view image is created using a virtual projection plane. Here, the virtual projection plane includes a planar bottom surface parallel to the ground and a wall surface extending obliquely upward from the periphery of the bottom surface. The virtual projection plane is represented by a bathtub curve in a vertical section passing through the center of the bottom surface and perpendicular to the bottom surface.

JP 2008-141643 A JP 2005-167638 A

However, such overhead images have the following problems.
An object located far from the host vehicle displayed in the overhead view image in the screen is displayed slightly enlarged outward from the center of the screen. For this reason, the user may feel uncomfortable by seeing an object located far from the own vehicle on the screen. In addition, the user may have difficulty grasping the sense of perspective between an object located near the own vehicle and an object located far from the own vehicle on the screen.
Further, when the field of view is widened (when the area to be displayed with the bird's-eye view image is widened), a portion (for example, a parking frame) located near the own vehicle is displayed small in the bird's-eye view image on the screen.

An object of the present invention is to prevent an object located far away from the host vehicle from being enlarged and displayed from the center of the screen to the outside when the object is displayed in the overhead view image on the screen. is there.
Furthermore, an object of the present invention is to prevent a parking frame or the like located near the host vehicle from being displayed small in the overhead image on the screen even when the field of view is widened.

  In order to solve the above problems, (1) one aspect of the present invention is an imaging unit that captures an image of the periphery of a vehicle and a virtual camera that is located above the vehicle in an image captured and acquired by the imaging unit. An image conversion unit that converts the overhead image captured by the virtual camera; and a display unit that displays the overhead image converted by the image conversion unit, and the image conversion unit converts the overhead image when the image is converted. The virtual projection surface to be projected is a solid curved surface, and the solid curved surface is an image display device represented by a bathtub curve in a vertical section perpendicular to the ground, wherein the virtual camera is a wide-angle camera, and There is provided an image display device characterized in that a central portion with less distortion in an image obtained by imaging by an imaging unit is positioned so as to include a horizontal portion of the bathtub curve.

  (2) In one aspect of the present invention, the actual camera having the adjusted characteristic by adjusting the distance characteristic from the image center in the irradiation side image with respect to the distance from the image center in the incident side image based on the distortion characteristic of the actual camera It is preferable that the distortion characteristic of the virtual camera is determined based on the distortion characteristic, and that the virtual camera is a wide-angle camera.

  (3) In one aspect of the present invention, the characteristic curve indicating the characteristic is logarithmically approximated to complement the characteristic curve, the complemented characteristic curve is adjusted, and the distortion characteristic of the real camera having the adjusted characteristic curve It is preferable to determine the distortion characteristics of the virtual camera based on the above.

  According to the invention of the aspect of (1), the image display device displays an overhead image projected on the virtual projection plane represented by the bathtub curve in a vertical section perpendicular to the ground and displayed on the screen of the display unit. When an object located far from the vehicle is displayed, it can be suppressed that the object is enlarged and displayed from the center of the screen to the outside.

  In addition, according to the invention of the aspect of (1), the image display device is displayed on the screen even when the display range of the overhead image is expanded by the display unit, that is, when the field of view of the overhead image is expanded. It can suppress that the parking frame etc. in a bird's-eye view image are displayed small.

  According to the invention of the aspect of (2), the image display device can apply the camera parameters of the real camera to the virtual camera, and can further enlarge the image center portion with less distortion, and can Also, the curved deformation of the outer portion can be suppressed, or the display of the outer portion can be enlarged. For this reason, the image display device can be adjusted so that the image is easy to see.

  In general, the optimum distortion characteristics vary depending on the mounting position and mounting angle of the actual camera. Therefore, in order to examine the optimum distortion characteristics for all conditions according to the camera mounting state and the like, it is necessary to secure a number of actual cameras having various characteristics. On the other hand, according to the invention of the aspect (2), it is possible to design on the desk the optimum distortion characteristics according to the request based on the camera parameters of the existing actual camera (finite number of actual cameras).

  According to the invention of the aspect of (3), the image display device complements the characteristic curve indicating the characteristic of the distance from the image center in the irradiation side image with respect to the distance from the image center in the incident side image by logarithmic approximation, Even when a wide-angle virtual camera is set at a high position, all image data can be converted into a bird's-eye view and displayed.

FIG. 1 is a diagram illustrating an example of an input image from a real camera. FIG. 2 is a diagram illustrating an example where the installation position of the virtual camera is low. FIG. 3 is a diagram illustrating an example in which the installation position of the virtual camera is high. FIG. 4 is a diagram showing an example of an overhead image obtained at the installation position (low installation position) of the virtual camera shown in FIG. FIG. 5 is a diagram illustrating an example of an overhead image obtained at the installation position (high installation position) of the virtual camera illustrated in FIG. 3. FIG. 6 is a diagram illustrating an example where the installation position of the virtual camera according to the base technology is low. FIG. 7 is a diagram illustrating an example in which the installation position of the virtual camera according to the base technology is high. FIG. 8 is a diagram illustrating an example of an overhead image obtained at the installation position (low installation position) of the virtual camera illustrated in FIG. 6. FIG. 9 is a diagram illustrating an example of an overhead image obtained at the installation position (high installation position) of the virtual camera illustrated in FIG. 7. FIG. 10 is a diagram illustrating an example of a bird's-eye view image used for explaining the problem of the base technology. FIG. 11 is a diagram used for explaining the problem of the base technology. FIG. 12 is a diagram illustrating an example of a bird's-eye view image used for explaining the problem of the base technology. FIG. 13 is a diagram illustrating an example of a bird's-eye view image used for explaining the problem of the base technology. FIG. 14 is a diagram illustrating an example in which the stereoscopic projection plane is set to a standard angle (reference angle). FIG. 15 is a diagram illustrating an example in which the stereoscopic projection plane is set to a sharp angle (set to a state where the stereoscopic projection plane stands). FIG. 16 is a diagram illustrating an example in which the stereoscopic projection plane is set to a dull angle (the stereoscopic projection plane is set to a lying state). FIG. 17 is a diagram showing an example of an overhead image obtained when the stereoscopic projection plane is set at a standard angle as shown in FIG. FIG. 18 is a diagram showing an example of an overhead image obtained when the stereoscopic projection plane is set at a sharp angle as shown in FIG. FIG. 19 is a diagram illustrating an example of an overhead image obtained when the stereoscopic projection plane is set at a dull angle as illustrated in FIG. 16. 20 is a diagram illustrating an example of an image in which a portion of the parking frame in the overhead image illustrated in FIG. 18 is enlarged. FIG. 21 is a diagram illustrating the shape of the parking frame in the image shown in FIG. FIG. 22 is a diagram illustrating an example of an input image (a captured image before being converted into a super wide angle (fisheye)). FIG. 23 is a diagram illustrating an example of a captured image obtained by making a real camera have an ultra-wide angle (fisheye). FIG. 24 is a diagram illustrating a configuration example of an image processing apparatus. FIG. 25 is a diagram illustrating an example of an Rc-Rd characteristic diagram. FIG. 26 is a diagram illustrating an example of the relationship between θ, φ, camera outer space (Rc space), camera inner space (Rd space), and the like. FIG. 27 is a diagram illustrating an installation example of a virtual camera. It is a figure which shows an example of the bird's-eye view image obtained by the image processing which the image conversion part performed. FIG. 29 is a diagram illustrating an example of an overhead image obtained by the base technology. FIG. 30 is a diagram used for explaining a process of dividing Rc at an appropriate interval. FIG. 31 is a diagram used for explaining a process of setting a plurality of vectors connecting the division points. FIG. 32 is a diagram used for explaining processing for setting the angle of each vector. FIG. 33 is a diagram used for explaining a process of creating a new Rc-Rd characteristic diagram by connecting vectors set with angles of respective vectors. FIG. 34 is a diagram for explaining the scale adjustment in the horizontal direction (Rc side) of the vector. FIG. 35 is a diagram for explaining the rotation adjustment of the vector angle. FIG. 36 is a diagram illustrating an example of an Rc-Rd characteristic diagram finally obtained. FIG. 37 is a diagram for explaining an example of an overhead image obtained by a virtual camera to which customized lens characteristics are applied. FIG. 38 is a diagram used for explaining that the image conversion processing has failed. FIG. 39 is a diagram used for explaining that the image conversion processing has failed. FIG. 40 is a diagram used for explaining that the image conversion process has failed. FIG. 41 is a diagram illustrating an example of an input image. FIG. 42 is a diagram showing an example in which teaching points are given to the input screen shown in FIG. FIG. 43 is a diagram showing various symbols. FIG. 44 is a diagram illustrating an example of an Rc-Rd characteristic diagram obtained by the second approximate expression (complement). FIG. 45 is a diagram illustrating an example of a θ-φ characteristic diagram. FIG. 46 is a diagram illustrating an example of the skewness G with respect to Rd = [1 + G (Rc)] − Rc. FIG. 47 is a diagram illustrating an example of a processing procedure according to the third embodiment.

(Prerequisite technology of this embodiment)
First, the prerequisite technology of this embodiment will be described with reference to the drawings.
First, a conventional method for acquiring a bird's-eye view image will be described with reference to FIGS. 1 to 5.
Here, FIG. 1 is a diagram illustrating an example of an input image from a real camera (electronic camera). FIG. 2 is a diagram illustrating an example in which the installation position (or setting position) of the virtual camera 1 for acquiring a bird's-eye view image is low. FIG. 3 is a diagram illustrating an example where the installation position of the virtual camera 1 is high. FIG. 4 is a diagram showing an example of an overhead image obtained at the installation position (low installation position) of the virtual camera 1 shown in FIG. FIG. 5 is a diagram illustrating an example of an overhead image obtained at the installation position (high installation position) of the virtual camera 1 illustrated in FIG. 3.

In the conventional method, when an input image as shown in FIG. 1 is acquired by imaging with a real camera, the image is converted into an image obtained by imaging with a virtual camera. Here, the virtual camera is a virtual camera set at a predetermined height behind the host vehicle.
However, if the installation position of the virtual camera 1 is low as shown in FIG. 2, the area displayed as the overhead image is narrowed as shown in FIG. That is, the visual field range is narrowed. For this reason, it is conceivable to move the installation position of the virtual camera upward in order to widen the visual field range. However, when the installation position of the virtual camera 1 is set upward as shown in FIG. 3, the image of the virtual camera is greatly deformed at a part far away from the own vehicle as shown in FIG. The surroundings cannot be confirmed.

Next, the prerequisite technology of this embodiment for acquiring a bird's-eye view image will be described.
FIG. 6 is a diagram illustrating an example of the base technology. This base technology converts an image captured and acquired by a camera into an overhead image captured by the virtual camera 1 located above the host vehicle. In the conversion model, when an image captured and acquired by a real camera is used as a bird's-eye view image, the bird's-eye view image is projected and converted on the bathtub stereoscopic projection plane 10 shown in FIG.

  Here, the bathtub three-dimensional projection surface 10 is a three-dimensional curved surface, which is a projection surface parallel to the ground (or a bathtub bottom surface, hereinafter referred to as a ground projection surface) 11 and a curved surface that stands up with respect to the ground projection surface. (Or a bathtub wall surface, hereinafter referred to as a stereoscopic projection surface) 12. The three-dimensional projection surface 12 is a curved surface represented by a bathtub curve in a vertical section passing through the origin at an arbitrary angle and perpendicular to the ground. In the base technology, an input image as shown in FIG. 1 is projected on such a bathtub stereoscopic projection surface 10 to convert it into an overhead image.

  7 to 9 are diagrams illustrating an example of the base technology. FIG. 7 is a diagram illustrating an example in which the virtual camera 1 is installed at a higher position than in the case of FIG. FIG. 8 is a diagram illustrating an example of an overhead image obtained at the installation position (low installation position) of the virtual camera 1 illustrated in FIG. 6. FIG. 9 is a diagram illustrating an example of an overhead image obtained at the installation position (high installation position) of the virtual camera 1 illustrated in FIG. 7.

  When the installation position of the virtual camera 1 is increased as shown as a change from FIG. 6 to FIG. 7, as shown as a change from FIG. 8 to FIG. The display shape of the part located far from the host vehicle (part in which the other vehicle is mainly displayed in this example) is maintained while the performance is secured.

  As described above, the base technology realizes creation of a bird's-eye view image that has both a wide field of view and a sense of distance that is easily grasped by a passenger. Specifically, the base technology is to create a bird's-eye view image with a bathtub stereoscopic projection surface that is an appropriate combination of the ground projection surface and the stereoscopic projection surface, thereby ensuring a field of view for the bird's-eye view image (securing the distant view of the vehicle) It is possible to achieve both of the two effects of ensuring the straightness and the straightness of the part in the vicinity of the host vehicle (for example, the part where the parking frame is displayed) and ensuring the straightness (securing the display in the vicinity of the host vehicle).

However, the base technology as described above has the following problems.
(Problem 1)
Like the part surrounded by the dotted line in the bird's-eye view image shown in FIG. 10, the deformation of the part at a position away from the host vehicle (the upper position of the bird's-eye view image) in the bird's-eye view image is slightly increased. As a result, the occupant receives the impression that the image spreads in the left-right direction.

  As shown in FIG. 11, when the pixels on the input image arranged at equal pitches are projected and converted on the bathtub stereoscopic projection plane to generate the overhead view image, the vehicle is projected on the stereoscopic projection plane. This is because the distant parts are wide and sparsely arranged, and the parts in the vicinity of the host vehicle projected onto the ground projection surface are narrow and densely arranged. That is, on the stereoscopic projection plane, the image is stretched and extended.

(Problem 2)
Further, a part (two parts surrounded by a dotted line) at a position away from the vehicle (an intermediate position in the vertical direction in the bird's-eye view image) in the bird's-eye view image shown in FIG. 12 is displayed slightly larger. As a result, the balance between the part close to the host vehicle and the part far from the host vehicle is deteriorated in the overhead view image. As a result, a bad effect such as it is difficult for the occupant to grasp the perspective from the overhead image.

(Problem 3)
Further, there are cases where it is desired to widen the display range around the vehicle while displaying a parking frame or the like in the vicinity of the host vehicle (when it is desired to widen the field of view). However, as shown in FIGS. However, it cannot be realized simply by lowering the installation position of the virtual camera. That is, as shown in FIG. 13B, only the parking frame is enlarged, and the effect of expanding the field of view cannot be obtained.

In addition, there is a trade-off relationship between securing the field of view realized by the underlying technology and ensuring the straightness and straightness of the vicinity of the host vehicle (for example, the display portion of the parking frame). While maintaining these effects, it is difficult to make adjustments to take countermeasures for the above problems 1 to 3.
Here, with reference to FIG. 14 to FIG. 21, it is described that there is a trade-off relationship between securing the field of view and ensuring the straightness and straightness of the vicinity of the host vehicle in the overhead view image.

  Here, FIG. 14 is a diagram illustrating an example in which the stereoscopic projection plane 12 is set to a standard angle (reference angle). FIG. 15 is a diagram illustrating an example in which the stereoscopic projection plane 12 is set to a sharp angle (set to a state in which the stereoscopic projection plane 12 stands). FIG. 16 is a diagram illustrating an example in which the stereoscopic projection surface 12 is set to a dull angle (the stereoscopic projection surface 12 is set to a lying state). FIG. 17 is a diagram showing an example of an overhead image obtained when the stereoscopic projection plane 12 is set at a standard angle as shown in FIG. FIG. 18 is a diagram illustrating an example of an overhead image obtained when the stereoscopic projection plane 12 is set at a sharp angle as illustrated in FIG. 15. FIG. 19 is a diagram illustrating an example of an overhead image obtained when the stereoscopic projection plane 12 is set at a dull angle as illustrated in FIG. 16. FIG. 20 is a diagram illustrating an example of an enlarged image of a part of the parking frame in the overhead view image illustrated in FIG. FIG. 21 is a diagram illustrating the shape of the parking frame in the image shown in FIG.

If the angle of the stereoscopic projection surface 12 is sharp, as shown in FIG. 18, in the overhead view image, the continuity between the portion of the ground projection surface and the portion of the stereoscopic projection surface is lost, and the boundary becomes conspicuous (the boundary portion). Causes bending deformation).
And as shown in FIG.20 and FIG.21, the parking frame 21 also bends in a dogleg shape by the bending deformation of an image. Therefore, if the occupant uses such a bird's-eye view image, there is a possibility that the host vehicle cannot be guided straight to the parking frame.

  If the angle of the three-dimensional projection plane is dull, as shown in FIG. 19, the shape and size of a part away from the host vehicle is enlarged, resulting in an image that is difficult to recognize or an image that greatly reduces the sense of perspective.

(Description of this embodiment)
Next, the present embodiment will be described with reference to the drawings.
(Principle of this embodiment)
In the present embodiment, a general camera is made to have an ultra-wide angle (fisheye), so that a central portion of a captured image of the camera becomes large and a peripheral portion of the captured image becomes small.

  Here, with reference to FIG. 22 and FIG. 23, a captured image obtained by making the camera a super wide angle (fisheye) will be described. FIG. 22 is a diagram illustrating an example of an input image (a captured image before being converted into a super wide angle (fish eye)), and FIG. 23 is an example of a captured image obtained by converting the camera into a super wide angle (fish eye). FIG.

As can be seen from a comparison between FIG. 22 and FIG. 23, the captured image is displayed in a large central portion and a peripheral portion is displayed small by widening the camera.
In the present embodiment, the above-described various problems are solved by widening the virtual camera in terms of software using the above principle.

(First embodiment)
First, the first embodiment will be described.
In the first embodiment, an image processing apparatus is used.
FIG. 24 shows a configuration example of the image processing apparatus 101 according to the first embodiment.

As shown in FIG. 24, the image processing apparatus 101 includes a camera (electronic camera) 102, a shift switch 103, an image display selection switch 104, an ECU (Electronic Control Unit) 110, and a monitor 105.
The camera 102 is mounted on the rear part of the host vehicle and images the rear of the host vehicle. The camera 102 outputs a captured image acquired by imaging to the ECU 110.

The shift switch 103 is attached to the transmission, and detects a shift position signal corresponding to the position (P, R, N, D, etc.) of the select lever operated by the occupant. Shift switch 103 then outputs a detection signal to ECU 110.
The image display selection switch 104 is mounted in the passenger compartment and is operated by the occupant to switch the overhead image display. The image display selection switch 104 outputs the selection state to the ECU 110.

The monitor 105 is attached in the passenger compartment and displays an image. The display of the monitor 105 is controlled by the ECU 110.
The ECU 110 performs various processes on the image processing apparatus 101. The ECU 110 includes, for example, a microcomputer and its peripheral circuits. Therefore, for example, the ECU 110 is configured by a CPU, a ROM, a RAM, and the like. The ROM stores one or more programs. The CPU executes various processes according to one or more programs stored in the ROM.

  In the present embodiment, the ECU 110 performs a process of converting a captured image (captured image behind the vehicle) acquired by the camera (real camera) 102 into an overhead image and displaying the overhead image on the monitor 105. Therefore, the ECU 110 has an image conversion unit 111. Thus, when the shift switch signal is R, ECU 110 captures a captured image (captured image behind the host vehicle) from camera 102. Then, the ECU 110 converts the captured image into an overhead image by the image conversion unit 111. At this time, ECU 110 outputs an overhead image to monitor 105 based on the operation of image display selection switch 104.

Here, the image processing performed by the image conversion unit 111 will be described.
First, the image conversion unit 111 converts the virtual camera into a wide-angle camera in terms of software. The procedure for converting a virtual camera into a wide-angle camera is as follows.
First, the real camera is calibrated. The parameters of the actual camera are acquired by this calibration. The parameter acquired here is the skewness (or strain strength) of the actual camera. In general, a lens having a characteristic that does not cause a large distortion in the vicinity of the center of the obtained captured image is used for the real camera, as in the case of the on-vehicle back-eye camera. In this embodiment, the distortion intensity of such an actual camera is acquired as a parameter by calibration.

Then, the acquired parameters are applied to the virtual camera to make the virtual camera a wide-angle camera.
Here, FIG. 25 shows an example of an Rc-Rd characteristic diagram (also referred to as a lens characteristic, an Rc-Rd correlation, or a distortion curve related to Rc-Rd input / output) indicating the degree of distortion. Here, the horizontal axis Rc is a value represented as Rc = tan θ using the incident-side lens viewing angle θ. The vertical axis Rd is a value represented as Rd = tan φ using the irradiation side lens viewing angle φ. In addition, the incident side refers to the outside (real space side) of the camera, and the irradiation side refers to the inside (imaging surface side) of the camera. Here, FIG. 26 shows an example of the relationship between θ, φ, camera outer space (Rc space), camera inner space (Rd space), and the like.

  FIG. 25 shows the lens characteristics (characteristics shown as solid lines) of the linear camera and the lens characteristics (characteristics shown as broken lines) of the real cameras. As shown in FIG. 25, in the case of an ideal linear camera without distortion, the incident angle and the irradiation angle have the same value. That is, in the case of a linear camera, θ = φ (Rd = Rc). That is, in the θ-φ characteristic diagram (or θ-φ space), φ ′ = dφ / dθ = 1. In the Rc-φ characteristic diagram (or Rc-φ space), Rd ′ = dRd / dRc = 1. On the other hand, in the case of a real camera (wide-angle camera), the purpose is to fit a wider area within the screen, so in the θ-φ characteristic diagram, φ ′ = dφ / dθ <1. In the Rc-Rd characteristic diagram, Rd ′ = dRd / dRc <1. Here, even in a real camera (wide-angle camera), as shown in a circle in FIG. 25, distortion is suppressed in a region where Rc is small (the influence of distortion is reduced). For example, if a parking frame is stored in this area, the linearity and orthogonality of the parking frame are maintained.

After the virtual camera 1 is converted to a wide-angle camera as described above, next, as shown as an example in FIG. 27, the central portion of the image with a moderate degree of distortion (an area with a low degree of distortion in the Rc-Rd characteristic diagram) The installation position of the virtual camera 1 is adjusted so as to be positioned on the projection plane 11, that is, so that the center portion of the image includes the horizontal portion of the bathtub curve.
As described above, the image conversion unit 111 performs image processing.

  FIG. 28 shows an example of an overhead image obtained by the above-described image processing performed by the image conversion unit 111. FIG. 29 shows an example of an overhead image obtained by the base technology. As can be seen from a comparison between FIG. 28 and FIG. 29, the overhead image shown in FIG. 28 is an image in which the above-described problems 1 to 3 are solved by the image processing of the image conversion unit 111. That is, for example, in the bird's-eye view image, deformation of a part away from the host vehicle (upper part of the image) is suppressed, and a part closer to the host vehicle (an intermediate part in the vertical direction of the image) is displayed larger than that part. It is suppressed.

(Second Embodiment)
Next, a second embodiment will be described.
In the second embodiment, it is possible to adjust the skewness obtained from one real camera to a value optimal for the virtual camera while considering the mounting position and angle of the real camera.

In the second embodiment, such an adjustment is realized by the image conversion unit 111. Therefore, the image conversion unit 111 according to the second embodiment performs processing in the following procedure. The processing will be described with reference to the Rc-Rd characteristic diagrams shown in FIGS.
First, the image conversion unit 111 supplements the Rc-Rd characteristic diagram with the skewness outside the teaching range of the real camera. The interpolation of the skewness is obtained by obtaining a teaching point in a region set in advance with respect to the skewness (Rc-Rd characteristic diagram) of the actual camera, and complementing the skewness outside the region based on the teaching point. Is. Specifically, the skewness complementing process is a process realized in an apparatus according to a third embodiment to be described later. Therefore, the distortion complement processing will be described in the third embodiment.

Next, as shown in FIG. 30, the image conversion unit 111 sets Rc to an appropriate interval (in advance) with respect to the Rc-Rd characteristic diagram (complemented Rc-Rd characteristic diagram) of the real camera acquired by the above-described complementing process. Divide at a set interval).
Next, as shown in FIG. 31, the image conversion unit 111 sets a plurality of vectors that connect the division points.

  Next, the image conversion unit 111 sets the angle ψi of each vector, as shown in FIG. For example, when the distortion is weakened, the angle ψi is increased in the direction of 45 ° (distortion of a linear camera without distortion). Further, when increasing the skewness, the angle ψi is decreased in the direction of 0 ° (= parallel).

Next, as illustrated in FIG. 33, the image conversion unit 111 connects vectors having the angle ψi and creates a new Rc-Rd characteristic diagram.
At this time, the scale adjustment in the horizontal direction (Rc side) of the vector (see FIG. 34) and the rotation adjustment of the vector angle (see FIG. 35) are effective vector adjustments for adjusting the skewness.

  FIG. 36 shows an example of an Rc-Rd characteristic diagram finally obtained by the above-described processing. In FIG. 36, the lens characteristic of the linear camera is indicated by a solid line, the lens characteristic of the real camera is indicated by a broken line, and the lens characteristic customized by the above-described processing is indicated by a one-dot chain line. In FIG. 36, the customized lens characteristics are such that the distortion is suppressed more than the distortion of the actual lens.

  FIG. 37 is a diagram illustrating an example of an overhead image obtained by a virtual camera to which such customized lens characteristics (Rc-Rd characteristic diagram) are applied. Here, FIG. 37A shows a bird's-eye view image (hereinafter referred to as a pre-processing bird's-eye view image) obtained by a virtual camera to which the lens characteristics of the real camera (lens characteristics shown by broken lines in FIG. 36) are applied as they are. (Similar to FIG. 28). FIG. 37B is a diagram illustrating an overhead image (hereinafter, a post-processing overhead image) obtained by a virtual camera to which customized lens characteristics (lens characteristics indicated by a one-dot chain line in FIG. 31) are applied.

  In the post-processing overhead image shown in FIG. 37 (b), an adjustment is made with priority given to the balance between the size and straightness of the parking frame and the display size of the object behind the host vehicle (mainly other vehicles in this example). This is the image obtained. As can be seen from this result, by manipulating the Rc-Rd characteristic diagram, the optimum parameters for the virtual camera are created from the parameters obtained by the same actual camera according to the specifications of the mounted vehicle, etc. Parameters can be applied to the virtual camera.

(Third embodiment)
Next, a third embodiment will be described.
In the third embodiment, the Rc-Rd characteristic diagram is obtained by the following first approximate expression or second approximate expression. In the third embodiment, the image conversion unit 111 realizes the processing.

(1) When the first approximate expression Rc <Rck: Rd = [1 + k ₁ [Rc ² ] + k ₂ [Rc ² ] ² + k ₃ [Rc ² ] ³ ] · Rc
When Rc> Rck and Rc <Rcx: Rd = {1 + A · Ln [Rc] + B} · Rc
When Rcx <Rc: Rd = U · Ln [Rc] + V

(2) Second approximate expression When Rc <Rcx: Rd = [1 + e ₁ θ + e ₂ θ ² + e ₃ θ ³ ] · Rc
= [1 + e ₁ [arctan (Rc)] + e ₂ [arctan (Rc)] ² + e ₃ [arctan (Rc)] ³ ] · Rc
When Rcx <Rc: Rd = U · Ln [Rc] + V

  Here, the boundary position combining the function models is Rc = Rcx. Also, Rco <Rcx <Rcp. Here, Rco is the outermost teaching point position (the position of the outermost teaching point in the area where the teaching point is obtained), and is the Rc space side calibration outermost projection point. Rcp is an inflection point, and is a projection point for switching between the logarithmic approximation formula and the approximation formula up to immediately before the Rd space side Rd ′ = dRd / dRc = 0. For Rck, an optimum value is experimentally defined under the condition of θ = 45 ° (Rck≈1 and Rck <1) in consideration of the convergence of the Rc high-order term by series expansion. As for Rcx, an optimum value is experimentally defined by targeting 80% to 90%, which is slightly inside Rcp (inflection point position).

The reason for using the first approximate expression and the second approximate expression as described above in the third embodiment is as follows.
Here, first, camera physical characteristics will be described.
The ideal camera (wide-angle camera) distortion model is based on the characteristics of an actual physical wide-angle camera, and the distance R from the image plane origin is in the range of 0 to + ∞ (in the case of θ, θ = 0). In the range of (° to 90 °), it is necessary to satisfy the physical characteristics of the assumed wide-angle camera. The physical characteristics (1) to (6) are as follows.

Physical characteristic (1): Incident angle θ ≧ irradiation angle φ (Rc ≧ Rd from R = tan α)
Since this physical characteristic (1) is intended for a wide-angle camera, it means that the irradiation light is bent inward at the rear stage (output side) of the lens. That is, when the skewness is G (Rc), the input / output relationship is expressed by the following equation.
Rc = [1 + G (Rc)] · Rc or φ = arctan {[1 + G (arctan (Rx))] · arctan (Rx)}
Here, G (Rc) = 0 represents a linear camera without distortion.

Physical characteristics (2): 0 ° ≦ θ <90 °, 0 ° ≦ φ <90 ° (0 ≦ Rc, 0 ≦ Rd)
This physical characteristic (2) means that the camera can only shoot to the side (because it cannot shoot the back of the lens).
Physical characteristic (3): φ ′ = dφ / dθ> 0 (Rd ′ = dRd / dRc> 0)
This physical characteristic (3) means that the object located outside the optical axis is displayed on the outside of the screen because it is a camera.

Physical characteristics (4): φ = 0 ° when θ = 0 ° (Rd = 0 when Rc = 0)
This physical characteristic (4) means that the optical axis passes through the center of the lens, and that the optical axis position does not cause bending at the time of input / output.
Physical characteristics (5): Extreme value lim _{θ → 90 °} when φ≈90 (lim _{Rc → ∞} when Rd = ∞)
This physical characteristic (5) means that the object directly beside is reflected beside, and the object beside is not reflected inward.

Physical characteristic (6): The radii Rd and Rc are always positive real numbers.
This physical characteristic (6) means that the finite range of the skewness G (Rc) is always 0 ≧ G (Rc) ≧ −1.
The physical characteristics (1) to (6) are as described above.

By the way, the following two models are generally known as approximate models of distortion of an actual camera.
(1) Tsai approximate model (hereinafter referred to as Tsai approximate model)
Rd = [1 + k ₁ [Rc ² ] + k ₂ [Rc ² ] ² + k ₃ [Rc ² ] ³ ] · Rc
Here, an approximate model of skewness is G (Rc) = k ₁ [Rc ² ] + k ₂ [Rc ² ] ² + k ₃ [Rc ² ] ³ .

(2) Approximate model based on Seidel's distortion and Snell's law (hereinafter referred to as Seidel approximation model)
Rd = [1 + e ₁ θ + e ₂ θ ² + e ₃ θ ³ ] · Rc
Here, an approximate model of skewness is G (Rc) = e ₁ θ + e ₂ θ ² + e ₃ θ ³ .

However, when the Tsai approximation model is used for estimating the degree of distortion of the camera, the approximation accuracy of the irradiation angle φ (= Rd) cannot be supplemented in the wide angle region of the incident angle θ (= Rc).
Therefore, in the technique disclosed in Japanese Patent Application Laid-Open No. 2009-130546 (hereinafter referred to as the prior art), the following complementary processing is performed on the Tsai approximation model.

In the prior art, a portion [k ₁ [Rc ² ] + k ₂ [Rc ² ] ² + k ₃ [Rc ² ] ³ ] showing the influence of the skewness in the equation of the Tsai approximation model is extracted, and the extracted equation is distorted. A function indicating degree (also referred to as skewness or a distortion approximation expression) G (Rc) is captured as the following expression.
Rd = {1 + G (Rc)} Rc

In the prior art, an approximate expression for the wide viewing angle region (Rc transient region) is handled as follows.
Rd = (1 + A · Ln [Rc] + B) Rc
Therefore, according to the equation, the skewness G (Rc) is handled as follows.
When Rc <Rck: G (Rc) is a Tsai conversion equation.
When Rc> Rck: G (Rc) is set to logarithmic conversion equation G (Rc) = A · Ln (Rc) + B.

  With this prior art, the approximate characteristics can be obtained up to the teaching point of around Rc = 1.0 in the Tsai approximation model, but the approximate characteristics can be obtained up to around Rc = 2.5, which is the outermost teaching point. became.

However, the prior art has the following problems.
The correlation between the skewness G (Rc) and the Rc−Rd characteristic can be expressed by an equation such as Rd = {1 + G (Rc)} · Rc. However, when the Rc-Rd characteristic diagram is created based on the approximate expression according to the prior art, there arises a problem that the image conversion process is broken in the wide viewing angle region (Rc transient region).

  For example, as shown in FIG. 38, the prior art shows a state in which the skewness G (Rc) <− 1 at a position where Rc≈10. From the definition of Rd = {1 + G (Rc)} · Rc, in this case, Rd <0, that is, the distance from the origin is a negative value, which deviates from the physical characteristics (1) of the camera. That is, the process based on the prior art breaks down at least in the vicinity of Rc≈10.

Furthermore, the consistency of the prior art is confirmed in detail.
As shown in FIG. 39, for the Rc-Rd characteristic, it can be confirmed that Rd ′, that is, the slope (increase rate) of Rd with respect to Rc is 0 or less at a position where Rc≈3.2 is exceeded. In a physical camera as shown in the physical characteristic (3) of the camera described above, Rd ′ = (dRd / dRc) <0 cannot be satisfied, and the prior art has an Rc that is further before Rc≈10. It can be seen that the image conversion process has already been broken at ≈3.2.

  The prior art is a technique that approximately complements the Rc-Rd characteristic in the wide-angle viewing angle region in the vicinity of Rc = 1.0 (tan θ = tan 45 °) with a logarithmic approximate expression of the skewness G (Rc). However, the effective range related to the camera characteristic approximation is limited to a viewing angle of Rc = 3.2 (tan θ = tan 72 °) or less from the above examination, and it is suggested that the outer wide-angle viewing area cannot be complemented. .

In the Rc-Rd characteristic diagram, the range satisfying the physical characteristic (3) (Rd ′ = dRd / dRc> 0) is such that, in the case of FIG. 38 and FIG. 39, the approximate complement limit is Rc≈3.2 (θ≈72). It is not possible to complement even the logarithmic approximate expression to the outside of the screen that is outside the complement range.
A similar problem occurs with the Seidel approximation model. Specifically, as shown in FIG. 40, the slope of the curve in the Rd-Rc characteristic diagram cannot be negative due to the physical characteristics of the camera, and the image conversion processing has failed. In addition, the output radius Rd being 0 or less is not possible due to the physical characteristics of the camera, and the image conversion processing has failed.

Therefore, even if it is obtained by approximating the skewness by the Tsai approximation model, the Seidel approximation model, or the approximation model of the prior art and trying to adjust the image by arbitrarily operating the skewness in the second embodiment, the reference and Since the skewness is not correct, the applicable range of the approximate model is limited.
In addition, the biggest cause of the above-described problem is that the model guarantee range of the skewness G (Rc) is limited because teaching to the outside region is insufficient during calibration.

However, in consideration of the resolution of the camera, it is difficult to give a teaching point at the outer position of the screen. Here, FIG. 41 shows an example of an input image, and FIG. 42 shows an example in which teaching points (circles shown in FIG. 41) are given to the input screen shown in FIG. As shown in FIG. 41, the resolution becomes worse toward the outer side of the screen. Therefore, as shown in FIG. 42, the teaching point is given in a complicated manner at the outer position of the screen. That is, it is difficult for the image conversion processing to accurately give the teaching point. Therefore, it is difficult to improve accuracy related to calibration or approximation processing by adding teaching points.
From various viewpoints as described above, in the third embodiment, the above-described first and second approximate expressions are proposed.

Here, as described above, the complementary processing using the logarithmic approximation similar to the first and second approximate expressions is also performed in the prior art. However, the biggest problem in the prior art is that the target of logarithmic approximation is the skewness G (Rc).
Here, according to the physical characteristic (6) of the camera, the degree of distortion G (Rc) must be 0 ≧ G (Rc) ≧ −1. However, in the prior art, the skewness G (Rc) is given by G (Rc) = A · Ln (Rc) + B. In this case, when Rc becomes + ∞, Ln (Rc) or the output radius Rd diverges to + ∞ or −∞. As a result, the skewness G (Rc) does not satisfy 0 ≧ G (Rc) ≧ −1.0, and the prior art is contrary to the physical characteristic (6).

  For this reason, in the first approximate expression according to the present embodiment, the logarithmic approximation target is not the skewness G (Rc), but the output radius Rd is directly logarithmic approximated with respect to the output radius Rd. It is carried out. Specifically, logarithmic approximation is performed by the following procedure. FIG. 43 shows symbols described below.

First, the limit radius position where Rd ′ = dRd / dRc = 0 is examined, and the limit radius position Rcz is defined by the linear space radius.
Next, the position of the outermost teaching point is checked, and similarly, the outermost teaching point is set to Rco on the linear space side.
Rc at a position between the outermost teaching point Rco and the limit radius Rcz (for example, an intermediate position between the outermost teaching point position Rco and the limit radius position Rcz) is defined as a logarithmic approximation switching point Rcp.

Next, the data from the outermost teaching point Rco to the logarithmic approximation switching point Rcp is estimated with an approximation model according to the prior art, and sampling points (ie, teaching points) for performing logarithmic approximation are obtained.
Then, based on the acquired sampling points, a logarithmic approximate expression is acquired, and a range larger than the logarithmic approximate switching point Rcp is complemented with the logarithmic approximate expression.

Here, the locus of the Rd curve after the logarithmic approximation switching point Rcp is represented by the following simultaneous equations together with the logarithmic approximation equation.
Rd = [1 + G (Rc)] * Rc = U * Ln [Rc] + V

Moreover, the camera distortion approximate expression shown in the prior art is expressed as the following expression.
Rd = {1 + A · Ln [Rc] + B} · Rc
Thereby, in the prior art, the skewness G (Rc) is expressed by the following equation.
G (Rc) = Rd / Rc-1
= {1 + A · Ln [Rc] + B} −1
= A · Ln [Rc] + B

Further, in the prior art, with respect to the following expression, it is possible to ensure excellent approximation accuracy with respect to the teaching point.
Rd = {1 + A · Ln [Rc] + B} · Rc
However, the degree of distortion G (Rc) in the Rc transient region is as follows.
lim _{Rc → ∞} G (Rc) = lim _{Rc → ∞} {A · Ln [Rc] + B} = ∞
Therefore, the skewness G (Rc) diverges to ± ∞ based on the sign of the approximate complementary coefficient A.

That is, the skewness G (Rc) exceeds the range of 0 ≧ G (Rc) ≧ −1 in the transient region of Rc, and the value of Rd based on the following equation is Rd> Rc (the sign of A is positive) or Rd <0 (the sign of A is negative), which always deviates from the physical characteristic (6) described above.
Rd = {1 + A.Ln [Rc] + B} .Rc = {1 + G (Rc)}. Rc

On the other hand, the following equation according to the third embodiment is problematic in that the approximate characteristic accuracy that is excellent with respect to the teaching point is low.
Rd = U · Ln [Rc] + V
However, in the third embodiment, the application range is limited to the points after the outermost teaching point, and the approximation accuracy is ensured by following the approximation formula according to the prior art in the intermediate area.

Further, the skewness G (Rc) at Rd = U · Ln [Rc] + V is expressed by the following equation.
G (Rc) = Rd / Rc-1 = [U · Ln (Rc)] / Rc + V / Rc-1
According to this equation, the degree of distortion G (Rc) in the Rc transient region is expressed by the following equation according to “Theoretical Theorem”.
lim _{Rc → ∞} G (Rc) ≈lim _{Rc → ∞} [U / Rc + V / Rc−1] = − 1
In this case, regardless of the values of the complementary variables U and V, the condition of 0 ≧ G (Rc) ≧ −1 that satisfies the above-described physical characteristic (6) is satisfied. That is, based on the approximate expression according to the third embodiment, the characteristics of the real camera can be approximated with high accuracy, and a camera model that satisfies the physical characteristics can be defined without breaking in the entire range of 0 ≦ Rc ≦ + ∞. .

For Rd = [1 + e ₁ θ + e ₂ θ ² + e ₃ θ ³ ] · R in the second approximate expression, Rd = [1 + e ₁ [arctanRc] + e ₂ [arctanRc] based on the relationship R = arctan (θ). ² + e ₃ [arctanRc] ³ ] · Rc Rc-Rd characteristics and tan (φ) = [1 + e ₁ θ + e ₂ θ ² + e ₃ θ ³ ] · arctan (θ) θ-φ characteristics Two types of cases are assumed.

However, the transient characteristic lim _{θ → ∞ in the} case of the θ-φ characteristic cannot be handled because the effective range of θ exceeds the condition of 0 ≦ θ ≦ 90 ° of the physical characteristic (2) described above. . Therefore, also for the second approximate expression, the expression is indicated by the Rc-Rd characteristic, and the Rc transition region after the outermost teaching point is complemented by logarithmic approximation. The complement is performed by the same processing procedure as that of the first approximation.

FIG. 44 shows an example of an Rc-Rd characteristic diagram for the second approximate expression. FIG. 45 shows an example of a θ-φ characteristic diagram according to the same equation.
Here, the change of θ with respect to the transient region of Rc is expressed by the following equation.
lim _{Rc → ∞} (θ) = lim _{Rc → ∞} arctan (Rc) = 90 °
Thus, θ converges to 90 °.

Further, it is clear that the change of Rd with respect to the transient region of Rc is as follows.
lim _{Rc → ∞} Rd = lim _{Rc → ∞} [U · Ln [Rc] + V] = ∞
Therefore, the change of φ with respect to the transient region of Rd is also expressed by the following equation.
lim _{Rd → ∞} (φ) = lim _{Rd → ∞} arctan (Rd) = 90 °
Similarly, φ converges to 90 ° from this equation.

  As shown in FIG. 45, the above-described relationship can completely satisfy the condition indicated by the above-described physical characteristic (5) where φ = 90 ° when θ = 90 °. In the whole area or in the range of 0 ≦ θ ≦ 90 °.

For the above reasons, in the third embodiment, the Rc-Rd characteristic diagram is obtained by the first approximate expression or the second approximate expression.
FIG. 47 shows an example of a processing procedure according to the third embodiment.
As shown in FIG. 47, first, the real camera is calibrated and the parameters of the real camera are acquired (step S1).

Next, a Tsai approximation model is adopted as an approximation model of skewness, and a logarithmic approximation formula shown in the prior art (Japanese Patent Laid-Open No. 2009-130546) is added (steps S2 and S3). Alternatively, a Seidel approximation model is adopted as an approximation model of skewness (step S4).
Next, the adopted approximate model is applied to the virtual camera (step S5). Then, logarithmic interpolation is performed for a region outside the teaching range in the Rc-Rd characteristic diagram (step S6).

Next, a conversion table of the overhead image conversion model is created (step S7). Then, the captured image is converted into an overhead image using the conversion table (step S8).
With the above processing, even when a virtual camera is used and the viewing angle is large, the approximate model can be made to correspond to it. As a result, the approximate model can be operated, so that the display size of the background can be adjusted and the perspective can be improved. In addition, since the approximate model can be operated, an optimum image can be set according to different mounted vehicles. Also, all convertible image data can be output and displayed.

  Further, although the embodiments of the present invention have been specifically described, the scope of the present invention is not limited to the illustrated and described exemplary embodiments, and effects equivalent to those intended by the present invention. All embodiments that provide are also included. Further, the scope of the present invention is not limited to the combination of features of the invention defined by claim 1 but can be defined by any desired combination of specific features among all the disclosed features. .

  DESCRIPTION OF SYMBOLS 1 Virtual camera, 101 Image processing apparatus, 102 Camera, 103 Shift switch, 104 Image display selection switch, 105 Monitor, 110 ECU, 111 Image conversion part

Claims (2)

A real camera that captures an image of the surroundings of the vehicle, an image conversion unit that converts an image captured and acquired by the real camera into an overhead image captured by a virtual camera that is a virtual camera located above the vehicle, and the image A display unit that displays the overhead image converted by the conversion unit, and the virtual projection plane that projects the overhead image when the image conversion unit converts the image is a three-dimensional curved surface, and the three-dimensional curved surface is on the ground An image display device represented by a bathtub curve in a vertical longitudinal section,
The virtual camera adjusts the characteristic of the distance from the image center in the irradiation side image with respect to the distance from the image center in the incident side image based on the distortion characteristic of the real camera, and the distortion characteristic of the real camera having the adjusted characteristic The distortion characteristic of the virtual camera is determined based on the image to be a wide-angle camera, and the central portion with less distortion in the image acquired by the real camera is positioned so as to include the horizontal portion of the bathtub curve An image display device characterized by the above. It complements the characteristic curve by performing a logarithmic approximation for the characteristic curve indicating the characteristic, complementary characteristics curve to adjust the distortion characteristics of the virtual camera based on the distortion characteristics of the real cameras with the adjusted characteristic curve The image display device according to claim 1 , wherein:

Images