JP6411100B2 - Vehicle surrounding image generation apparatus and vehicle surrounding image generation method - Google Patents

Description

  The present invention relates to a vehicle surrounding image generation device and a vehicle surrounding image generation method, and in particular, is suitable for use in an apparatus that generates a vehicle surrounding image by combining images captured by a plurality of in-vehicle cameras installed in the host vehicle. It is.

  2. Description of the Related Art Conventionally, a technique has been provided in which a plurality of cameras are installed in a host vehicle, images around the host vehicle are captured by the plurality of cameras, and the captured images are combined and displayed on a display. For example, a technique for generating and displaying a top view image viewed from a virtual viewpoint above the host vehicle is well known. According to this technology, the driver of the own vehicle can grasp obstacles (such as a moving body approaching the own vehicle) reflected in the image by checking the vehicle surrounding image displayed on the display, and It is possible to drive to prevent a collision with an object.

  FIG. 9 is a diagram illustrating an arrangement example of a plurality of cameras installed to generate a vehicle surrounding image and an example of an imaging range. In FIG. 9, the front camera 1a is disposed in front of the host vehicle 100 and captures an image in the range of the front H1. The left side camera 1b is disposed on the left side of the host vehicle 100 and captures an image in the range of the left side H2. The right side camera 1c is disposed on the right side of the host vehicle 100 and captures an image in the range of the right side H3. The rear camera 1d is disposed behind the host vehicle 100 and captures an image in the range of the rear H4.

  As shown in FIG. 9, overlapping areas H12, H13, H24, and H34 exist in the imaging ranges H1 to H4 of the cameras 1a to 1d. For two images captured for one overlapping area (for example, in the case of the left front overlapping area H12, a captured image by the front camera 1a and a captured image by the left side camera 1b) It is determined in advance to which area the image is to be cut out and used. Therefore, when an obstacle is reflected in one of the two images and no obstacle is reflected in the other image, the other image without the obstacle is used for the overlapping area. As a result, there has been a problem that an obstacle may not appear in the generated vehicle surrounding image.

  FIG. 10 is a diagram for explaining this conventional problem. FIG. 10 shows an example of the range of images taken when the host vehicle 100 is stopped before the intersection. As shown in FIG. 10A, at this intersection, a block fence 300 exists on the left side of the host vehicle 100. In addition, it is assumed that the pedestrian 200 is approaching the direction of the host vehicle 100 from the left side of the road that intersects the traveling road of the host vehicle 100. It is assumed that the host vehicle 100 is stopped at the head of this intersection.

  In this case, as shown in FIG. 10B, the pedestrian 200 is shown in the image taken by the front camera 1a. On the other hand, as shown in FIG. 10C, the image captured by the left side camera 1b is shielded by the block fence 300 and the pedestrian 200 is not captured. In this case, as shown in FIG. 10 (d), the captured image of the left side camera 1b in which the pedestrian 200 is not captured in the overlapping region H12 of the imaging range H1 of the front camera 1a and the imaging range H2 of the left side camera 1b. If it is adopted and the vehicle surrounding image is synthesized, the pedestrian 200 will not appear in the generated vehicle surrounding image.

  Conventionally, by displaying the same image on the display as if the outside of the vehicle was seen through the display installation area from the driver's viewpoint position, a vehicle surrounding image including a portion that was a blind spot from the driver's position was displayed. A technique that is provided to a driver has been proposed (see, for example, Patent Document 1).

JP 2007-96638 A

  However, in the technique described in Patent Document 1, an image outside the vehicle can be displayed on a part that becomes a blind spot by a display installed in the vehicle, but as shown in FIG. 10, it is installed in a plurality of directions of the host vehicle. When compositing images taken by each of the cameras, the problem that the obstacle does not appear in the vehicle surrounding image because the part where the obstacle is not captured is cut out and combined cannot be solved .

  The present invention has been made to solve such problems, and is displayed when one vehicle surrounding image is generated and displayed from images of the surroundings of the host vehicle taken by a plurality of cameras. An object of the present invention is to make it possible to more reliably confirm the presence of an approaching moving body to the host vehicle in a vehicle surrounding image.

In order to solve the above-described problem, in the present invention, when a vehicle surrounding image is generated by synthesizing images in a plurality of directions taken by a plurality of imaging devices installed in different places of the host vehicle, a shooting range is set. In the image of the overlapping area, which is an overlapping part, if there is an approaching moving body for the own vehicle in either one of the images, the closer the estimated distance between the approaching moving body and the own vehicle is, the shorter the approaching moving body is. The two images are mixed while the mixing ratio of the existing image is increased .

  According to the present invention configured as described above, in the image of the overlapping region that is a portion where the imaging ranges in two directions overlap, an approaching moving body is reflected in one image, but in the other image Even if the approaching moving object is not shown because it is hidden by the stationary three-dimensional object, an image showing the presence of the approaching moving object in one image is included in the vehicle surrounding image. Since the composition is performed, the displayed vehicle surrounding image always includes an image indicating the presence of the approaching moving body, and the presence of the approaching moving body to the host vehicle can be confirmed in the vehicle surrounding image. it can.

It is a figure which shows the structural example of the image processing system to which the vehicle surrounding image generation apparatus by this embodiment is applied. It is a figure which shows the example of arrangement | positioning and imaging | photography range of the vehicle-mounted camera by this embodiment. It is a figure which shows an example of the table information which showed the correspondence of the estimated distance between an approaching mobile body and the own vehicle, and the mixing rate of the image in an overlap area | region. It is a figure which shows an example of the vehicle surrounding image displayed by the image process part of this embodiment. It is a flowchart which shows the operation example of the vehicle surrounding image generation apparatus by this embodiment. It is a figure which shows the example which variably sets the cutout area | region of an image in an overlap area | region. It is a figure which shows the example which cuts out the area | region where an approaching mobile body is included in the image of an overlapping area. It is a figure which shows the other example of the vehicle surrounding image displayed by the image process part of this embodiment. It is a figure which shows the example of arrangement | positioning and imaging | photography range of a vehicle-mounted camera. It is a figure for demonstrating the conventional problem.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of an image processing system to which the vehicle surrounding image generation device according to the present embodiment is applied. As shown in FIG. 1, the image processing system of the present embodiment includes an in-vehicle camera 1, an image processing unit 2, and a display device 3. Among these, the image processing unit 2 corresponds to the vehicle surrounding image generation device of the present invention.

  The in-vehicle camera 1 is a plurality of imaging devices installed at different locations of the host vehicle, and includes a front camera 1a for photographing the front of the vehicle, a left side camera 1b for photographing the left side of the vehicle, and a right side of the vehicle. It is comprised by the right side camera 1c for imaging | photography.

  FIG. 2 is a diagram illustrating an example of the arrangement and shooting range of the in-vehicle camera 1. In FIG. 2, the front camera 1a is disposed in front of the host vehicle 100 and captures an image in the range of the front H1. The left side camera 1b is disposed on the left side of the host vehicle 100 and captures an image in the range of the left side H2. The right side camera 1c is disposed on the right side of the host vehicle 100 and captures an image in the range of the right side H3.

  As shown in FIG. 2, there are overlapping areas in the shooting ranges H1 to H3 of the cameras 1a to 1c. That is, there is an overlapping region H12 between the front shooting range H1 by the front camera 1a and the left shooting range H2 by the left side camera 1b. In addition, an overlapping region H13 exists between a front shooting range H1 by the front camera 1a and a left shooting range H3 by the right side camera 1c.

  The image processing unit 2 includes an image acquisition unit 21, an image synthesis unit 22, a display control unit 23, a moving body determination unit 24, and a mapping table storage unit 25 as functional configurations. Here, the functional blocks 21 to 24 can be configured by any of hardware, DSP (Digital Signal Processor), and software. For example, when configured by software, each of the functional blocks 21 to 24 is actually configured to include a CPU, RAM, ROM, and the like of a computer, and a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory is stored. Realized by operating.

  The image acquisition unit 21 acquires images in a plurality of directions taken by the plurality of vehicle-mounted cameras 1a to 1c. The image composition unit 22 synthesizes images in a plurality of directions acquired by the image acquisition unit 21 to generate a vehicle surrounding image. In the present embodiment, the image synthesis unit 22 generates a left front vehicle surrounding image by synthesizing the front captured image and the left captured image. In addition, the image composition unit 22 synthesizes the front captured image and the right captured image to generate a right front vehicle surrounding image.

  When the image composition unit 22 performs image composition, it uses the mapping table (coordinate conversion information) stored in the mapping table storage unit 25 to perform viewpoint conversion processing on the image acquired by the image acquisition unit 21. Do. The mapping table storage unit 25 is a table that stores the correspondence between the pixel data of the images taken by the in-vehicle cameras 1a to 1c and the pixel data of the vehicle surrounding image on the left front and the vehicle surrounding image on the right front. Conversion information indicating which pixel of an image captured by the in-vehicle cameras 1a to 1d corresponds to which pixel of the vehicle surrounding image is stored.

  Regarding the overlapping regions H12 and H13, the image composition unit 22 mixes images captured by the two in-vehicle cameras 1 (the front camera 1a and the left side camera 1b, or the front camera 1a and the right side camera 1c). Perform synthesis. That is, in the left overlapping region H12, the image composition unit 22 performs image composition by mixing the image captured by the front camera 1a and the image captured by the left side camera 1b. Similarly, in the overlap region H13 on the right side, the image composition unit 22 performs image composition by mixing the image captured by the front camera 1a and the image captured by the right side camera 1c.

For example, the pixel value of the pixel i in the front image captured by the front camera 1a is pFi, and the pixel value of the pixel j in the left image captured by the left side camera 1b is pLj. Here, it is assumed that the pixel i in the front captured image and the pixel j in the left captured image are pixels arranged in the same position in the composite image by coordinate conversion based on the mapping table. In this case, the pixel value p of the composite image obtained by mixing the two pixel values can be expressed by the following Expression 1, for example.
p = pFi × α + pLj × (1−α) (Equation 1)

  Note that α is a mixing ratio of the front captured image in this pixel, and takes a value from 0 to 1. When α = 0, the front captured image is completely transmitted and only the left captured image is rendered. Conversely, when α = 1, the left captured image is completely transmitted and only the forward captured image is rendered. In the present embodiment, the image composition unit 22 can perform image composition by changing the mixing ratio α for each pixel of the composite image.

  The display control unit 23 controls the display device 3 to display the left front vehicle surrounding image and the right front vehicle surrounding image generated by the image combining unit 22. For example, the display control unit 23 is provided in two display areas on the screen of the display device 3, and displays a left-front vehicle surrounding image and a right-front vehicle surrounding image in each display area.

  The moving body determination unit 24 captures one of the images in the overlapping regions H12 and H13, which are portions where the imaging ranges of the two in-vehicle cameras overlap, among the captured images in a plurality of directions acquired by the image acquisition unit 21. It is determined whether an approaching moving body for the host vehicle is present in the image. That is, the moving body determination unit 24 determines whether or not an approaching moving body exists in one of the front captured image and the left captured image in the left front overlapping region H12. Similarly, the moving body determination unit 24 determines whether or not there is an approaching moving body in either the front captured image or the right captured image in the right front overlapping region H13.

  For example, as shown in FIG. 10, a pedestrian 200 is shown in the image of the front shooting range H1 shot by the front camera 1a, whereas the left shooting range shot by the left side camera 1b. When the image of H2 is shielded by the block ridge 300 and the pedestrian 200 is not captured, the moving body determination unit 24 only includes the own vehicle in the image captured in front of the front camera 1a in the image of the overlapping region H12. It is determined that there is an approaching moving body (pedestrian 200) for.

  If the image captured by the left side camera 1b does not include the block ridge 300 and the pedestrian 200 is also captured in the left captured image, the moving body determination unit 24 determines which one of the images in the overlap area H12. It is determined that the approaching moving body is not present in one of the captured images. Of course, even when the pedestrian 200 is not shown in both the front captured image and the left captured image, the moving body determination unit 24 determines that the approaching moving body is included in any one of the captured images in the image of the overlapping region H12. It is determined that it does not exist. The same determination is made for the other overlapping region H13.

  Note that the approaching moving body is not limited to a pedestrian. For example, a two-wheeled vehicle or another vehicle may be used. Whether an object shown in the captured image is an approaching moving body can be determined by, for example, an image recognition process using pattern matching. That is, the similarity between the shape pattern of the approaching moving body stored in advance and the shape of the object shown in the captured image is obtained, and when the similarity is equal to or greater than a predetermined threshold, the object is the approaching moving body. Is determined.

  Alternatively, a movement vector of an object shown in the photographed image is obtained from images continuously photographed at predetermined time intervals, and when the movement vector is directed toward the own vehicle, the object is moved closer. You may make it determine with a body. The determination methods listed here are merely examples, and the approaching moving body may be determined by other known methods.

  When the moving body determination unit 24 determines that the approaching moving body is present in one of the captured images, the image composition unit 22 includes an image representing the presence of the approaching moving body in the vehicle surrounding image. In this way, the images are synthesized. Specifically, the image composition unit 22 sets the mixing ratio of any one of the images determined that the approaching moving body exists in the overlapping regions H12 and H13 as the estimated distance between the approaching moving body and the host vehicle. Accordingly, the mixing ratio α of one of the images is increased as the estimated distance is shorter.

  For example, as shown in FIG. 3, when the estimated distance R between the approaching mobile body and the host vehicle is larger than the first threshold th1 (R> th1), the estimated distance R is equal to or less than the first threshold th1 and the second When the estimated distance R is equal to or greater than the threshold th2 (th1 ≧ R ≧ th2), the estimated distance R is smaller than the second threshold th2 (th2> R), and the mixing ratios α1, α2, and Table information in which α3 (α1 <α2 <α3) is set is prepared. Then, the image composition unit 22 sets any one of α1, α2, and α3 as a mixing ratio α of the captured image on the side where the approaching moving body exists, according to the estimated distance R between the approaching moving body and the host vehicle. Read from the information and apply to Equation 1.

  For example, in the situation shown in FIG. 10, the moving body determination unit 24 determines that an approaching moving body (pedestrian 200) exists only in the front captured image by the front camera 1a in the image of the overlapping area H12 on the left front. In this case, the image compositing unit 22 sets α1, α2, α3 according to the estimated distance R between the pedestrian 200 and the host vehicle 100 as the mixing ratio α of the forward captured image determined that the pedestrian 200 exists. Is read from the table information and applied to Equation 1.

  Accordingly, when the host vehicle 100 is stopped at the beginning of the intersection as shown in FIG. 10A, the pedestrian 200 approaches the direction of the host vehicle 100 from the left side of the road that intersects the traveling road of the host vehicle 100. Then, according to the estimated distance R between the pedestrian 200 and the host vehicle 100 at that time, the value of the mixing ratio α applied to Equation 1 is variably set in the order of α1 → α2 → α3. As a result, the left-front vehicle surrounding image displayed on the display device 3 by the display control unit 23 changes as shown in FIGS. 4 (a) → (b) → (c). As the estimated distance R becomes smaller, the image of the pedestrian 200 appears more clearly in the vehicle surrounding image.

  The estimated distance R between the approaching moving body and the host vehicle can be calculated by a captured image recognition process. That is, since the information of the shooting range by the in-vehicle camera 1 is known, the position from the position of the own vehicle (installation position of the in-vehicle camera 1) to the approaching moving body depends on the position in the captured image where the approaching moving body is reflected. The distance R can be estimated. That is, the distance (number of pixels) on the captured image can be converted into the distance R from the host vehicle to the approaching moving body.

  FIG. 5 is a flowchart showing an operation example of the vehicle surrounding image generation device according to the present embodiment configured as described above. Note that the flowchart shown in FIG. 5 starts when shooting by the in-vehicle camera 1 is started.

  First, the image acquisition unit 21 acquires images in a plurality of directions captured by the plurality of in-vehicle cameras 1a to 1c (step S1). Next, the moving body determination unit 24 determines whether or not an approaching moving body exists in the images of the overlapping regions H12 and H13 among the images in the plurality of directions acquired by the image acquisition unit 21 (step S2). . Here, when it is determined that the approaching moving body exists in the images of the overlapping regions H12 and H13, the moving body determining unit 24 moves the approaching moving body only in one of the two images of the overlapping region. Is further determined (step S3).

  Here, when it is determined that the approaching mobile body exists only in one of the images, the mobile body determination unit 24 calculates the estimated distance between the approaching mobile body and the host vehicle that are reflected in the image ( Step S4). Then, the image composition unit 22 refers to the table information illustrated in FIG. 3 and sets the mixing ratio α corresponding to the estimated distance R calculated by the moving body determination unit 24 (step S5).

  On the other hand, when it is determined in step S2 that no approaching moving body is captured in the images of the overlapping regions H12 and H13, or in step S3, both of the two images forming the overlapping region are approached. When it is determined that the moving object is captured, the image composition unit 22 sets a predetermined mixing ratio α (for example, an arbitrary value of 0.5 or less) (step S6).

  Thereafter, the image synthesis unit 22 generates a vehicle surrounding image by synthesizing the images in a plurality of directions acquired by the image acquisition unit 21 (step S7). At the time of this image composition, for the overlapping regions H12 and H13, the image composition is performed by applying the mixing ratio α set in step S5 or step S6 to Equation 1. Finally, the display control unit 23 causes the display device 3 to display the vehicle surrounding image generated by the image composition unit 22 (step S8).

  As described above in detail, in the present embodiment, when a vehicle surrounding image is generated by synthesizing images in a plurality of directions taken by a plurality of in-vehicle cameras 1 installed at different locations of the host vehicle, In the image of the overlapping regions H12 and H13, which are portions where the shooting ranges of the camera 1 overlap, if there is an approaching moving body with respect to the host vehicle in any one of the images, By setting the mixing rate α to a value corresponding to the estimated distance between the approaching mobile body and the host vehicle, the closer the approaching mobile body is to the host vehicle, the clearer the presence of the approaching mobile body is in the vehicle surrounding image. The images are combined so that they appear in

  According to this embodiment configured as described above, in the two images forming the overlapping regions H12 and H13, the approaching moving body is shown in one image, but the stationary solid object is shown in the other image. Even if the approaching moving body is not shown in the image, the images are synthesized so that an image representing the presence of the approaching moving body is included in the vehicle surrounding image. As a result, the displayed vehicle surrounding image always includes an image representing the presence of the approaching moving body, and the presence of the approaching moving body to the host vehicle can be confirmed in the vehicle surrounding image.

  In the above-described embodiment, an example has been described in which the mixing ratio α of any one of the captured images determined that the approaching moving body is present is variable according to the estimated distance between the approaching moving body and the host vehicle. However, the present invention is not limited to this. For example, the mixing rate α of any one of the images determined to have an approaching moving body is made larger than the mixing rate (1-α) of the other image, and the mixing rate α is set to the approaching moving body and the host vehicle. You may make it be a fixed value irrespective of the distance between.

  Moreover, although the said embodiment demonstrated the example which variably sets the mixture rate (alpha) of the said 2 image, when an approaching moving body is reflected only in any one of the two images in the overlap area | regions H12 and H13. However, the present invention is not limited to this. For example, as shown in FIG. 6, the image composition unit 22 may be configured to combine images by cutting out images of different regions from the images captured by the two in-vehicle cameras 1 in the overlapping regions H12 and H13. Good.

In the example of FIG. 6, of the overlap area H12 on the left front side, a front shot image by the front camera 1a is used for the cutout area H12- _F , and a left shot image by the left side camera 1b is used for the cutout area H12- _L. Is used to synthesize a vehicle surrounding image. Similarly, of the overlapping area H13 on the right front side, the front captured image by the front camera 1a is used for the cutout area H13- _F , and the right side captured image by the right side camera 1c is used for the cutout area H13- _R. Cars around the vehicle image.

  In this case, the image composition unit 22 cuts out at least an image of a region where the approaching moving body is present from any one of the overlapping regions H12 and H13 determined to have the approaching moving body. For example, as shown in FIG. 7, a pedestrian 200 is shown in the front image captured by the front camera 1 a, whereas the left image captured by the left side camera 1 b is blocked by the block fence 300 and walked. When the person 200 is not photographed, in the overlap area H12 on the left front side, the area captured by the front camera 1a is cut out to the area where the pedestrian 200 is photographed, and the remaining area is left-handed by the left side camera 1b. Cut out from the shot image.

  FIG. 7A shows a state in which an image captured in front of the front camera 1a is cut out in the entire overlap region H12. On the other hand, FIG. 7B shows a state in which a part of the overlap region H12 is cut out from a photographed image in front of the front camera 1a and the rest is cut out from a photographed image on the left by the left side camera 1b. Even in such a configuration in which the image cutout region is variably set, the displayed vehicle surrounding image always includes an image indicating the presence of the approaching moving body, and the vehicle surrounding image approaches the host vehicle. The presence of the moving object can be confirmed.

  In addition, when the image composition unit 22 cuts out the region extracted from the two captured images in the overlapping regions H12 and H13, and the moving body determination unit 24 determines that an approaching moving body exists in one of the images, As shown in FIG. 8, an icon 80 representing the presence of an approaching moving body may be further combined with a vehicle surrounding image formed by combining a plurality of captured images.

  Moreover, although the said embodiment demonstrated the example which produces | generates a vehicle surrounding image from the picked-up image by the three vehicle-mounted cameras 1 of the front camera 1a, the left side camera 1b, and the right side camera 1c, this invention is not limited to this. For example, as shown in FIG. 9, when the vehicle surrounding image (for example, the top view image) is generated from the image captured by the in-vehicle camera 1 in four directions by adding the rear camera 1 d, the present invention relates to the image synthesis of the overlapping region. It is possible to apply.

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the gist or the main features thereof.

DESCRIPTION OF SYMBOLS 1 In-vehicle camera 2 Image processing part 21 Image acquisition part 22 Image composition part 23 Display control part 24 Moving body determination part

Claims (2)

An image acquisition unit for acquiring images in a plurality of directions respectively captured by a plurality of imaging devices installed in different places of the host vehicle;
An image synthesis unit that synthesizes the images in the plurality of directions acquired by the image acquisition unit to generate a vehicle surrounding image;
Among the images in the plurality of directions acquired by the image acquisition unit, in the image of the overlapping area where the imaging ranges by the two imaging devices overlap, the approaching moving body with respect to the host vehicle is in any one of the images. A moving body determination unit that determines whether or not it exists,
The image synthesizing unit is configured to synthesize images by mixing images captured by the two imaging devices in the overlapping region, and the moving body determination unit moves the approaching movement into any one of the images. When it is determined that a body is present, the mixing ratio of any one of the images determined that the approaching mobile body is present is variable according to the estimated distance between the approaching mobile body and the host vehicle, The vehicle surrounding image generation device characterized in that the mixing ratio of any one of the images is increased as the estimated distance is shorter . A first step in which the image acquisition unit of the vehicle surrounding image generation device respectively acquires images in a plurality of directions taken by a plurality of imaging devices installed in different places of the host vehicle;
The moving body determination unit of the vehicle surrounding image generation device is any one of the images in the overlapping region that is the portion where the imaging ranges by the two imaging devices overlap among the images in the plurality of directions acquired by the image acquisition unit. A second step of determining whether an approaching moving body for the host vehicle is present in one image;
An image synthesis unit of the vehicle surrounding image generation device includes a third step of generating a vehicle surrounding image by combining the images in the plurality of directions acquired by the image acquisition unit;
In the third step, the image synthesizing unit is configured to synthesize an image by mixing the images captured by the two imaging devices in the overlapping region, and the moving body determination unit performs any one of the above. When it is determined that the approaching moving body is present in the image of the image, the mixing ratio of any one of the images determined that the approaching moving body is present is estimated between the approaching moving body and the host vehicle. A vehicle surrounding image generation method, wherein the vehicle surrounding image generation method is variable according to a distance, and the mixing ratio of one of the images is increased as the estimated distance is shorter .

Images