JP6415382B2 - Moving object image generation apparatus and navigation apparatus - Google Patents

Description

  The present invention relates to a moving body image generating apparatus that generates an image to be displayed on a display device provided on a moving body, and a navigation apparatus having the moving body image generating apparatus.

  2. Description of the Related Art Conventionally, an image conversion technique has been developed for converting an image captured using a camera into an image equivalent to a camera image area viewed from above (hereinafter referred to as a “bird's-eye view image”). In particular, a so-called “around view monitor” (registered trademark) that generates a bird's-eye image of a certain area around a moving object from images taken using a plurality of cameras mounted on the moving object such as a vehicle. Or technologies such as “Surround View System” have been developed. These bird's-eye images are used for driving assistance of a moving object, for example, by displaying an obstacle at a position that is difficult to see from the driver's seat of the moving object and notifying the driver.

  In general, a camera mounted on a moving body has a limited shooting range, and an area outside the shooting range cannot capture an image. In addition, for example, when the moving body is located in front of the T-shaped road or the crossroad, a blind spot is generated in the shooting range of the camera due to a building and a fence beside the road, and an image of this blind spot area is also shot. I can't. In the prior art, there is a problem that a bird's-eye view image of an area where an image cannot be taken with a camera mounted on a moving body cannot be displayed.

  With respect to this problem, the vehicle surrounding monitoring device of Patent Document 1 supplements the blind spot included in the bird's-eye image converted from the image captured by the camera mounted on the vehicle using the image captured in the past by this camera. Yes. Thereby, the blind spots included in the bird's-eye view image are reduced, and the situation around the vehicle is more accurately notified.

JP 2004-40523 A

  The vehicle surrounding monitoring apparatus of Patent Document 1 supplements the blind spot of a bird's-eye view image using an image captured in the past by a camera mounted on the vehicle. In this configuration, there is a problem that the blind spot on the road where the vehicle travels for the first time cannot be supplemented. Moreover, the image for a complement is an image image | photographed in the past, and there existed a subject which cannot use the image image | photographed in real time for a complement.

  The present invention has been made to solve the above-described problems, and is converted from an image obtained by capturing an area in which an image cannot be captured by a camera mounted on the moving body, regardless of the traveling history of the moving body. It is an object of the present invention to provide a moving body image generation apparatus capable of generating a display image including a bird's-eye view image. It is another object of the present invention to provide a navigation apparatus having such a moving body image generation apparatus.

The moving body image generating apparatus of the present invention includes a first bird's-eye image generating unit that generates a first bird's-eye image from an image captured using a first camera mounted on the first moving body, a first bird's-eye image, A display image generating unit that generates a display image including a second bird's-eye view image generated from an image captured by an external imaging device that is arranged separately from the first moving body using the second camera; An image output unit that outputs the display image to a display device provided in the first moving body, and the display image generation unit generates a display image by combining the first bird's-eye image with the second bird's-eye image. Then, the display image generation unit indicates the first bird's-eye image in the area other than the imaging area of the second camera in the imaging area of the first camera, and the first camera in the imaging area of the second camera. In the area other than the imaging area, a display image showing the second bird's-eye view image is generated and displayed. The image generation unit indicates an area where the imaging area of the first camera and the imaging area of the second camera overlap, and an area where the difference value between the first bird's-eye image and the second bird's-eye image is equal to or greater than a threshold indicates the second bird's-eye image. And the area | region where a difference value is less than a threshold value produces | generates the image for a display which shows a 1st bird's-eye view image .

  According to the present invention, an external imaging device arranged as a separate body from the first moving body generates a display image including a second bird's-eye image generated from an image captured using the second camera. Thereby, the display image containing the 2nd bird's-eye view image converted from the image which image | photographed the area | region which cannot image | photograph an image with a 1st camera can be produced | generated irrespective of the driving | running | working log | history of a 1st moving body.

1 is a block diagram including an image generation apparatus according to Embodiment 1 of the present invention. It is a block diagram which shows an example of the hardware constitutions of the image generation apparatus which concerns on Embodiment 1 of this invention. It is a block diagram which shows the principal part of the display image generation part which concerns on Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the image generation apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the display image generation part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the imaging | photography area | region and blind spot of a 1st camera in the state which the own vehicle is approaching the T-shaped road. It is explanatory drawing which shows the imaging region of the 1st camera in the state shown in FIG. 6, and the imaging region of the 2nd camera. It is explanatory drawing which shows the 1st area | region, 2nd area | region, and overlapping area | region in the state shown in FIG. It is explanatory drawing which shows the area | region which displays a 1st bird's-eye view image, and the area | region which displays a 2nd bird's-eye view image among the overlapping areas shown in FIG. It is a block diagram which shows the principal part of the other display image generation part which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows the locus | trajectory image in the state shown in FIG. It is a block diagram including the other image generation apparatus which concerns on Embodiment 1 of this invention. It is a block diagram including the image generation apparatus which concerns on Embodiment 2 of this invention. It is a block diagram which shows the principal part of the display image generation part which concerns on Embodiment 2 of this invention. It is a flowchart which shows operation | movement of the display image generation part which concerns on Embodiment 2 of this invention. It is explanatory drawing which shows the image for a display which concerns on Embodiment 2 of this invention. It is a block diagram which shows the principal part of the other display image generation part which concerns on Embodiment 2 of this invention. It is explanatory drawing which shows the other image for a display which concerns on Embodiment 2 of this invention. It is a block diagram including the image generation apparatus which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows an example of the message image which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the other example of the message image which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the other example of the message image which concerns on Embodiment 3 of this invention. It is explanatory drawing which shows the other example of the message image which concerns on Embodiment 3 of this invention.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing the main parts of the host vehicle 1 and the external photographing device 2. As shown in FIG. 1, the host vehicle 1 includes a first camera 11, a receiving unit 12, an image generation device 13, and a display device 14. The external photographing apparatus 2 includes a second camera 21, a second bird's-eye image generation unit 22, an information addition unit 23, and a transmission unit 24. With reference to FIG. 1, the image generation apparatus 13 of Embodiment 1 is demonstrated.

  The external photographing device 2 is a photographing device that is arranged as a separate body from the host vehicle 1 and can communicate with the host vehicle 1 through a wireless communication line. The external photographing device 2 is configured by, for example, a monitoring camera device installed on a street or a photographing device mounted on another vehicle different from the host vehicle 1.

  The second bird's-eye image generation unit 22 is an image obtained by using the second camera 21 provided in the external photographing device 2 and is equivalent to a bird's-eye view image (hereinafter referred to as “viewing the photographing region of the second camera 21 from above”). "Second bird's-eye view image"). For this image conversion, information indicating the mounting position of the second camera 21, information indicating the distortion of the lens of the second camera 21, and the like (hereinafter collectively referred to as “second conversion information”) are used. .

  The information addition unit 23 adds information to the data of the second bird's-eye image generated by the second bird's-eye image generation unit 22. The additional information includes, for example, information indicating the shooting time of the image from which the second bird's-eye image is converted, information indicating the orientation of the second camera 21 at the shooting time, and information indicating the position of the external shooting device 2 at the shooting time. In addition, information indicating a shooting region of the second camera 21 at the shooting time (hereinafter collectively referred to as “compositing information”) is included.

  The transmission unit 24 transmits data of the second bird's-eye view image to which information is added by the information addition unit 23 via a wireless communication line.

  The receiving unit 12 receives data transmitted by the transmitting unit 24. The second bird's-eye image decoding unit 33 decodes the second bird's-eye image from the data received by the receiving unit 12. The combining information decoding unit 34 decodes the combining information from the data received by the receiving unit 12.

  The first camera 11 is composed of a plurality of cameras mounted on the host vehicle 1. The first camera 11 captures an area within a certain range around the host vehicle 1.

  The first bird's-eye view image generation unit 31 is equivalent to a bird's-eye view image (hereinafter referred to as a “first view image”) obtained by viewing an image captured using the first camera 11 from above. It is to convert to. For this image conversion, information indicating the mounting position of the first camera 11 and information indicating the distortion of the lens of the first camera 11 (hereinafter collectively referred to as “first conversion information”) are used. .

  The mobile body information generation unit 32 uses the GPS (Global Positioning System) receiver, the vehicle speed sensor, the gyro sensor, and the like mounted on the host vehicle 1 to obtain information about the host vehicle 1 (hereinafter referred to as “moving body information”). Is to be generated. The moving body information includes, for example, information indicating the position, direction, traveling direction, traveling speed, and steering angle of the moving body.

  The display image generation unit 35 includes the first bird's-eye image generated by the first bird's-eye image generation unit 31, the moving body information generated by the moving body information generation unit 32, and the second bird's-eye image decoded by the second bird's-eye image decoding unit 33. And the image (henceforth a "display image") containing a 1st bird's-eye image and a 2nd bird's-eye view image is produced | generated using the information for a synthesis decoded by the information decoding part 34 for a synthesis | combination.

  The image output unit 36 outputs the display image data generated by the display image generation unit 35 to the display device 14 provided in the host vehicle 1. The first bird's-eye image generation unit 31, the moving body information generation unit 32, the second bird's-eye image decoding unit 33, the composition information decoding unit 34, the display image generation unit 35, and the image output unit 36 constitute the image generation device 13. ing.

  The display device 14 is configured by, for example, an FPD (Flat Panel Display) or HUD (Head-Up Display) mounted on the host vehicle 1. The display device 14 displays data input from the display image generation unit 35 as an image.

  FIG. 2 is a block diagram illustrating an example of a hardware configuration of the image generation apparatus 13. The image generation apparatus 13 includes a storage device 41 such as a semiconductor memory, and a processing circuit 42 such as a CPU (Central Processing Unit) or a dedicated system LSI (Large Scale Integration). The functions of the first bird's-eye image generation unit 31, the moving body information generation unit 32, the second bird's-eye image decoding unit 33, the composition information decoding unit 34, the display image generation unit 35, and the image output unit 36 shown in FIG. The processing circuit 42 executes the program stored in the storage device 41 shown in FIG. A plurality of processing circuits 42 may cooperate to execute the program stored in the plurality of storage devices 41 to realize the above function.

  FIG. 3 is a block diagram illustrating a main part of the display image generation unit 35. As shown in FIG. 3, the display image generation unit 35 includes an area determination unit 51, a difference calculation unit 52, and a bird's eye image synthesis unit 53.

  The area determination unit 51 calculates the shooting area of the first camera 11 at the shooting time of the original image of the first bird's-eye view image, using the moving body information generated by the moving body information generation unit 32. The area determination unit 51 uses the calculated information indicating the shooting area of the first camera 11 and the information indicating the shooting area of the second camera 21 included in the combining information decoded by the combining information decoding unit 34. Of the shooting area of the first camera 11, the area excluding the shooting area of the second camera 21 (hereinafter referred to as “first area”) and the shooting area of the first camera 11 out of the shooting areas of the second camera 21. A region to be excluded (hereinafter referred to as “second region”) is determined. The area determination unit 51 determines an area where the imaging area of the first camera 11 and the imaging area of the second camera 21 overlap (hereinafter referred to as “overlapping area”).

  The difference calculation unit 52 uses the first bird's-eye image generated by the first bird's-eye image generation unit 31 and the second bird's-eye image decoded by the second bird's-eye image decoding unit 33 to determine the overlapping region determined by the region determination unit 51. The difference value between the first bird's-eye image and the second bird's-eye image is calculated.

  The bird's-eye image synthesis unit 53 uses the first bird's-eye image generated by the first bird's-eye image generation unit 31 and the second bird's-eye image decoded by the second bird's-eye image decoding unit 33 to determine the first determined by the region determination unit 51. A display image indicating the first bird's-eye view image or the second bird's-eye view image is generated in each of the first region, the second region, and the overlapping region.

  Specifically, in the display image, the first area shows the first bird's-eye image, and the second area shows the second bird's-eye image. Moreover, the area | region where the difference value computed by the difference calculation part 52 among the overlapping area | regions is less than a predetermined threshold shows a 1st bird's-eye view image, and the area | region where a difference value is more than a threshold value shows a 2nd bird's-eye view image. .

  At this time, the bird's-eye image combining unit 53 is included in the information indicating the direction of the host vehicle 1 included in the moving body information generated by the moving body information generating unit 32 and the combining information decoded by the combining information decoding unit 34. Using the information indicating the direction of the second camera 21, the first bird's-eye image and the second bird's-eye image are aligned in the horizontal direction. Further, the bird's-eye view image combining unit 53 arranges the first bird's-eye view image and the second bird's-eye view image with the same display magnification.

  Next, the operation of the image generation device 13 will be described with reference to the flowchart of FIG. The own vehicle 1 has an operation input unit (not shown) such as a button provided on the dashboard or a touch panel integrated with the FPD constituting the display device 14. When the operation input unit receives an input of an operation for instructing the display of the bird's-eye view image, the image generation device 13 starts the processes of steps ST1 to ST4.

  First, in step ST1, the first bird's-eye image generation unit 31 converts an image captured using the first camera 11 into a first bird's-eye image. In step ST <b> 2, the moving body information generating unit 32 generates moving body information using a GPS receiver, a vehicle speed sensor, a gyro sensor, and the like mounted on the host vehicle 1. In step ST3, the second bird's-eye image decoding unit 33 decodes the second bird's-eye image from the data received by the receiving unit 12. In step ST4, the synthesis information decoding unit 34 decodes the synthesis information from the data received by the reception unit 12.

  Next, in step ST5, the display image generation unit 35 decodes the first bird's-eye image generated in step ST1, the moving body information generated in step ST2, the second bird's-eye image decoded in step ST3, and the decoding in step ST4. A display image including the first bird's-eye image and the second bird's-eye image is generated using the synthesized information and the like.

  Next, in step ST <b> 6, the image output unit 36 outputs the display image data generated in step ST <b> 6 to the display device 14. The display device 14 displays the data input in step ST6 as an image.

  Next, details of the process of step ST5 shown in FIG. 4 will be described with reference to the flowchart of FIG.

  First, in step ST51, the area determination unit 51 calculates the shooting area of the first camera 11 at the shooting time of the original image of the first bird's-eye view image using the moving body information generated in step ST2 of FIG. . The area determination unit 51 uses the calculated information indicating the shooting area of the first camera 11 and the information indicating the shooting area of the second camera 21 included in the information for synthesis decoded in step ST4 of FIG. The first area, the second area, and the overlapping area are determined.

  Next, in step ST52, the difference calculation unit 52 uses the first bird's-eye image generated in step ST1 in FIG. 4 and the second bird's-eye image decoded in step ST3 in FIG. 4 to determine the overlap determined in step ST51. A difference value between the first bird's-eye image and the second bird's-eye image in the region is calculated.

  Next, in step ST53, the bird's-eye image synthesis unit 53 uses the first bird's-eye image generated in step ST1 in FIG. 4 and the second bird's-eye image decoded in step ST3 in FIG. 4 to make the determination in step ST51. A display image indicating the first bird's-eye view image or the second bird's-eye view image is generated in each of the first region, the second region, and the overlapping region. At this time, the bird's-eye view image combining unit 53 indicates information indicating the direction of the host vehicle 1 included in the moving body information generated in step ST2 and the direction of the second camera 21 included in the combining information decoded in step ST4. Using the information, the horizontal direction of the first bird's-eye image and the second bird's-eye image are aligned. Further, the bird's-eye view image combining unit 53 arranges the first bird's-eye view image and the second bird's-eye view image with the same display magnification.

Next, specific examples of display images will be described with reference to FIGS.
Assume that the host vehicle 1 is entering a T-shaped road. An imaging region A of the first camera 11 at this time is shown in FIG. A blind spot is generated in the shooting area A of the first camera 11 due to buildings and fences on both sides of the road. The conventional image generation device that outputs only the first bird's-eye view image to the display device has a problem that it cannot display the bird's-eye view image of the blind spot.

  In contrast, the image generation device 13 according to the first embodiment generates a display image in which the first bird's-eye image is combined with the second bird's-eye image. For example, as shown in FIG. 7, it is assumed that a monitoring camera device is installed on the road side in the vicinity of the T-junction, and this monitoring camera device constitutes the external photographing device 2. FIG. 7 shows a shooting area B of the second camera 21 provided in the external shooting device 2 in addition to the shooting area A of the first camera 11 mounted on the host vehicle 1.

  FIG. 8 shows a first area a which is an area excluding the imaging area B of the second camera 21 in the imaging area A of the first camera 11, and the first camera 11 in the imaging area B of the second camera 21. A second area b which is an area excluding the imaging area A is shown. The bird's-eye image synthesis unit 53 generates a display image in which the first area a indicates the first bird's-eye image and the second area b indicates the second bird's-eye image.

  Also, the difference calculation unit 52 calculates a difference value for the overlapping region C, which is a region where the shooting region A of the first camera 11 and the shooting region B of the second camera 21 overlap. In general, in the overlapping region C, the region c2 that is the blind spot of the first camera 11 has a larger difference value between the first bird's-eye image and the second bird's-eye image than the region c1 that is not the blind spot of the first camera 11. The threshold value used by the bird's-eye image synthesis unit 53 to determine the image shown in the overlapping region C is set to a value that can distinguish the region c1 and the region c2.

  That is, the difference value of the area c1 is less than the threshold value, and the difference value of the area c2 is greater than or equal to the threshold value. As shown in FIG. 9, the bird's-eye image synthesis unit 53 generates a display image indicating the first bird's-eye image in the area c1 and the second bird's-eye image in the area c2.

  Thus, the blind spot of the first bird's-eye image can be complemented by generating a display image in which the second bird's-eye image is combined with the first bird's-eye image. In particular, unlike the vehicle periphery monitoring device of Patent Document 1, if there is an external photographing device 2, the blind spot of the bird's-eye view image can be complemented even on the road on which the host vehicle 1 travels for the first time. Moreover, the 2nd bird's-eye view image converted from the image image | photographed in real time by the external imaging device 2 can be used for complementation.

  Note that the display device 14 may display the entire display image output by the image output unit 36 or may display only a part thereof. For example, when the host vehicle 1 is moving forward, only the display image indicating the area in front of the host vehicle 1 may be displayed. Similarly, when the host vehicle 1 is moving backward, only the display image indicating the area behind the host vehicle 1 may be displayed.

  The image generation device 13 stores a graphic image corresponding to the shape of the host vehicle 1 in advance, and the bird's-eye image synthesis unit 53 combines the graphic image in addition to the first bird's-eye image and the second bird's-eye image. It is also possible to generate a business image. Alternatively, the image generation device 13 stores in advance a bird's-eye image obtained by photographing the host vehicle 1 from above, and the bird's-eye image synthesis unit 53 adds the bird's-eye image of the host vehicle 1 in addition to the first bird's-eye image and the second bird's-eye image. May be used to generate a display image.

  6 to 9, the external photographing device 2 may be a photographing device mounted on another vehicle entering the T-junction. In addition, the receiving unit 12 receives the second bird's-eye image and the synthesis information from both the monitoring camera device and the imaging device mounted on the other vehicle, and the display image generating unit 35 includes a plurality of images in the first bird's-eye image. A display image obtained by combining the second bird's-eye images may be generated. That is, the display image generation unit 35 can synthesize a plurality of arbitrary bird's-eye images within the range of the processing capability of the processing circuit 42 and the like.

  The states shown in FIGS. 6 to 9 are examples, and the timing at which the image generation device 13 generates the display image and displays it on the display device 14 is limited when the host vehicle 1 enters the T-junction. It is not something. A display image may be generated and displayed at any timing according to an operation input to the operation input unit. For example, when the host vehicle 1 enters a crossroad, when the host vehicle 1 moves backward, or when the host vehicle 1 overtakes a large vehicle, the display image is generated and displayed so that it is difficult to see from the driver's seat. A bird's-eye view image showing the area can be presented to the driver.

  In addition to the first bird's-eye image and the second bird's-eye image, the display image may indicate a predicted travel locus of the host vehicle 1. A block diagram of the display image generation unit 35 in this case is shown in FIG. The trajectory image generation unit 54 generates a trajectory image indicating the predicted travel trajectory of the host vehicle 1 using information indicating the steering angle of the host vehicle 1 included in the mobile body information generated by the mobile body information generation unit 32. The trajectory image superimposing unit 55 generates a display image by superimposing the trajectory image generated by the trajectory image generating unit 54 on the image obtained by synthesizing the second bird's eye image with the first bird's eye image combining unit 53. FIG. 11 shows an example of the trajectory image D in the same state as the examples shown in FIGS. The trajectory image D shows a predicted travel trajectory when the host vehicle 1 makes a right turn on a T-junction.

  Further, the second bird's-eye view image may be generated by the image generation device 13 instead of the external photographing device 2. A block diagram in this case is shown in FIG. The information adding unit 23 adds information to image data captured by the second camera 21. In the additional information, in addition to the information for synthesis used when the second bird's-eye image is synthesized with the first bird's-eye image, the second conversion information used for the conversion from the image photographed by the second camera 21 to the second bird's-eye image. It is included. The image generation device 13 includes a second bird's-eye image generation unit 37 instead of the second bird's-eye image decoding unit 33 illustrated in FIG. 1. The second bird's-eye image generation unit 37 decodes the image captured by the second camera 21 and the second conversion information from the data received by the reception unit 12, and uses the second conversion information to generate the second bird's-eye image. Generate.

  Further, the first bird's-eye image and the second bird's-eye image may be a still image or a moving image. When the first bird's-eye view image and the second bird's-eye view image are moving images, the display image is also a moving image, so that a display image with higher real-time property can be generated and displayed.

  Further, the storage device 41 and the processing circuit 42 constituting the image generation device 13 may be a storage device and a processing circuit built in a navigation device mounted on the host vehicle 1. Furthermore, the display device 14 may be configured by an FPD integrated with the navigation device.

  In addition, the bird's-eye image synthesis unit 53 generates a display image that indicates the second bird's-eye image when the difference value is less than the threshold value among the overlapping regions, and indicates the first bird's-eye image when the difference value is greater than or equal to the threshold value. It may be what you do. Thereby, when a blind spot is included in the second bird's-eye image instead of the first bird's-eye image, a display image that complements the blind spot of the second bird's-eye image can be generated.

  In addition, the image generation device 13 may determine whether there is an obstacle in the overlapping region and a region between the overlapping region and the host vehicle 1 (hereinafter referred to as “determination target region”). . When an obstacle exists in the determination target area, there is a high possibility that the blind spot of the first camera 11 exists in the overlapping area. Therefore, the bird's-eye image synthesis unit 53 complements the blind spot of the first bird's-eye image by generating a display image indicating the second bird's-eye image in the region where the difference value is equal to or greater than the threshold among the overlapping regions. On the other hand, when there is no obstacle in the determination target area, the possibility that the blind spot of the first camera 11 exists in the overlapping area is low. Therefore, the bird's-eye image synthesis unit 53 supplements the blind spot of the second bird's-eye image by generating a display image indicating the first bird's-eye image in the region where the difference value is equal to or greater than the threshold among the overlapping regions.

  A process in which the image generation device 13 determines whether there is an obstacle will be described. For example, an ECU (Electronic Control Unit) mounted on the host vehicle 1 detects an obstacle around the host vehicle 1 using an ultrasonic sensor provided on the exterior of the host vehicle 1. The image generation device 13 determines whether there is an obstacle in the determination target region using the detection result by the ECU.

  Or when the 1st camera 11 is a camera which image | photographs a moving image, the image generation apparatus 13 shows the vector (what is called a "motion vector") which shows the moving amount | distance of the feature point between the frames of the moving image image | photographed with the 1st camera 11. It is used to determine whether there is an obstacle in the determination target area. That is, generally, the greater the distance between the first camera 11 and the subject, the smaller the motion vector of the feature point corresponding to the subject. For this reason, when there is no obstacle around the host vehicle 1 and the first camera 11 is photographing the road surface, the feature point closer to the upper end of the frame has a smaller motion vector and is closer to the lower end of the frame. The motion vector increases as the point increases. On the other hand, when an obstacle exists around the host vehicle 1 and the first camera 11 is photographing the obstacle, the motion vector is substantially constant regardless of the position in the vertical direction of the frame.

  Therefore, the image generation device 13 uses the traveling speed of the host vehicle 1 included in the moving body information, the mounting position of the first camera 11 and the distortion of the lens included in the first conversion information, and the like around the host vehicle 1. When the host vehicle 1 travels at the current traveling speed in a state where no obstacle is present, the first camera 11 calculates a reference vector corresponding to the motion vector of the moving image that captures the determination target region. The image generation device 13 compares the motion vector of the moving image in which the first camera 11 captures the determination target area with the reference vector, and determines that an obstacle exists in the determination target area when the difference is large, and the difference is small Determines that no obstacle exists in the determination target area. When a video processor is used for the processing circuit 42 of the image generation device 13, the video processor has a function of analyzing a motion vector, so that it is not necessary to add a dedicated processor for obstacle determination, and it can be easily performed. Can be realized.

  When the image generation device 13 does not have a function for determining the presence or absence of an obstacle, or when the image generation device 13 cannot determine the presence or absence of an obstacle, the bird's-eye image synthesis unit 53 Generate both a display image showing the second bird's-eye image in an area where the difference value is greater than or equal to a threshold value and a display image showing the first bird's-eye image in an area where the difference value is greater than or equal to the threshold value among the overlapping areas. It may be what you do. The passenger of the host vehicle 1 checks the two display images displayed side by side on the display device 14 and determines which display image is the one that correctly complements the blind spot.

  Alternatively, when the image generation device 13 does not have a function of determining the presence or absence of an obstacle, or when the image generation device 13 cannot determine the presence or absence of an obstacle, the bird's-eye view image synthesis unit 53 A display image may be generated by superimposing the first bird's-eye image and the second bird's-eye image at a transmittance of 50% on a region where the difference value is equal to or greater than a threshold value.

  In addition, when the receiving unit 12 cannot receive data, the second bird's-eye image decoding unit 33 cannot decode the second bird's-eye image, the combining information decoding unit 34 cannot decode the combining information, When an operation indicating that it is not necessary to synthesize two bird's-eye images is input to the operation input unit, the display image generation unit 35 does not synthesize the first bird's-eye image and the second bird's-eye image. Only the image may be output to the image output unit 36 as a display image.

  In addition, the moving body on which the image generation device 13 is mounted is not limited to a vehicle. The image generation apparatus 13 of the present invention can be used for any moving object including a vehicle, a railway, a ship, an aircraft, and the like.

  As described above, the image generation device 13 according to the first embodiment generates the first bird's-eye view image from the image captured using the first camera 11 mounted on the host vehicle 1 (first moving body). The image generation unit 31, the first bird's-eye image, and the image captured by the external photographing device 2 arranged as a separate body apart from the host vehicle 1 (first moving body) are generated using the second camera 21. A display image generation unit 35 that generates a display image including the second bird's-eye view image, and an image output unit 36 that outputs the display image to the display device 14 provided in the host vehicle 1 (first moving body). Prepare. Thereby, the display image containing the 2nd bird's-eye view image converted from the image which image | photographed the area | region which cannot image | photograph with the 1st camera 11 irrespective of the driving | running | working log | history of the own vehicle 1 can be produced | generated.

  In addition, the display image generation unit 35 is an area where the imaging area of the first camera 11 and the imaging area of the second camera 21 overlap each other, where the difference value between the first bird's-eye image and the second bird's-eye image is greater than or equal to a threshold value. Indicates a second bird's-eye view image, and an area for which the difference value is less than the threshold value generates a display image indicating the first bird's-eye view image. Thereby, the display image which complemented the blind spot contained in the 1st bird's-eye image with the 2nd bird's-eye image can be produced | generated.

  Further, the display image generation unit 35 generates a trajectory image indicating the predicted travel trajectory of the host vehicle 1 (first moving body), and displays a display image in which the trajectory image is superimposed on the first bird's-eye image and the second bird's-eye image. Generate. By supplementing the blind spot included in the first bird's-eye view image and displaying the predicted traveling locus of the host vehicle 1, on the T-shaped road or the crossroad, attention should be paid to obstacles present in the blind spot and not to be. Can be presented to the driver in an easy-to-understand manner.

  The external photographing device 2 is a surveillance camera device installed on the street. Alternatively, the external photographing device 2 is a photographing device mounted on another vehicle (second moving body) different from the host vehicle 1 (first moving body). By using the second bird's-eye view image converted from the image photographed in real time by these external photographing devices 2, it is possible to generate and display a display image with high real-time properties. The external photographing device 2 may include both a surveillance camera device installed on the street and a photographing device mounted on another vehicle.

Embodiment 2. FIG.
With reference to FIG. 13, an image generation apparatus 13 that generates a display image in which a first bird's-eye image and a second bird's-eye image are superimposed on a map image will be described. In FIG. 13, the same blocks as those in the block diagram of the first embodiment shown in FIG. Further, since the hardware configuration of the image generation apparatus 13 according to the second embodiment is the same as that of the first embodiment, description will be given with reference to FIG.

  The navigation device 15 is configured by, for example, a car navigation device mounted on the host vehicle 1 or a portable information terminal such as a smartphone brought into the host vehicle 1. The navigation device 15 guides the travel route of the host vehicle 1 using the map information stored in the map information storage unit 16 and the moving body information generated using a GPS receiver, a vehicle speed sensor, a gyro sensor, and the like. It has a function. The navigation device 15 outputs information indicating a travel route to be guided (hereinafter referred to as “route information”) to the display image generation unit 35.

  FIG. 14 shows a block diagram of a main part of the display image generation unit 35. As shown in FIG. 14, the display image generating unit 35 includes a map image generating unit 61 and a bird's eye image superimposing unit 62.

  The map image generation unit 61 uses the mobile body information generated by the mobile body information generation unit 32, the route information output by the navigation device 15, and the map information stored in the map information storage unit 16. A map image including the travel route is generated.

  The bird's-eye image superimposing unit 62 generates a map image generated by the map image generating unit 61 from the first bird's-eye image generated by the first bird's-eye image generating unit 31 and the second bird's-eye image decoded by the second bird's-eye image decoding unit 33. Display images superimposed on the corresponding positions are generated.

  At this time, the bird's-eye image superimposing unit 62 superimposes only the first bird's-eye image that indicates the area in front of the host vehicle 1 in the first bird's-eye image. Further, the bird's-eye image superimposing unit 62 uses the information indicating the position of the external imaging device 2 included in the combining information decoded by the combining information decoding unit 34 to use the travel route of the host vehicle 1 in the second bird's-eye image. Only the second bird's-eye view image received from the external imaging device 2 at a position ahead of the host vehicle 1 is superimposed.

  Further, the bird's-eye view image superimposing unit 62 includes information indicating the direction of the host vehicle 1 included in the moving body information generated by the moving body information generating unit 32 and the combining information decoded by the combining information decoding unit 34. Using the information indicating the direction of the two cameras 21, the horizontal direction of the first bird's-eye image and the second bird's-eye image is superimposed according to the direction of the map image. The bird's-eye image superimposing unit 62 superimposes the display magnifications of the first bird's-eye image and the second bird's-eye image in accordance with the display magnification of the map image.

  Next, the detailed operation of the display image generation unit 35 will be described with reference to the flowchart of FIG. The operations of the blocks other than the display image generation unit 35 in the image generation device 13 are the same as those in the first embodiment, and thus will be described with reference to FIG.

  First, in step ST61, the map image generation unit 61 obtains the moving body information generated in step ST2 in FIG. 4, the route information output from the navigation device 15, and the map information stored in the map information storage unit 16. The map image including the current position of the host vehicle 1 and the travel route is generated.

  Next, in step ST62, the bird's-eye image superimposing unit 62 generates the map image generated in step ST61 from the first bird's-eye image generated in step ST1 in FIG. 4 and the second bird's-eye image decoded in step ST3 in FIG. Display images superimposed on the corresponding positions are generated.

  At this time, the bird's-eye image superimposing unit 62 superimposes only the first bird's-eye image that indicates the area in front of the host vehicle 1 in the first bird's-eye image. Further, the bird's-eye image superimposing unit 62 uses the information indicating the position of the external photographing device 2 included in the composition information decoded in step ST4 of FIG. Only the second bird's-eye view image received from the external photographing device 2 at a position ahead of the host vehicle 1 is superimposed.

  Further, the bird's-eye view image superimposing unit 62 includes information indicating the direction of the host vehicle 1 included in the moving body information generated in step ST2 of FIG. 4 and the second camera included in the composition information decoded in step ST4 of FIG. The horizontal direction of the first bird's-eye image and the second bird's-eye image is superimposed according to the direction of the map image using the information indicating the direction of 21. The bird's-eye image superimposing unit 62 superimposes the display magnifications of the first bird's-eye image and the second bird's-eye image in accordance with the display magnification of the map image.

Next, a specific example of the display image will be described with reference to FIG.
FIG. 16 shows a map image F including the travel route E of the host vehicle 1. A first bird's-eye view image is superimposed on a triangular area G in front of the host vehicle 1. Note that the display image generation unit 35 may generate a display image with the region G as shown in FIG. As a result, the driver of the host vehicle 1 can grasp the shooting range of the first camera 11.

  In addition, the second bird's-eye view image is superimposed on an area H including a route ahead of the host vehicle 1 in the travel route E. The display image generation unit 35 generates a display image in which the second bird's-eye image is superimposed only on a part of the region H according to the data reception state by the reception unit 12 or the like. May be.

  In this way, by generating the display image in which the first bird's-eye image and the second bird's-eye image are superimposed on the map image, the bird's-eye image of the route ahead of the host vehicle 1 in the travel route E of the host vehicle 1 is obtained. It can be displayed in real time. Thereby, the present state of the road on which the host vehicle 1 will travel in the future can be presented to the driver of the host vehicle 1 in an easily understandable manner.

  Note that the display image generation unit 35 may change the transparency of the second bird's-eye image superimposed on the map image. A block diagram in this case is shown in FIG. The bird's-eye image transmitting unit 63 includes information indicating the position of the host vehicle 1 included in the moving body information generated by the moving body information generating unit 32 and an external photographing device included in the combining information decoded by the combining information decoding unit 34. The distance between the own vehicle 1 and the external photographing device 2 is calculated using the information indicating the position of 2. The bird's-eye image transmission unit 63 changes the transparency of the second bird's-eye image according to the calculated distance.

  For example, the bird's-eye image transmission unit 63 displays the second bird's-eye image generated from the image photographed by the external photographing device 2 at a position where the distance to the host vehicle 1 exceeds a reference distance (for example, 600 meters) as a translucent display. An image is generated. In this case, as shown in FIG. 18, the area h1 within the area H whose distance from the host vehicle 1 is within the reference distance is displayed without transmitting the second bird's-eye view image (that is, the transparency is 0%). On the other hand, in the area h2 in which the distance from the host vehicle 1 in the area H exceeds the reference distance, the second bird's-eye view image is displayed translucently (for example, the transparency is 50%).

  The reference distance may be a fixed value set in advance in the display image generation unit 35 or may be a variable value calculated by the display image generation unit 35. For example, the display image generation unit 35 uses the information indicating the traveling speed of the host vehicle 1 included in the moving body information, and lengthens the reference distance as the traveling speed increases. Alternatively, the display image generation unit 35 uses the information indicating the road type included in the map information, and the reference distance is greater when the host vehicle 1 is traveling on a highway than when traveling on a normal road. Lengthen.

  Alternatively, the bird's-eye image transmission unit 63 may generate a display image with higher transparency as the second bird's-eye image generated from the image captured by the external imaging device 2 located away from the host vehicle 1. good. In this case, in the area H shown in FIG. 16, the second bird's-eye view image is displayed so that the transparency gradually increases as the distance from the host vehicle 1 increases.

  Further, the storage device 41 and the processing circuit 42 constituting the image generating device 13 may be a storage device and a processing circuit built in the navigation device 15. Further, the display device 14 may be configured by an FPD integrated with the navigation device 15.

  Further, the map information storage unit 16 may be configured by a storage device provided outside the host vehicle 1, such as a server installed in a data center.

  In addition, the image generation device 13 generates a display image in which the second bird's-eye image is superimposed on the first bird's-eye image according to the first embodiment, and the first bird's-eye image and the second bird's-eye image according to the second embodiment. And a function of generating a display image in which the image is superimposed on the map image. In this case, the image generation device 13 may be configured to be able to select which of the two functions is to be executed in accordance with an operation input to the operation input unit.

  As described above, in the image generation device 13 according to the second embodiment, the display image generation unit 35 generates a map image including the travel route of the host vehicle 1 (first moving body), and uses the travel route of the map image. A display image in which the first bird's-eye image and the second bird's-eye image are superimposed is generated. Thereby, the image which superimposed the bird's-eye view image including the area | region which cannot be image | photographed with the 1st camera 11 mounted in the own vehicle 1 on a map image can be displayed.

  In addition, the display image generation unit 35 displays the second bird's-eye image generated from the image captured by the external imaging device 2 whose distance from the host vehicle 1 (the first moving body) exceeds the reference distance as a translucent display. An image is generated. Alternatively, the display image generation unit 35 displays a display image having a higher transparency as the second bird's-eye image generated from the image captured by the external imaging device 2 located at a position away from the host vehicle 1 (first moving body). Generate. Since the second bird's-eye image showing a point away from the current position of the host vehicle 1 becomes a past image when the host vehicle 1 reaches the shooting point of the second bird's-eye image, the reliability of the information indicated by the image is displayed. May be low. By increasing the transparency of the second bird's-eye view image at a position away from the host vehicle 1, the reliability of the second bird's-eye view image can be presented in an easily understandable manner.

Embodiment 3 FIG.
With reference to FIG. 19, an image generation apparatus 13 that generates a message image indicating the content of information received from the external imaging apparatus 2 will be described. In FIG. 19, the same blocks as those in the block diagram of the second embodiment shown in FIG. Moreover, since the hardware configuration of the image generation apparatus 13 according to the third embodiment is the same as that of the second embodiment, the description will be given with reference to FIG.

  In the third embodiment, the external photographing device 2 is configured by a photographing device mounted on another vehicle different from the host vehicle 1. The transmission unit 24 of the external photographing apparatus 2 and the reception unit 12 of the host vehicle 1 can communicate with each other by so-called “vehicle-to-vehicle communication”.

  The external photographing device 2 has a function of executing image recognition processing on an image photographed using the second camera 21. The external photographing device 2 generates information (hereinafter referred to as “presentation information”) to be presented to a passenger of a vehicle other than the vehicle on which the external photographing device 2 is mounted, using the result of the image recognition process. Yes.

  For example, when a road is covered with snow in an image photographed by the second camera 21, the external photographing device 2 generates presentation information indicating bad weather. At this time, it is more preferable that the external photographing device 2 generates presentation information that also indicates the type of bad weather (for example, snow) if possible according to the accuracy of the image recognition process.

  In addition, when a plurality of vehicles are arranged at narrow intervals in the image captured by the second camera 21, the external imaging device 2 generates presentation information indicating a traffic jam. At this time, it is more preferable that the external photographing device 2 generates the presentation information that also indicates the lane in which the traffic jam occurs, using map information or the like, if possible according to the accuracy of the image recognition processing. .

  Moreover, when the vehicle has stopped in the image image | photographed with the 2nd camera 21, the external imaging device 2 produces | generates the information for presentation which shows stop information. At this time, it is more preferable that the external photographing device 2 generates the presentation information that also indicates the lane where the vehicle is stopped, if possible according to the accuracy of the image recognition processing, using map information or the like. .

  The information adding unit 23 adds the presentation information generated by the external imaging device 2 to the data of the second bird's-eye image in addition to the composition information similar to those in the first and second embodiments. The presentation information is added, for example, in the form of text data indicating the contents of the presentation information.

  The presentation information decoding unit 38 decodes the presentation information from the data received by the receiving unit 12. The message image generation unit 39 generates a message image indicating the content of the presentation information decoded by the presentation information decoding unit 38.

  The display position determination unit 40 uses the moving body information generated by the moving body information generation unit 32, the combining information decoded by the combining information decoding unit 34, the map information stored in the map information storage unit 16, and the like. The display position of the message image in the display area of the display device 14 is determined.

  The image output unit 36 outputs the message image data generated by the message image generation unit 39 to the display device 14 in addition to the display image data generated by the display image generation unit 35. At this time, the image output unit 36 outputs the message image to the display device 14 so as to be displayed at the display position determined by the display position determination unit 40.

  First bird's-eye image generation unit 31, moving body information generation unit 32, second bird's-eye image decoding unit 33, composition information decoding unit 34, display image generation unit 35, image output unit 36, presentation information decoding unit 38, message The image generation unit 13 is configured by the image generation unit 39 and the display position determination unit 40.

Next, specific examples of message images will be described with reference to FIGS.
FIG. 20A shows a state in which the display device 14 is composed of an FPD, and a display image similar to that in the second embodiment is displayed on the display screen of the FPD. That is, in the map image F including the travel route E of the host vehicle 1, the first bird's-eye image is superimposed on the region G, and the second bird's-eye image is superimposed on the region H.

  For example, it is assumed that the external photographing device 2 mounted on another vehicle that is 600 meters ahead of the host vehicle 1 in the travel route E transmits presentation information indicating bad weather together with the second bird's-eye view image. At this time, the message image generation unit 39 generates a message image I that calls attention to bad weather as shown in FIG. The display position determination unit 40 determines that a position 600 meters ahead of the host vehicle 1 on the travel route E in the map image F is the display position of the message image I as shown in FIG. The image output unit 36 outputs the message image I to the display device 14 so that the message image I is displayed at the display position determined by the display position determination unit 40.

  FIG. 21A shows a state in which the display device 14 includes an FPD and an HUD, and the HUD projects a message image separately from the map image displayed on the FPD.

  For example, the external photographing device 2 mounted on another vehicle that is 200 meters ahead of the host vehicle 1 in the travel route of the host vehicle 1 transmits information for presentation indicating traffic congestion together with the second bird's-eye view image. And At this time, the message image generation unit 39 generates a message image J that calls attention to traffic congestion as shown in FIG. The display position determination unit 40 uses the map information, the information indicating the position of the external photographing device 2, the information indicating the position of the host vehicle 1, and the like, as shown in FIG. In the display area, the position overlapping the point 200 meters away from the driver's seat is determined as the display position of the message image J. For example, when the travel route is a gentle left curve, it is determined that the upper left position of the windshield as viewed from the driver's seat is the display position of the message image J. The image output unit 36 outputs the message image J to the display device 14 so as to be projected onto the display position determined by the display position determination unit 40.

  Note that the message image generation unit 39 calculates the arrival time until the host vehicle 1 reaches the position of the external photographing device 2 that transmitted the presentation information, using the map information, the moving body information, the combining information, and the like. A message image including this arrival time may be generated. An example of the message image K in this case is shown in FIG. 22B, and an example of a state in which the message image K is projected on the HUD is shown in FIG.

  In addition, the message image generation unit 39 calculates a route distance between the position of the external imaging device 2 that transmitted the presentation information and the position of the host vehicle 1 using map information, moving body information, composition information, and the like. However, a message image including this route distance may be generated. An example of the message image L in this case is shown in FIG. 23B, and an example of a state in which the message image L is projected on the HUD is shown in FIG.

  Further, the display device 14 is composed of a HUD, and the image output unit 36 automatically displays a display image in which the first bird's-eye image, the second bird's-eye image, and the message image are superimposed on the map image as shown in FIG. It may be projected onto the windshield of the vehicle 1.

  Further, the presentation information is not limited to information indicating bad weather, information indicating traffic congestion, and stop information. Any information that is preferably presented to the driver of the vehicle may be transmitted as presentation information. Further, the message images shown in FIGS. 20 to 23 are examples, and message images such as any characters, shapes, sizes, colors, brightness, scrolling, and blinking states can be displayed according to the design and required specifications of the display device 14. It may be displayed.

  As described above, the image generation device 13 according to the third embodiment generates a message image indicating the content of the presentation information using the presentation information received from the other vehicle (second moving body) by inter-vehicle communication. A message image generation unit 39 and a display position determination unit 40 that determines the display position of the message image in the display area of the display device 14 according to the position of the external photographing device 2 are provided. The image output unit 36 outputs the message image to the display device 14 so as to be displayed at the display position determined by the display position determination unit 40. By displaying the contents of the information received by inter-vehicle communication, the driver is cautioned about events that occur outside the shooting range of the first camera 11 and where the driver of the host vehicle 1 is not visible. Can do. In particular, a place where an event that has caused the driver of the host vehicle 1 to be alerted by transmitting the presentation information together with the second bird's-eye image and presenting the message image together with the second bird's-eye image has occurred. Can be conveyed more accurately.

  In the present invention, within the scope of the invention, any combination of the embodiments, or any modification of any component in each embodiment, or omission of any component in each embodiment is possible. .

  DESCRIPTION OF SYMBOLS 1 Own vehicle, 2 External imaging device, 11 1st camera, 12 Reception part, 13 Image generation apparatus, 14 Display apparatus, 15 Navigation apparatus, 16 Map information storage part, 21 2nd camera, 22 2nd bird's-eye view image generation part, 23 information adding unit, 24 transmitting unit, 31 first bird's-eye image generating unit, 32 mobile object information generating unit, 33 second bird's-eye image decoding unit, 34 combining information decoding unit, 35 display image generating unit, 36 image output unit , 37 Second bird's-eye image generation unit, 38 Presentation information decoding unit, 39 Message image generation unit, 40 Display position determination unit, 41 Storage device, 42 Processing circuit, 51 Region determination unit, 52 Difference calculation unit, 53 Bird's-eye image synthesis Unit, 54 a trajectory image generation unit, 55 a trajectory image superimposition unit, 61 a map image generation unit, 62 a bird's eye image superimposition unit, and 63 a bird's eye image transmission unit.

Claims (8)

A first bird's-eye image generation unit that generates a first bird's-eye image from an image captured using a first camera mounted on the first moving body;
A display image including the first bird's-eye image and a second bird's-eye image generated from an image photographed by an external photographing device arranged as a separate body apart from the first moving body using a second camera. A display image generating unit to generate,
An image output unit for outputting the display image to a display device provided in the first moving body,
The display image generation unit generates the display image by combining the second bird's-eye image with the first bird's-eye image,
The display image generation unit indicates the first bird's-eye view image in an area other than the imaging area of the second camera in the imaging area of the first camera, and the imaging area of the second camera. The area excluding the shooting area of the first camera generates the display image indicating the second bird's-eye view image,
In the display image generation unit, an area where the imaging area of the first camera and the imaging area of the second camera overlap is an area where a difference value between the first bird's-eye image and the second bird's-eye image is a threshold value or more. the second shows a bird's-eye image, and the difference value area is smaller than the threshold value element image generating device moves you and generates the display image showing the first bird's-eye image. The display image generation unit generates a trajectory image indicating a predicted travel trajectory of the first moving body, and generates the display image in which the trajectory image is superimposed on the first bird's-eye image and the second bird's-eye image. The moving body image generating apparatus according to claim 1 . A first bird's-eye image generation unit that generates a first bird's-eye image from an image captured using a first camera mounted on the first moving body;
A display image including the first bird's-eye image and a second bird's-eye image generated from an image photographed by an external photographing device arranged as a separate body apart from the first moving body using a second camera. A display image generating unit to generate,
An image output unit for outputting the display image to a display device provided in the first moving body,
The display image generation unit generates a map image including a travel route of the first moving body, and superimposes the first bird's-eye image and the second bird's-eye image on a region including the travel route of the map image. moving object image generating device moves you and generating a display image. The display image generation unit translucently displays the second bird's-eye view image generated from an image captured by the external imaging device at a position where the distance from the first moving body exceeds a reference distance. 4. The moving body image generating apparatus according to claim 3 . The display image generation unit generates the display image with higher transparency as the second bird's-eye image generated from the image captured by the external imaging device located at a position away from the first moving body. The moving body image generating apparatus according to claim 3, wherein: The moving image generating apparatus according to any one of claims 1 to 5 , wherein the external photographing device includes a monitoring camera device installed on a street. A first bird's-eye image generation unit that generates a first bird's-eye image from an image captured using a first camera mounted on the first moving body;
A display image including the first bird's-eye image and a second bird's-eye image generated from an image photographed by an external photographing device arranged as a separate body apart from the first moving body using a second camera. A display image generating unit to generate,
An image output unit for outputting the display image to a display device provided in the first moving body,
The external photographing device includes a photographing device mounted on a second moving body different from the first moving body,
Using the presentation information received from the second mobile body by inter-vehicle communication, a message image generation unit that generates a message image indicating the content of the presentation information;
A display position determination unit that determines the display position of the message image in the display area of the display device according to the position of the external photographing device;
The image output unit, the message image the display position determining unit to output to the display device so as to display on the display position determined by the image generation apparatus for be that moving body, wherein. A navigation apparatus comprising the moving body image generating apparatus according to any one of claims 1 to 7 .

Images