JP4818862B2 - Composite video display device - Google Patents

Description

  The present invention relates to a composite video display device that displays on a display device a composite video obtained by synthesizing captured video images taken by a plurality of imaging devices that capture a vehicle periphery.

As a conventional composite video display device, a flag table associated with each pixel position of a plurality of imaging devices is prepared, and a pixel displayed at a pixel position corresponding to a pixel area previously written in the flag table The average value of the data is corrected to be equal, and the average value is obtained from the pixel data of the overlapping shooting regions in the adjacent camera images, so that a composite video that does not cause a sense of incongruity at the joints of the plurality of shooting devices is obtained. Some are generated in real time from each captured video (see, for example, Patent Document 1).
JP 2002-324235 A

  However, in the conventional technique described above, correction is performed so that a captured image captured by one image capturing apparatus is matched with a captured image captured by another image capturing apparatus. In addition, there is a problem that a difference may occur in the color scheme of the synthesized video outside the vehicle.

  The present invention has been made in order to solve the conventional problems, and suppresses a difference in color scheme that occurs in the composite image with respect to the outside of the actual vehicle, and generates a composite image that does not cause a sense of incongruity at the joint. An object of the present invention is to provide a composite video display device that can be generated from a captured video.

  The composite video display device of the present invention includes video adjustment means for performing video adjustment on each shot video shot by a plurality of shooting devices that shoot the periphery of the vehicle, and combines the shot videos that have been subjected to the video adjustment. In the composite video display device for displaying a video on a display device, the video adjustment unit may include the imaging pixel of the captured video captured by another imaging device among the captured pixels of the captured video targeted for the video adjustment. The first statistical value obtained by statistically calculating the pixel value of the corresponding photographing pixel in the composite image, and the photographed image to be subjected to the image adjustment among the photographing pixels of the photographed image photographed by the other photographing device A second statistical value obtained by statistically calculating a pixel value of a photographing pixel that overlaps and corresponds to the photographed pixel in the synthesized video, and an average value of the first statistical value and the second statistical value is calculated as the first statistical value. 1 series The composite gain is calculated from a shooting pixel corresponding to an optical axis of a shooting device that has shot the shot image to be image-adjusted by calculating a target gain divided by a value, to a shooting pixel of the shot image shot by the other shooting device. The adjustment gain of the image adjustment to be applied to the corresponding photographic pixels is changed from 1 to the target gain.

  With this configuration, the composite video display device of the present invention is gradually close in brightness, etc., between each shot video and the shot video shot by another shooting device whose shooting range overlaps with the shooting device that shot each shot video. Therefore, it is possible to generate a composite image from each of the plurality of photographing devices that does not cause a sense of incongruity while suppressing a difference in color scheme that occurs in the composite image with respect to the outside of the actual vehicle. .

  The video adjusting means may perform the video adjustment on each captured video using the adjustment gain calculated one field before.

  With this configuration, the composite video display device of the present invention can perform video adjustment while displaying the composite video on the display device, so that the composite video can be displayed on the display device in real time.

  In addition, the video adjustment unit may calculate the adjustment gain while displaying each pixel of the composite video on the display device.

  With this configuration, the composite video display device of the present invention calculates the adjustment gain while displaying the composite video on the display device, so that the composite video can be displayed on the display device in real time.

  Note that the above-described composite video display device may be configured by an integrated circuit.

  The present invention is a composition having an effect that it is possible to generate a composite image that does not cause a sense of incongruity between joint images from a plurality of photographed images of a plurality of photographing devices while suppressing a difference in color scheme that occurs in the composite image with respect to an actual vehicle. A video display device can be provided.

  Embodiments of the present invention will be described below with reference to the drawings. In the present embodiment, an example in which the composite video display device of the present invention is configured by an ECU (Electronic Control Unit) provided in a vehicle, and the video adjustment means of the present invention is configured by a DSP (Digital Signal Processor). explain.

  FIG. 1 shows an ECU according to an embodiment of the present invention.

  Connected to the ECU 1 are cameras 2a to 2d as a plurality of photographing devices for photographing the periphery of the vehicle and a display device 4 for outputting a video signal processed by the ECU 1.

  As shown in FIG. 2, the cameras 2a to 2d are attached to the vehicle so as to shoot the shooting ranges 20a to 20d on the left side, the front side, the right side, and the rear side of the vehicle, respectively. It is designed to generate.

  Here, of the shooting range 20a of the camera 2a, a range that does not overlap with the shooting ranges 20b to 20d of the other cameras 2b to 2d is defined as a shooting range 21a. Similarly, of the shooting ranges 20b to 20d of the cameras 2b to 2d, ranges that do not overlap with shooting ranges of other cameras are set as shooting ranges 21b to 21d, respectively.

  In addition, among the shooting ranges that overlap between the shooting range 20a of the camera 2a and the shooting range 20b of the camera 2b, a range in which the shooting pixel corresponds to a pixel position of a composite video described later is set as a shooting range 22a, and the shooting pixel is combined. A range that does not correspond to the pixel position of the video is defined as a shooting range 23a.

  Similarly, of the overlapping shooting ranges between the shooting range 20b of the camera 2b and the shooting range 20c of the camera 2c, the range in which the shooting pixel corresponds to the pixel position of the composite video is the shooting range 22b, and the shooting pixel is the composite video. A range that does not correspond to the pixel position is set as an imaging range 23b.

  Similarly, of the overlapping shooting ranges between the shooting range 20c of the camera 2c and the shooting range 20d of the camera 2d, the range where the shooting pixel corresponds to the pixel position of the composite video is set as the shooting range 22c. A range that does not correspond to the pixel position of the composite video is defined as a shooting range 23c.

  Similarly, of the overlapping shooting ranges between the shooting range 20d of the camera 2d and the shooting range 20a of the camera 2a, the range in which the shooting pixel corresponds to the pixel position of the composite video is defined as the shooting range 22d. A range that does not correspond to the pixel position of the synthesized video is defined as a shooting range 23d.

  In FIG. 1, the ECU 1 includes video memories 10a to 10d for storing video data obtained from video signals representing captured images respectively captured by the cameras 2a to 2d, and videos stored in the video memories 10a to 10d, respectively. In order to synthesize a photographed image with a DSP 11 programmed to synthesize the photographed images taken by the cameras 2a to 2d based on the data and display the synthesized image in which the periphery of the vehicle is viewed from the sky on the display device 4. And a flash memory 12 for storing information and the like referred to by the DSP 11.

  The flash memory 12 stores in advance image data representing an image of the vehicle (hereinafter simply referred to as “vehicle image data”) and map information.

  The map information has a number of elements equal to the number of pixels of the composite video, and is composed of a table representing the correspondence between the pixel positions of the composite video and the captured pixels of the captured images captured by the cameras 2a to 2d.

  Each element of the map information represents a photographic pixel corresponding to the corresponding pixel position of the composite video, and by arranging the photographic pixel corresponding to the corresponding pixel position, a composite video in which the periphery of the vehicle is looked down from above is obtained in advance. It has been established. In the present embodiment, each element of the map information represents each photographic pixel with a memory address indicating any position in the storage area of the video memories 10a to 10d.

  The DSP 11 displays the synthesized video on the display device 4 by reading each pixel value of the synthesized video from the video memories 10a to 10d storing the video data according to the map information.

  For example, as shown in FIG. 3, when the value of the element (1, C) of the map information 40 is (0, C), the DSP 11 stores the storage areas of the video memories 10a to 10d in which the video data is stored. The pixel value obtained from the position indicated by the memory address (0, C) that is the value of the element in 41 is read as the pixel value of the pixel position (1, C) of the synthesized video 41.

  FIG. 4 shows a synthesized video synthesized by the DSP 11 according to the map information.

  As shown in FIGS. 2 and 4, the map information associates each photographic pixel forming the video in the photographic range 21a in the photographic video taken by the camera 2a with each pixel position in the video area 31a in the composite video. ing.

  Similarly, the map information is obtained by mapping each captured pixel that forms a video in the shooting range 21b to 21d in each of the captured images captured by the cameras 2b to 2d at each pixel position in the video areas 31b to 31d in the composite video. Corresponds.

  In the map information, each captured pixel forming the video in the shooting range 22a in the captured video captured by the cameras 2a and 2b is associated with each pixel position in the video area 32a in the composite video in an overlapping manner. .

  Similarly, in the map information, each shooting pixel forming the video in the shooting range 22b in the shot video shot by the cameras 2b and 2c is associated with each pixel position in the video area 32b in the composite video in an overlapping manner. Yes.

  Similarly, map information corresponds to each pixel position in the video area 32c in the composite video by overlapping each video pixel forming the video in the video range 22c in the video captured by the cameras 2c and 2d. Attached.

  Likewise, the map information corresponds to each pixel position in the video area 32d in the composite video by overlapping each captured pixel forming the video in the shooting range 22d in the captured video captured by the cameras 2d and 2a. Attached.

  In the following description, each of the video areas 32a to 32d is referred to as an “overlapping video area”. In the synthesized video, the region 33 that does not correspond to any shooting pixel is referred to as a “non-video region”.

  In FIG. 1, the DSP 11 displays a difference in luminance between captured images captured by cameras with overlapping imaging ranges, such as the camera 2 a and the camera 2 b, before displaying the synthesized image on the display device 4. Image adjustment is performed to adjust the pixel values read from the video memories 10a to 10d so as to decrease.

  Further, as shown in FIG. 5, the DSP 11 reads each pixel value from the vehicle image data stored in the flash memory 12 and causes the display device 4 to display each pixel in the non-video area 33 with each read pixel value. Thus, the vehicle image 35 is superimposed on the synthesized video.

  The operation of the ECU 1 configured as described above will be described with reference to FIG.

  First, video data obtained from video signals representing shot images taken by the cameras 2a to 2d are stored in the video memories 10a to 10d by the DSP 11 (S1).

  Next, the DSP 11 executes the following processing for each scan (S2 to S17) and each pixel (S3 to S16) for one field of the display device 4.

  First, the DSP 11 reads out the pixel value of the shooting pixel associated with the corresponding pixel position by the map information from the video data stored in the video memories 10a to 10d (S4).

  Next, the DSP 11 determines whether or not the corresponding pixel position is included in the non-video area 33 (S5). If it is determined that the corresponding pixel position is included in the non-video area 33, the DSP 11 reads out the pixel value corresponding to the corresponding pixel position from the vehicle image data stored in the flash memory 12 (S6), and reads out the pixel value. The corresponding pixel is displayed on the display device 4 with the obtained pixel value (S7).

  On the other hand, when it is determined that the corresponding pixel position is not included in the non-video area 33, the DSP 11 determines whether the corresponding pixel position is included in any overlapping video area (S8). If it is determined that the corresponding pixel position is not included in any overlapping video area, the DSP 11 determines the luminance component of the pixel value of the photographic pixel read from the video data stored in the video memories 10a to 10d. An adjustment gain for adjustment is calculated (S9).

  Here, the DSP 11 calculates the adjustment gain from 1 to the target gain from the optical axis of the camera to the overlapping video area closest to the pixel position. Note that the target gain used here is calculated in step S18, which will be described later, processed for the previous field. If the target gain has not been calculated yet, the DSP 11 operates the ECU 1 last time. The target gain calculated last may be used, and Steps S9 and S10 and Steps S13 and S14 described later may be omitted.

  For example, as shown in FIG. 7, in the synthesized image, the position of the corresponding pixel 37 is in the video area 31a, and the length of the perpendicular line 39 drawn from the position of the corresponding pixel 37 to the line 38 corresponding to the optical axis of the camera 2a is expressed as Lx. When the length from the position of the corresponding pixel 37 to the closest overlapping video region 32a is Ly and the target gain is Gab on the straight line obtained by extending the perpendicular line 39, the DSP 11 determines the luminance of the pixel value of the photographing pixel. As the adjustment gain G for adjusting the component, 1+ (Gab−1) × Lx / (Lx + Ly) is calculated.

  In FIG. 6, the DSP 11 adjusts the luminance component of the pixel value of the photographic pixel with the adjustment gain calculated as described above (S10), and causes the display device 4 to display the corresponding pixel with the pixel value adjusted with the luminance component (S11). ).

  On the other hand, when it is determined in step S8 that the corresponding pixel position is included in any of the overlapping video areas, the DSP 11 determines the pixel value of each shooting pixel read from the video data stored in the video memories 10a to 10d. The luminance component is added for each of the overlapping video areas 32a to 32d and the cameras 2a to 2d corresponding to the overlapping video areas 32a to 32d (S12). Each added value thus added is stored in the flash memory 12 by the DSP 11 as shown in FIG.

  In FIG. 6, the DSP 11 calculates an adjustment gain for adjusting the luminance component of the pixel value of each shooting pixel that is associated with the corresponding pixel position in an overlapping manner (S 13), and each shooting pixel with the calculated adjustment gain. The luminance component of the pixel value is adjusted (S14), and the corresponding pixel is displayed on the display device 4 with the pixel value obtained by averaging the luminance components of the adjusted pixel values (S15).

  When the above processing for each scanning (S2 to S17) and each pixel (S3 to S16) for one field of the display device 4 is completed, the DSP 11 performs the following based on the addition value added in step S12. A target gain for adjusting each of the shot images shot by the cameras 2a to 2d in the field is calculated (S18).

  FIG. 9 is a flowchart showing step S18 in detail. In the following, each camera whose shooting range overlaps with a specific camera is referred to as an “adjacent camera”. For example, the camera adjacent to the camera 2a refers to the camera 2d and the camera 2b.

  First, for each adjacent camera (S21 to S24) of each camera 2a to 2d (S20 to S25), the DSP 11 stores in the flash memory 12 for an overlapping video area corresponding to an imaging range where the camera and the adjacent camera overlap. An average value of the stored addition values for each camera (see FIG. 8) is calculated (S22), and a gain for making the addition value for the corresponding camera equal to the average value is calculated as a target gain (S23).

  For example, when the corresponding camera is the camera 2a and the adjacent camera is the camera 2b, the DSP 11 uses the addition value Pab for the camera 2a in the overlapping video area 32a and the addition value Pba for the camera 2b in the overlapping video area 32a, The target gain Gab = ((Pab + Pba) / 2) / Pab is calculated.

  When the corresponding camera is the camera 2a and the adjacent camera is the camera 2d, the DSP 11 uses the addition value Pad for the camera 2a in the overlapping video area 32d and the addition value Pda for the camera 2d in the overlapping video area 32d, Target gain Gad = ((Pda + Pad) / 2) / Pad is calculated.

  As described above, the DSP 11 calculates a target gain for adjusting each of the captured images captured by the cameras 2a to 2d. Here, the DSP 11 prevents the brightness of the composite video displayed on the display device 4 from changing abruptly due to the above-described processing, the target gain calculated this time, the target gain calculated last time, The amount of fluctuation may be suppressed.

For example, the target gain calculated in step S23 is G _g , and the previously calculated target gain is G _n−1 (where n is the execution of step S23 for each of the overlapping video regions 32a to 32d and each of the cameras 2a to 2d). When the number of times is an integer equal to or greater than 1 and G ₀ is 1.) and the suppression rate of the target gain fluctuation amount is w, the DSP 11 sets G _g − (G _g − as the current target gain G _n. G _n-1 ) × w may be calculated.

In place of this, the DSP 11 uses a predetermined variation A so that it decreases as the value of (G _g −G _n−1 ) decreases, and G _n− as the current target gain G _{n. 1} + A may be calculated.

  In FIG. 6, when the target gain for adjusting each of the captured images captured by the cameras 2 a to 2 d is calculated, the operation of the ECU 1 returns to step S <b> 1.

  In the ECU 1 according to the embodiment of the present invention, the luminance of each photographed video and each photographed image photographed by another photographing device whose photographing range overlaps with the photographing device that photographed each photographed video is gradually increased. In order to adjust the video so that they are close to each other, it is possible to generate a composite video from each video of a plurality of imaging devices that suppresses a difference in color scheme that occurs in the composite video with respect to the actual outside of the vehicle and does not cause discomfort at the joints. it can.

  In the present embodiment, an example in which the composite video generation means in the present invention is configured by a DSP and a flash memory has been described, but it may also be configured by another processor such as a CPU (Central Processing Unit) or an integrated circuit. Good.

  In this embodiment, the DSP 11 is described as adjusting the luminance component of the pixel value of each shooting pixel. However, the DSP 11 further adjusts the color difference component of the pixel value of each shooting pixel in the same manner as the luminance component. Alternatively, instead of the luminance component, the respective color components of the three primary colors of red, green and blue may be adjusted in the same manner as the luminance component.

  As described above, the composite video display device according to the present invention suppresses the difference in the color scheme that occurs in the composite video with respect to the outside of the actual vehicle, and displays the composite video that does not cause discomfort at the joints. For example, it is useful as a composite video display device for generating a composite video obtained by combining video images captured by a plurality of video capture devices that capture the periphery of a vehicle.

The block diagram of ECU in one embodiment of the present invention Mounting diagram of a camera connected to an ECU according to an embodiment of the present invention The conceptual diagram for demonstrating the synthetic | combination of the image | video by DSP which comprises ECU in one embodiment of this invention The conceptual diagram which shows the synthetic | combination image | video produced | generated by arrange | positioning the imaging | photography pixel which makes each imaging | photography image | video each image | photographed with the camera connected to ECU in one embodiment of this invention based on map information The conceptual diagram which shows the synthetic | combination image | video displayed on the display apparatus connected to ECU in one embodiment of this invention The flowchart for operation | movement description of ECU in one embodiment of this invention The conceptual diagram for demonstrating calculation of the adjustment gain by DSP which comprises ECU in one embodiment of this invention The conceptual diagram which shows the addition table of the luminance component produced | generated and referred by DSP which comprises ECU in one embodiment of this invention Flow chart for explaining in detail a part of the flow shown in FIG.

Explanation of symbols

1 ECU
2a, 2b, 2c, 2d Camera 4 Display device 10a, 10b, 10c, 10d Video memory 11 DSP
12 Flash memory

Claims (4)

A composite image that includes a video adjustment unit that performs image adjustment on each captured image captured by a plurality of imaging devices that capture the periphery of the vehicle, and displays a composite image obtained by combining the captured images that have undergone the image adjustment on a display device In the display device,
The video adjustment means includes a pixel value of a photographic pixel corresponding to the photographic pixel of the photographic video photographed by another photographic device in the composite video, among the photographic pixels of the photographic video to be subject to the video adjustment. Among the first statistic value obtained by statistic and the photographic pixel of the photographic image taken by the other photographic device, corresponding to the photographic pixel of the photographic image to be subject to the video adjustment in the composite video. Calculating a second statistical value obtained by statistically calculating the pixel value of the photographing pixel, and calculating a target gain obtained by dividing an average value of the first statistical value and the second statistical value by the first statistical value; Applying from the photographic pixel corresponding to the optical axis of the photographic device that photographed the photographic image to be subject to the video adjustment to the photographic pixel corresponding to the photographic pixel of the photographic image photographed by the other photographing device in the composite video. Serial mixed video display device, characterized in that the adjusting gain of a picture prepared from 1 gradually varied to the target gain. 2. The composite video display apparatus according to claim 1, wherein the video adjustment unit performs the video adjustment on each captured video using an adjustment gain calculated one field before. The composite video display device according to claim 2, wherein the video adjustment unit calculates the adjustment gain while displaying each pixel of the composite video on the display device. An integrated circuit comprising video adjusting means for performing video adjustment on each shot video shot by a plurality of shooting devices for shooting the periphery of a vehicle, and displaying a synthesized video obtained by synthesizing each shot video subjected to the video adjustment on a display device In
The video adjustment means includes a pixel value of a photographic pixel corresponding to the photographic pixel of the photographic video photographed by another photographic device in the composite video, among the photographic pixels of the photographic video to be subject to the video adjustment. Among the first statistic value obtained by statistic and the photographic pixel of the photographic image taken by the other photographic device, corresponding to the photographic pixel of the photographic image to be subject to the video adjustment in the composite video. Calculating a second statistical value obtained by statistically calculating the pixel value of the photographing pixel, and calculating a target gain obtained by dividing an average value of the first statistical value and the second statistical value by the first statistical value; Applying from the photographic pixel corresponding to the optical axis of the photographic device that photographed the photographic image to be subject to the video adjustment to the photographic pixel corresponding to the photographic pixel of the photographic image photographed by the other photographing device in the composite video. Integrated circuit, characterized in that the adjusting gain of the serial image adjustment from 1 gradually varied to the target gain.

Images