JP7712460B2 - DISPLAY SYSTEM, MOBILE BODY, IMAGE PROCESSING DEVICE, AND DISPLAY SYSTEM - Google Patents

Description

The present invention relates to a display system, a moving object, an image processing device, and a display system .

Conventionally, there are known techniques for displaying images of the external area of a moving object such as a vehicle. For example, Patent Document 1 discloses a technique for controlling the power supply to a monitor that displays images from a camera installed in a vehicle.

JP 2009-40113 A

Until now, there has been room for improvement in the technology used to display images of the external area of a moving object.

The present disclosure relates to an image processing device, an imaging device, a display device, and a display system that improve the convenience of technology for displaying images of the external area of a moving object.

A display system according to an embodiment of the present disclosure is a display system mounted on a vehicle,
The vehicle further includes a control means for causing a marker to be displayed on a display means when an object is detected to the right rear or left rear of the vehicle while the vehicle is moving at a speed less than a reference value .

A moving object according to an embodiment of the present disclosure includes the above-described display system .

An image processing device according to an embodiment of the present disclosure is an image processing device mounted on a vehicle,
The control unit causes a marker to be displayed on a display means when an object is detected to the right rear or left rear of the vehicle while the vehicle is moving at a speed less than a reference value .

A display system according to one embodiment of the present disclosure is a display system mounted on a vehicle, and includes an imaging means for imaging the rear of the vehicle and generating a second image, and a control means for superimposing a marker on the second image when an object is detected to the right rear or left rear of the vehicle while the vehicle is moving at a speed less than a reference value .

An image processing device according to one embodiment of the present disclosure is an image processing device mounted on a vehicle having a display means for displaying images, and is equipped with a control means for causing the display means to display a marker indicating that an object has been detected to the right rear or left rear of the vehicle moving at a moving speed less than a reference value .

The image processing device, imaging device, display device, and display system according to an embodiment of the present disclosure improve the convenience of the technology for displaying images of the external area of a moving object.

1 is a block diagram showing a schematic configuration of a display system according to an embodiment of the present invention. FIG. 2 is a view of a vehicle equipped with a display system as viewed from the left side. FIG. 2 is a diagram illustrating an external appearance of the display device of FIG. 1 . A figure showing a first example of a first image. 5 is a diagram showing a first example of a second image corresponding to the display area of the first image in FIG. 4 . 13A and 13B are diagrams illustrating an example of a third marker superimposed on a detection target. 1A and 1B are diagrams illustrating a first example of a first marker and a second marker displayed around a detection target. 13A and 13B are diagrams illustrating a second example of the first marker and the second marker displayed around the detection target. 13A and 13B are diagrams illustrating a third example of a first marker and a second marker displayed around a detection target. FIG. 13 is a diagram showing a second example of the first image. 11 is a diagram showing a second example of a second image corresponding to the display area of the first image in FIG. 10 . A figure showing a third example of the first image. 13 is a diagram showing a third example of a second image corresponding to the display area of the first image in FIG. 12 . A figure showing a fourth example of the first image. FIG. 15 is a diagram showing a fourth example of a second image corresponding to the display area of the first image in FIG. 14 . A figure showing a fifth example of the first image. FIG. 17 is a diagram showing a fifth example of a second image corresponding to the display area of the first image in FIG. 16 . 17 is a diagram showing another example of the second image corresponding to the display area of the first image in FIG. 16 . A figure showing a sixth example of the first image. A figure showing a sixth example of a second image corresponding to the display area of the first image in Figure 19. FIG. 13 is a diagram showing another example of the second image. A figure showing a first modified example of the sixth example of the second image. FIG. 13 is a diagram showing a second modified example of the sixth example of the second image; A figure showing a third modified example of the sixth example of the second image. A figure showing a seventh example of the first image. A figure showing a seventh example of the second image corresponding to the display area of the first image in Figure 25. A figure showing an eighth example of the first image. FIG. 2 is a diagram showing the positional relationship between a vehicle and an object to be detected; A figure showing an eighth example of the second image corresponding to the display area of the first image in Figure 24. A figure showing a modified example of the eighth example of the second image. FIG. 31 is a diagram showing an example in which the display range of FIG. 30 is changed to a wide view display.

The following describes an embodiment of the present invention with reference to the drawings.

(Display System)
Referring to FIG. 1, a display system 10 according to an embodiment of the present invention will be described.

As shown in FIG. 1, the display system 10 includes an imaging device 20, an image processing device 30, and a display device 40. The imaging device 20 and each component of the display system 10 can transmit and receive information via, for example, a network 51. The network 51 may include, for example, a wireless network, a wired network, or a CAN (Controller Area Network), etc.

In other embodiments, some or all of the components of the display system 10 may be integrated into a single device. For example, the image processing device 30 may be built into the imaging device 20 or the display device 40.

As shown in FIG. 2, the imaging device 20, the image processing device 30, and the display device 40 may be provided in a moving object 50. The "moving object" in this disclosure may include, for example, vehicles, ships, and aircraft. The vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, residential vehicles, and fixed-wing aircraft running on runways. The automobiles may include, for example, passenger cars, trucks, buses, motorcycles, and trolley buses. The industrial vehicles may include, for example, industrial vehicles for agriculture and construction. The industrial vehicles may include, for example, forklifts and golf carts. The industrial vehicles for agriculture may include, for example, tractors, cultivators, transplanters, binders, combines, and lawnmowers. The industrial vehicles for construction may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, and road rollers. The vehicles may include those that run by human power. The classification of vehicles is not limited to the above examples. For example, the automobile may include an industrial vehicle that can run on a road. The same vehicle may be included in multiple classifications. Marine vehicles may include, for example, watercraft, boats, and tankers. Aircraft may include, for example, fixed-wing and rotary-wing aircraft.

The imaging device 20 is capable of capturing an image of the external area of the moving body 50. The position of the imaging device 20 may be any position inside or outside the moving body 50. For example, as shown in FIG. 2, the imaging device 20 is located at the rear of the moving body 50 and is capable of capturing an image of the external area behind the moving body 50. The position of the image processing device 30 may be any position within the moving body 50. The display device 40 is visible to the subject 60. The position of the display device 40 may be any position within the moving body 50. For example, as shown in FIG. 2, the display device 40 is located within the dashboard of the moving body 50.

(Imaging device)
The imaging device 20 will be described in detail below. For example, as shown in FIG 1, the imaging device 20 includes an imaging optical system 21, an imaging element 22, a communication unit 23, and a control unit 24.

The imaging optical system 21 forms an image of a subject. For example, the imaging optical system 21 may include an aperture and one or more lenses.

The imaging element 22 has a plurality of pixels arranged two-dimensionally. The imaging element 22 may include, for example, a charge coupled device (CCD) imaging element or a complementary metal oxide semiconductor (CMOS) imaging element. The imaging element 22 can capture an image of a subject formed by the imaging optical system 21 and generate a captured image.

The communication unit 23 may include a communication interface capable of communicating with an external device. The communication unit 23 may be capable of transmitting and receiving information via the network 51. The external device may include, for example, the image processing device 30. The "communication interface" in this disclosure may include, for example, a physical connector and a wireless communication device. The physical connector may include an electrical connector compatible with transmission by electrical signals, an optical connector compatible with transmission by optical signals, and an electromagnetic connector compatible with transmission by electromagnetic waves. The electrical connector may include a connector conforming to IEC 60603, a connector conforming to the USB standard, a connector compatible with an RCA terminal, a connector compatible with an S terminal defined in EIAJ CP-1211A, a connector compatible with a D terminal defined in EIAJ RC-5237, a connector conforming to the HDMI (registered trademark) standard, and a connector compatible with a coaxial cable including BNC (British Naval Connector or Baby-series N Connector, etc.). The optical connector may include various connectors conforming to IEC 61754. The wireless communication device may include a wireless communication device that complies with various standards including Bluetooth (registered trademark) and IEEE 802.11. The wireless communication device includes at least one antenna.

The control unit 24 includes one or more processors. In this disclosure, a "processor" may include a dedicated processor specialized for a specific process, and a general-purpose processor that executes a specific function by loading a specific program. The dedicated processor may include a digital signal processor (DSP) and an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 24 may be either a system-on-a-chip (SoC) in which one or more processors work together, or a system in a package (SiP).

The control unit 24 controls the operation of the entire imaging device 20. The control unit 24 may cause the imaging element 22 to generate an image at an arbitrary frame rate. The frame rate may be, for example, approximately equal to the frame rate at which the display device 40 can display. The control unit 24 may execute a predetermined image processing on the generated image. The image processing may include, for example, exposure adjustment processing, white balance processing, and distortion correction processing. The control unit 24 outputs the image to the image processing device 30 via the communication unit 23. For example, the control unit 24 may sequentially output the image at the above-mentioned frame rate. Hereinafter, each image output at the above-mentioned frame rate is also simply referred to as a frame. The multiple images output from the imaging device 20 are also referred to as a first image. For example, when the frame rate is 60 fps (Flame per Seconds), 60 images per second are output as the first image.

(Image Processing Device)
The image processing device 30 will now be described in detail. The image processing device 30 includes a communication unit 31, a storage unit 32, and a control unit 33.

The communication unit 31 may include a communication interface capable of communicating with various external devices. The external devices may include, for example, the imaging device 20, the display device 40, an ECU (Electronic Control Unit or Engine Control unit) provided in the mobile object 50, a speed sensor, an acceleration sensor, a rotation angle sensor, a steering angle sensor, an engine RPM sensor, an accelerator sensor, a brake sensor, an illuminance sensor, a raindrop sensor, a mileage sensor, an obstacle detection device using millimeter wave radar, ultrasonic sonar, etc., an ETC (Electronic Toll Collection System) receiving device, a GPS (Global Positioning System) device, a navigation device, a server on the Internet, a mobile phone, etc.

The communication unit 31 may include a communication interface for pedestrian-vehicle communication, road-to-vehicle communication, and vehicle-to-vehicle communication. The communication unit 31 may include a receiver compatible with optical beacons of DSRC (Dedicated Short-Range Communication) and VICS (Vehicle Information and Communication System) provided in Japan. The communication unit 31 may include a receiver compatible with road traffic information systems of other countries.

The communication unit 31 may be capable of acquiring various information from an external device. For example, the communication unit 31 may be capable of acquiring mobile object information and environmental information.

The mobile body information may include any information related to the mobile body 50. The mobile body information may include, for example, the speed, acceleration, turning gravity, inclination, direction, and turning status of the mobile body 50, the steering angle of the steering wheel, the temperature of the cooling water, the remaining amount of fuel, the remaining amount of battery, the battery voltage, the engine speed, the gear position, the presence or absence of a reverse signal, the presence or absence of accelerator operation, the accelerator opening, the presence or absence of brake operation, the brake depression, the presence or absence of operation of the parking brake, the difference in the rotation speed of the front and rear wheels or the four wheels, the tire pressure, the expansion and contraction amount of the damper, the spatial position of the driver's eyes, the number and seat position of the occupants, the fastening information of the seat belt, the opening and closing of the doors, the opening and closing of the windows, the temperature inside the vehicle, the operation of the air conditioning, the set temperature of the air conditioning, the air conditioning blowing volume, the setting of the outside air circulation, the operation status of the wipers, the driving mode, the connection information with the external device, the current time, the average fuel consumption, the instantaneous fuel consumption, the lighting status of various lamps, the position information of the mobile body 50, and the route information to the destination of the mobile body 50. The various lamps may include, for example, headlamps, fog lamps, backup lamps, position lamps, and turn signals.

The environmental information may include any information related to the external environment of the mobile unit 50. The environmental information may include, for example, the brightness around the mobile unit 50, the weather, the air pressure, the outside temperature, map information, traffic information, road construction information, temporary changes in the speed limit of the road, objects detected by other vehicles, and the status of traffic lights.

The storage unit 32 may include a temporary storage device and a secondary storage device. The storage unit 32 may be configured using, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The semiconductor memory may include a volatile memory and a non-volatile memory. The magnetic memory may include, for example, a hard disk and a magnetic tape, etc. The optical memory may include, for example, a CD (Compact Disc), a DVD (Digital Versatile Disc), and a BD (Blu-ray Disc) (registered trademark), etc. The storage unit 32 stores various information and programs necessary for the operation of the image processing device 30.

The control unit 33 includes one or more processors. The control unit 33 controls the operation of the entire image processing device 30.

The control unit 33 may acquire moving body information and environmental information from an external device via the communication unit 31. The control unit 33 may determine a predicted course of the moving body 50, for example, based on the moving body information. Hereinafter, the predicted course of the moving body 50 is also referred to as a first predicted course.

The control unit 33 may acquire a first image from the imaging device 20 via the communication unit 31. The first image includes a detection area and a display area.

The control unit 33 may detect at least a portion of the detection target within the detection area on the acquired first image. The detection area on the first image may be at least a portion of the area on the captured image, which is each frame of the first image. Each frame of the first image may be referred to as a captured image. The detection area on the first image may be larger than the display area. The detection area on the first image may include the display area. The control unit 33 may detect the detection target inside the display area. The control unit 33 may detect the detection target outside the display area and inside the detection area. The area inside the detection area and the display area may be referred to as the first area. The area inside the detection area and outside the display area may be referred to as the second area.

The detection target may include a plurality of types of objects. The types of objects may include, for example, people, other moving objects, driving paths, lanes, white lines, gutters, sidewalks, crosswalks, road signs, traffic signs, guardrails, walls, and traffic lights. The types of detection targets that the control unit 33 can detect are not limited to these. For example, when a part of the detection target in the first image is hidden behind another object, at least a part of the detection target may include a part of the detection target that is not hidden behind the other object. For example, when the lower half of the pedestrian's body is hidden behind an obstacle in the first image, the control unit 33 may be able to detect the upper half of the pedestrian. Any object detection algorithm may be employed to detect at least a part of the detection target. For example, the control unit 33 may detect at least a part of the detection target by an algorithm such as pattern matching or feature point extraction using the captured images that are each frame of the first image.

When at least a portion of the detection target is detected on the first video, the control unit 33 may determine a predicted path of the detection target based on the first video. Hereinafter, the predicted path of the detection target is also referred to as the second predicted path. Any algorithm may be employed to determine the second predicted path. For example, the control unit 33 may determine the second predicted path based on changes in the orientation and position of the detection target on the captured image, which is each frame of the first video.

When at least a part of the detection target is detected on the first image, the control unit 33 may estimate the relative positional relationship between the moving body 50 and the detection target based on the first image. The relative positional relationship may include, for example, the distance between the moving body 50 and the detection target, and whether or not the first predicted path of the moving body 50 overlaps with the second predicted path of the detection target. Any algorithm may be adopted for estimating the distance between the moving body 50 and the detection target. For example, the control unit 33 may estimate the distance between the moving body 50 and the detection target by a motion stereo method using the captured images that are each frame of the first video signal. In another embodiment, the control unit 33 may obtain information indicating the relative positional relationship between the moving body 50 and the detection target from an external device via the communication unit 31.

When the distance between the moving body 50 and the detection target decreases, the control unit 33 may determine which of the moving body 50 and the detection target contributes more to the decrease in the distance. Any algorithm may be adopted to determine the contribution of the moving body 50 and the detection target to the decrease in the distance. In one example, the control unit 33 may detect the moving speed of the moving body 50 based on the moving body information. The control unit 33 may detect the moving speed of the detection target based on the change in the position of the detection target on the captured image, which is, for example, each frame of the first video. The control unit 33 may determine that the contribution of either the moving body 50 or the detection target, which has a higher moving speed, is greater to the decrease in the distance. In another example, the control unit 33 may determine that the contribution of the detection target is greater to the decrease in the distance when the moving speed of the moving body 50 is less than a reference value. The control unit 33 may determine that the contribution of the moving body 50 is greater to the decrease in the distance when the moving speed of the moving body 50 is equal to or greater than a reference value. The reference value may be set arbitrarily, for example, to approximately zero. Details of the operation of the image processing device 30 according to the contribution of the moving body 50 and the detection target to the decrease in the distance will be described later.

When at least a portion of the detection target is detected on the first image, the control unit 33 may determine whether or not there is a possibility that the moving body 50 and the detection target will come into contact based on the first image. Any algorithm may be employed to determine the possibility that the moving body 50 and the detection target will come into contact. For example, the control unit 33 may determine that there is a possibility that the moving body 50 and the detection target will come into contact when at least one of the following conditions is satisfied: the distance between the moving body 50 and the detection target is less than a predetermined threshold, and the rate at which the distance is decreasing is equal to or greater than a predetermined threshold. Details of the operation of the image processing device 30 according to the presence or absence of such possibility will be described later.

The control unit 33 may display the second image corresponding to the display area on the first image acquired from the imaging device 20 on the display device 40. Specifically, the control unit 33 may output the second image to the display device 40 via the communication unit 31. For example, when the control unit 33 detects the reverse movement of the moving body 50 based on the moving body information, the control unit 33 may display the second image on the display device 40. For example, the control unit 33 may detect the reverse movement based on the shift position of the transmission gear. For example, the control unit 33 may detect the reverse movement based on a reverse signal output from the moving body when the moving body is reversed. The second image may be, for example, an image obtained by cutting out the display area on the captured image, which is each frame of the first image. The display area on the first image may be at least a part of the area on the captured image, which is each frame of the first image. The display area may be smaller than the detection area. The display area may be included in the detection area. The position, shape, and size of the display area may be determined arbitrarily. The control unit 33 may change the position, shape, and size of the display area. The position, shape, and size of the display area may change, so that the display area and the detection area, for example, approximately coincide with each other.

The control unit 33 may composite various markers into the second image and display them on the display device 40. Compositing includes overwriting and mixing. The markers may include, for example, one or more images. The control unit 33 may dynamically change the display mode of at least a portion of the markers superimposed on the second image. The display mode may include, for example, the position, size, shape, color, and shading of at least a portion of the marker on the second image. When displaying a marker corresponding to a detection target detected on the first image, the control unit 33 may determine the display mode of the marker depending on the type of the detection target. Details of the operation of the image processing device 30 that displays various markers on the display device 40 will be described later.

(Display device)
The display device 40 will be described in detail. The display device 40 may include, for example, a liquid crystal display and an organic EL (Electroluminescence) display. The display device 40 may display, for example, a second image input from the image processing device 30 via a network 51. The display device 40 may function as a touch screen capable of accepting an operation by a user. The display device 40 may include a switch and a key capable of accepting an operation by a user. The switch may include a physical switch and an electronic switch. The key may include a physical key and an electronic key. When the display device 40 accepts a user operation on the display device itself, the display device 40 may transmit a user input based on the operation to the image processing device 30.

The display device 40 can be arranged in various places on the moving object 50. Figures 3A to 3E are diagrams showing examples of the arrangement of the display device 40 when the moving object 50 is a vehicle. Figure 3A shows an in-dashboard display device 40a stored in the dashboard of the vehicle. Figure 3B shows an on-dashboard display device 40b arranged on the dashboard. In Figure 3B, the display device 40b is built into the vehicle 50. The display device 40b may be detachably attached to the dashboard. Figure 3C shows a display device 40c that is built into the rearview mirror and can display an image as needed. Figure 4D shows a display device 40d built into the instrument panel. In Figure 4D, the display device 40d is arranged adjacent to instruments such as a speedometer and a tachometer. In one embodiment, the entire instrument panel may be a display device 40d such as an LCD, and a second image may be displayed therein along with images of the speedometer and tachometer. Figure 4E shows a display device 40e using a portable information terminal, for example, a tablet terminal. The display device 40e can be a mobile phone display.

The second image and various markers that the image processing device 30 causes the display device 40 to display will be specifically described with reference to Figures 4 to 18. In this disclosure, the terms "up-down direction" and "left-right direction" for an image or an image correspond to two-dimensional directions in the image or the image. In this disclosure, the terms "height direction," "horizontal direction," and "depth direction" for an image or an image correspond to three-dimensional directions in the space in which the image or the image is projected.

(First example (reference example))
Fig. 4 shows a first example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 4, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64 that are shown inside the display area 62 on the first image.

The control unit 33 determines whether or not one or more conditions are satisfied based on the relative positional relationship between the detection target detected inside the display area 62 on the first image and the moving object 50. The one or more conditions may include, for example, a first condition that the detection target is located within the first predicted path 65 of the moving object 50. The one or more conditions may include, for example, a second condition that at least a portion of the first predicted path 65 of the moving object 50 overlaps with at least a portion of the second predicted path of the detection target. If it is determined that the one or more conditions are satisfied, the control unit 33 may cause the display device 40 to display a predetermined marker corresponding to the detection target by superimposing it on the second image. The predetermined marker may include a first marker, a second marker, and a third marker.

In the first example, the control unit 33 may determine that the one or more conditions are satisfied for the pedestrian 63. In such a case, the control unit 33 may cause a marker corresponding to the pedestrian 63 to be displayed. The control unit 33 may determine that the one or more conditions are not satisfied for the vehicle 64. In such a case, the control unit 33 does not cause a marker corresponding to the vehicle 64 to be displayed.

Figure 5 shows an example of a second image corresponding to the display area 62 of the first image shown in Figure 4. When the aspect ratio of the display area 62 of the first image differs from the aspect ratio of the screen of the display device 40, the control unit 33 may cut out the display area 62 of the first image, and then output to the display device 40 the second image that has been transformed to match the aspect ratio of the screen of the display device 40. As shown in Figure 5, a pedestrian 63 and a vehicle 64 are shown in the second image.

The control unit 33 may, for example, cause the display device 40 to display guide lines 66 indicating at least a part of the first predicted path 65 of the moving body 50 shown in FIG. 4 by superimposing the guide lines 66 on the second image. The control unit 33 may dynamically change the guide lines 66 in response to changes in the steering angle of the steering wheel, for example.

The first image has a larger range than the display area 62. The control unit 33 may change the range of the display area 62. The control unit 33 may superimpose an icon image 67 on the second image and display it on the display device 40. For example, the outline 67a of the icon image 67 shown in FIG. 5 indicates the maximum range when the range of the display area 62 is changed. The white rectangle 67b of the icon image 67 indicates the display area 62. The icon image 67 shown in FIG. 5 indicates the relative position and size of the display area 62 with respect to the maximum range of the display area 62.

FIG. 6 shows an example of a marker superimposed on a pedestrian 63 on the second image. Hereinafter, the marker is also referred to as a third marker 68. A contour 69 of the third marker 68 may approximately match a contour of the pedestrian 63 detected on the second image. An area 70 inside the contour 69 of the third marker 68 may be filled with a color or pattern corresponding to a type of detection target, such as a "person". When a pedestrian 63 is detected on the first image, the control unit 33 may superimpose the third marker 68 on the pedestrian 63 on the second image and display it on the display device 40. With this configuration, the subject 60 can easily view the pedestrian 63 on the second image. When a predetermined time has elapsed since the control unit 33 displayed the third marker 68, the control unit 33 may hide the third marker 68.

Figure 7 shows an example of two types of markers superimposed around a pedestrian 63 on the second image, for example. Hereinafter, the two types of markers are also referred to as a first marker 71 and a second marker 72. For example, the control unit 33 may hide the third marker 68, and then cause the first marker 71 and the second marker 72 to be superimposed on the second image and displayed on the display device 40.

The control unit 33 may move the position of the first marker 71 to follow the pedestrian 63 on the second image. The subject 60 can easily catch the pedestrian 63 because the first marker 71 follows the pedestrian 63. The first marker 71 is displayed around the pedestrian 63 at a distance from the pedestrian 63. The subject 60 can easily grasp the behavior of the pedestrian 63 when the first marker 71 is displayed on the display device 40. The control unit 33 may change the superimposed position of the second marker 72 relative to the superimposed position of the first marker 71 on the second image. The control unit 33 may move the second marker relatively based on the position of the first marker 71.

For example, when the distance between the moving object 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the contribution of the moving object 50 to the decrease in the distance is large. In such a case, the control unit 33 may move the second marker 72 toward the first marker 71. First, the control unit 33 displays the second marker 72 at a position away from the first marker 71. Next, the control unit 33 moves the second marker 72 toward the first marker 71 until the distance between the second marker 72 and the first marker 71 becomes a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position away from the first marker 71, and repeats the above-mentioned process. In this example, since the second marker 72 is moving toward the object, the first marker 71, the subject 60 may understand that the second marker 72 is approaching the first marker 71.

For example, when the distance between the moving body 50 and the pedestrian 63 is decreasing, the control unit 33 may determine that the pedestrian 63 has contributed greatly to the decrease in the distance. In such a case, the control unit 33 may move the second marker 72 away from the first marker 71. First, the control unit 33 displays the second marker 72 at a position close to the first marker 71. Next, the control unit 33 moves the second marker 72 away from the first marker 71 until the distance between the second marker 72 and the first marker 71 becomes a predetermined distance. Next, the control unit 33 erases the second marker 72. Next, the control unit 33 displays the second marker 72 again at a position close to the first marker 71, and repeats the above-mentioned process. In this example, since the second marker 72 has moved away from the object, the first marker 71, the subject 60 may understand that the second marker 72 is moving away toward the first marker 71.

The control unit 33 changes the direction of movement of the second marker 72 relative to the first marker 71 depending on the degree of contribution of the moving body 50 and the pedestrian 63 to the reduction in the distance between the moving body 50 and the pedestrian 63. Depending on the direction of movement of the second marker 72, the subject 60 can identify, for example, whether the moving body 50 is approaching the pedestrian 63 or the pedestrian 63 is approaching the moving body 50.

The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 on the second image as shown in FIG. 8, for example. The control unit 33 may superimpose the first marker 71 and the second marker 72 having a shape similar to the contour line 69 of the pedestrian 63 on the second image as shown in FIG. 9, for example. The control unit 33 may repeatedly enlarge or reduce the second marker 72 around the first marker 71 as the center. The control unit 33 switches between enlarging and reducing the second marker 72 depending on the contribution of the moving body 50 and the pedestrian 63 to the reduction in the distance between the moving body 50 and the pedestrian 63.

When the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 becomes less than a predetermined threshold, the control unit 33 may superimpose a new marker on the second image. Hereinafter, the new marker is also referred to as a fourth marker. The fourth marker may include any image. For example, the fourth marker may include an image showing an exclamation mark "!". According to this configuration, for example, when the pedestrian 63 displaying the first marker 71 and the second marker 72 and the moving body 50 approach each other to a certain degree or more, the fourth marker is superimposed and displayed on the second image. The subject 60 can identify, for example, that the pedestrian 63 is located in the vicinity of the moving body 50 by the fourth marker. In another embodiment, the control unit 33 may change the display mode of the first marker 71 and the second marker 72 when the distance between the detection target displaying the first marker 71 and the second marker 72 and the moving body 50 becomes less than a predetermined threshold. The control unit 33 may, for example, change the color of the first marker 71 and the second marker 72. With this configuration, the subject 60 can also identify that, for example, a pedestrian 63 is located near the moving object 50 by the change in the color of the markers.

The control unit 33 may detect two detection targets arranged in front and behind in the depth direction. The control unit 33 may display a first marker 71 and a second marker 72 on each of the two detection targets located in front and behind. The control unit 33 may attach different first markers 71 and second markers 72 to the two detection targets. For example, the control unit 33 may attach a first marker 71 and a second marker 72 to the first detection target located in the back that are less noticeable than the first marker 71 and the second marker 72 attached to the second detection target located in the front. For example, the control unit 33 may change the first marker 71 and the second marker 72 attached to the first detection target located in the back to be darker in color, have a higher transmittance, and have thinner lines than the first marker 71 and the second marker 72 attached to the second detection target located in the front.

(Example 2-4 (reference example))
10 shows a second example of a detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. In the example shown in FIG. 10, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63a that is shown inside the display area 62 in the first image, and pedestrians 63b and 63c that are shown outside the display area 62 but inside the detection area 61. The processing of the control unit 33 regarding the pedestrian 63a is similar to the processing regarding the pedestrian 63 shown in FIG. 4, for example.

When the detection position on the first image where the detection target is detected is outside the display area 62 and inside the detection area 61, the control unit 33 may superimpose a marker corresponding to the detection target on the second image and display it on the display device 40. Hereinafter, the marker is also referred to as a fifth marker. When it is determined that there is a possibility that the moving body 50 and the detection target may come into contact, the control unit 33 may display the fifth marker on the display device 40. When the detection position on the first image where the detection target is detected is to the right of the display area 62, the control unit 33 may superimpose the fifth marker on the right edge of the second image and display it on the display device 40. When the detection position on the first image where the detection target is detected is to the left of the display area 62, the control unit 33 may superimpose the fifth marker on the left edge of the second image and display it on the display device 40.

In the second example, the detection position where pedestrian 63b is detected is to the right of display area 62. The control unit 33 may determine that there is a possibility of contact between moving body 50 and pedestrian 63b. In such a case, the control unit 33 may cause the display device 40 to display a fifth marker corresponding to pedestrian 63b superimposed on the right edge of the second image. Details of the fifth marker corresponding to pedestrian 63b will be described later. The detection position where pedestrian 63c is detected is to the left of display area 62. The control unit 33 may determine that there is no possibility of contact between moving body 50 and pedestrian 63c. In such a case, the control unit 33 may not cause the fifth marker corresponding to pedestrian 63b to be displayed.

Figure 11 shows an example of a second image corresponding to the display area 62 of the first image shown in Figure 10. As shown in Figure 11, a pedestrian 63a is shown in the second image. Pedestrians 63b and 63c are not shown in the second image.

11, the control unit 33 may, for example, superimpose an obstacle image 74 on the second image and display it on the display device 40. The obstacle image 57 shows the detection result of an obstacle detection device using an ultrasonic sonar or the like provided on the moving body 50. The obstacle image 74 may include an image 74a, an image 74b, and an image 74c. The image 74a is an image of the moving body 50 viewed from above. The image 74b is an image showing that an obstacle has been detected to the left rear of the moving body 50. The image 74c is an image showing that an obstacle has been detected to the right rear of the moving body 50. The detection result of the obstacle by the obstacle detection device and the detection result of the detection target by the control unit 33 do not necessarily have to match. For example, in the example shown in FIG. 11, the obstacle image 74 shows that an obstacle has been detected to the right rear and left rear of the moving body 50. On the other hand, the control unit 33 may determine that there is no possibility of contact between the moving body 50 and the pedestrian 63c present to the left rear of the moving body 50. In such a case, the control unit 33 does not need to display the fifth marker corresponding to the pedestrian 63c.

An example of the fifth marker 73 corresponding to a pedestrian 63b is shown in FIG. 11. The fifth marker 73 may include an icon image 73a and a band image 73b. The icon image 73a may be an image corresponding to a "person" type of detection target. The subject 60 who views the icon image 73a can recognize that a person is present to the right of the second image. The band image 73b is, for example, a band-shaped image extending in the vertical direction on the second image. The band image 73b may be filled with a color or pattern corresponding to a "person" type of detection target. The control unit 33 may move the band image 73b within the right end region 73c on the second image. The control unit 33 may change the moving speed and width of the band image 73b.

The fifth marker 73 will be described in detail. The control unit 33 may determine the width of the band image 73b according to the distance between the moving object 50 and the pedestrian 63b. For example, the control unit 33 may increase the width of the band image 73b as the distance becomes closer. The subject 60 who views the band image 73b can recognize the distance between the moving object 50 and the pedestrian 63b based on the width of the band image 73b.

The control unit 33 may determine that, for example, the contribution of the moving body 50 is large with respect to the reduction in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the band image 73b in a first direction within the right end region 73c on the second image. The first direction may be, for example, a direction from the outside to the inside in the left-right direction on the second image. The control unit 33 may determine that, for example, the contribution of the pedestrian 63b is large with respect to the reduction in the distance between the moving body 50 and the pedestrian 63b. In such a case, the control unit 33 repeatedly moves the band image 73b in a second direction within the right end region 73c on the second image. The second direction may be, for example, a direction from the inside to the outside in the left-right direction on the second image. The subject 60 who views the band image 73b can recognize whether the moving body 50 is approaching the pedestrian 63b or the pedestrian 63b is approaching the moving body 50 based on the movement direction of the band image 73b.

The control unit 33 may determine the movement speed of the band image 73b according to the rate at which the distance between the moving body 50 and the pedestrian 63b is decreasing. For example, the faster the rate at which the distance is decreasing, the faster the movement speed of the band image 73b may be. The subject 60 who views the band image 73b can recognize the rate at which the distance between the moving body 50 and the pedestrian 63b is decreasing based on the movement speed of the band image 73b.

When the control unit 33 detects a user input corresponding to a predetermined user operation while the fifth marker 73 is displayed, the control unit 33 may change the display area 62 so that the detected position of the pedestrian 63b on the first image is included inside the display area 62. For example, as shown in FIG. 12, the control unit 33 may lengthen the display area 62 on the first image in the left-right direction and move the display area 62 to the right side of the detection area 61. With this configuration, the pedestrian 63b is displayed on the second image, as shown in FIG. 13, for example.

The above-mentioned specific user operation may include any user operation. For example, the above-mentioned specific user operation may include a first user operation that changes the steering angle of the moving body 50. The fifth marker 73 may function as a GUI (Graphic User Interface) that accepts a second user operation. Hereinafter, the GUI is also referred to as an interface image. In such a case, the above-mentioned specific user operation may include the second user operation.

The control unit 33 may automatically change the display area 62 so that the detected position of the pedestrian 63b in the first image is included inside the display area 62. In this case, the control unit 33 may maintain the automatic change of the display area 62 until the pedestrian 63b is no longer detected in the detection area 61 of the first image.

The control unit 33 may change the icon image 67 in response to changes in the display area 62, for example as shown in FIG. 13.

The control unit 33 may change the display area 62 on the first image in response to, for example, a pinch-in operation and a pinch-out operation on the display device 40. For example, as shown in FIG. 14, the control unit 33 may cause the display area 62 to approximately coincide with the detection area 61. In such a case, as shown in FIG. 15, for example, all detection targets within the detection area 61 are displayed on the display device 40.

(5th example (reference example))
Fig. 16 shows a fifth example of a detection area 61 in a first image acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 16, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a vehicle 64a that appears within a first predicted path 65 of the moving body 50, and a vehicle 64b and a pedestrian 63d that appear outside the first predicted path 65.

In the fifth example, a case will be described where the external area of the moving body 50 is dark, for example at night or inside a tunnel. When the external area of the moving body 50 is dark, the characteristic values of the first image and the second image may decrease. The characteristic values may include any parameter related to the visibility of the images. For example, the characteristic values may include at least one of the brightness value and the contrast ratio of the images. When the characteristic value of the second image decreases, the visibility of the second image may decrease.

The control unit 33 may execute specific image processing on a region corresponding to the detection target on the second image. The specific image processing may include a first process of superimposing a marker corresponding to the detection target on the region. Hereinafter, the marker is also referred to as a sixth marker. The sixth marker may include, for example, an image that substantially matches the contour shape of the detection target on the second image. According to this configuration, the sixth marker is displayed superimposed on the detection target on the second image. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target on the second image. The specific image processing may include a second process of changing the characteristic value of the region corresponding to the detection target on the second image. For example, the control unit 33 may change the characteristic value of the region so as to improve the visibility of the region on the second image. According to this configuration, the visibility of the detection target on the second image is improved. Therefore, even if the characteristic value of the second image is low, the subject 60 can easily recognize the detection target on the second image.

The control unit 33 may execute the above-mentioned specific image processing when one or more conditions are satisfied. The one or more conditions may include a condition that the detection target is located within the first predicted path 65 of the moving body 50. The one or more conditions may include a condition that the first predicted path 65 of the moving body 50 overlaps with the second predicted path of the detection target. The one or more conditions may include a condition that the distance between the moving body 50 and the detection target is less than a predetermined threshold. The one or more conditions may include a condition that a characteristic value of at least a portion of an area of the second image is less than a predetermined threshold.

In the fifth example, the control unit 33 may determine that one or more of the above-mentioned conditions are satisfied for each of the vehicles 64a, 64b, and the pedestrian 63d. In such a case, the control unit 33 may cause the display device 40 to display three sixth markers 75a, 75b, and 75c corresponding to the vehicles 64a, 64b, and the pedestrian 63d, respectively, superimposed on the second image, as shown in FIG. 17, for example. The control unit 33 may cause the sixth markers to be displayed superimposed on the detection target in a bright place.

The control unit 33 may change the shape of the guide line 66. The control unit 33 may change the shape of the guide line 66 in the area where the guide line 66 overlaps with the detection target. FIG. 18 shows one example of the shape of the guide line 66. The guide line 66 in FIG. 18 is not displayed in the area where the guide line 66 overlaps with the sixth marker 75a. The example of the shape of the guide line 66 is not limited to being erased, and other design changes are allowed. Design changes include changing the color, changing the transparency, changing the type to a dashed line, changing the line thickness, and blinking. The control unit 33 may change the shape of the guide line 66 when the sixth marker is not displayed. The control unit 33 may change the shape of the guide line 66 when the first marker 71 and the second marker 72 are displayed on the detection target.

As described above, according to the display system 10 of one embodiment, various markers corresponding to the detection target detected on the first image are superimposed on the second image and displayed on the display device 40. A subject 60 who views the markers can recognize at a glance the relative positional relationship between the moving body 50 and the detection target. This improves the convenience of the technology that displays an image of the external area of the moving body 50.

Below, with reference to Figures 19 to 31, examples of various images that the image processing device 30 synthesizes with the guide wall image 80 and other images in the display area on the first image and displays on the display device 40 will be specifically described.

(Example 6)
FIG. 19 shows a sixth example of the detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed so as to capture an image of the rear of the moving body 50, which is a vehicle. In the example shown in FIG. 19, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The detection area 61 is not limited to a shape longer in the left-right direction than in the up-down direction, and various shapes such as a square, a vertically elongated rectangle, and a circle are possible. The display area 62 can have various positions, sizes, and shapes. For example, the display area 62 is not limited to being located at the center of the detection area 61, and may be shifted to the left or right. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be the display area 62. The control unit 33 may change the display area 62 on the first image in response to a pinch-in operation, a pinch-out operation, or the like on the display device 40.

The control unit 33 generates a second image by synthesizing a guide wall image 80 showing the first predicted path 65 of the moving body 50 in a display area on the first image acquired from the imaging device 20. The control unit 33 may output an image in which the guide wall image 80 is superimposed on the second image and display it on the display device 40. The guide wall image 80 can show the predicted path when the moving body 50 moves backward. The guide wall image 80 is a marker that gives the subject 60 a three-dimensional impression. The guide wall image 80 can be considered to be an image of a virtual guide wall that guides the predicted path and is arranged in the real space shown in the first image, projected onto the first image captured by the imaging device 20. The guide wall image 80 can be viewed as consisting of a plurality of virtual semi-transparent walls extending from the road surface to a predetermined height in the height direction within the field of view of the subject 60. The guide wall image 80 can also be said to be a three-dimensional display of the guide line displayed on the display device 40 when the vehicle moves backward, using a wall-like display. Wall-like displays include displays in other forms such as a surface, a membrane, a board, etc. The three-dimensional display of the guide wall image 80 makes it easier for the subject 60 to grasp the location of the parking space on the road surface, the distance to obstacles, etc.

The control unit 33 may recognize a detection target within the display area 62 of the first image. The control unit 33 may detect a detection target that is inside the display area 62 on the first image and is reflected within the first predicted path 65. When a detection target is detected, the control unit 33 may synthesize a virtual plane as a first recognition wall image 83 on the moving body 50 side of the recognized detection target. When the detection target moves, the first recognition wall image 83 moves together with the detection target.

20 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 19. The guide wall image 80 is disposed above and away from the lower end of the second image. The guide wall image 80 may include two side wall images 81a and 81b extending in the height direction and the depth direction. The side wall image 81a is located on the left side in the horizontal direction, and the side wall image 81b is located on the right side in the horizontal direction. The area between the two side wall images 81a and 81b shows the first predicted course 65 along which the moving body 50, which is a vehicle, will pass. The interval between the side wall images 81a and 81b may be set to show the vehicle width of the moving body 50. In FIG. 20, the side wall images 81a and 81b are displayed linearly. The control unit 33 may obtain moving body information from the communication unit 31 and curve the side wall images 81a and 81b according to the predicted course. In this case, the moving body information includes the steering angle of the steering wheel.

The guide wall image 80 may include multiple distance wall images 82a, 82b, 82c that extend in the height direction and horizontal direction and indicate the distance in the depth direction from the moving body 50. The distance wall images 82a, 82b, 82c may connect between the side wall images 81a and 81b. Three distance wall images 82a, 82b, and 82c may be shown in FIG. 20. The number of distance wall images is not limited to three, and may be two or four or more. The distance in the depth direction between the distance wall images 82a and 82b may be narrower than the distance in the depth direction between the distance wall images 82b and 82c. The height direction length of the distance wall images 82a, 82b, 82c may be longer or shorter than the height direction length of the side wall images 81a and 81b.

A frame may be displayed around each of the side wall images 81a, 81b and the distance wall images 82a, 82b, and 82c of the guide wall image 80. The guide wall image 80 may include at least one of the auxiliary lines 84a extending in the depth direction, the auxiliary lines 84b extending in the horizontal direction, and the auxiliary lines 84c extending in the height direction. The guide wall image 80 may include all of the auxiliary lines 84a extending in the depth direction, the auxiliary lines 84b extending in the horizontal direction, and the auxiliary lines 84c extending in the height direction. The auxiliary lines 84a extending in the depth direction may be displayed on the wall surfaces of the side wall images 81a and 81b. The auxiliary lines 84b extending in the horizontal direction may be displayed on the wall surfaces of the distance wall images 82a, 82b, and 82c. The auxiliary lines 84c extending in the height direction can be displayed on the walls of the sidewall images 81a, 81b and the distance wall images 82a, 82b, and 82c. Any number of auxiliary lines 84a, 84b, and 84c can be displayed, one or more. When there are multiple auxiliary lines 84a, 84b, and 84c, the intervals between the auxiliary lines 84a, 84b, and 84c can be equal. The intervals between the auxiliary lines 84a, 84b, and 84c may be different depending on conditions such as the height from the road surface and the distance from the moving body 50. The frames displayed on the periphery of each of the sidewall images 81a, 81b and the distance wall images 82a, 82b, and 82c can be considered as part of the multiple auxiliary lines 84a, 84b, and 84c. In FIG. 20 and the following figures, even when there are multiple auxiliary lines 84a, 84b, and 84c, only one of each is appropriately labeled.

When the distance wall image 82a is projected onto the real space shown by the first video image, it is a first distance wall image that is closer to the moving body 50 than the other distance wall images 82b and 82c. The distance wall image 82a can be different in color from the other distance wall images 82b and 82c. For example, the distance wall images 82b and 82c can be displayed in semi-transparent white, and the distance wall image 82a can be displayed in semi-transparent red. The side wall images 81a and 81b can be displayed in semi-transparent white. The above color settings are examples, and the color of the side wall images 81a and 81b may be different from that of the distance wall images 82b and 82c.

The transmittance of the sidewall images 81a, 81b may be changed depending on the position in the depth direction. The transmittance of the sidewall images 81a, 81b may be set to be lower toward the front in the depth direction and higher toward the back. The saturation or brightness of the sidewall images 81a, 81b may be made different depending on the position in the depth direction. For example, the saturation can be made higher toward the front in the depth direction. The saturation of an image can also be said to be the color intensity of the image. Continuous changes in the transmittance, saturation, brightness, chromaticity, etc. can be said to have a gradation.

The transmittance of the distance wall images 82a, 82b, and 82c may be changed according to the height from the road surface. The transmittance of the distance wall images 82a, 82b, and 82c may be set to be higher as the distance from the road surface increases in the height direction. The saturation or brightness of the distance wall images 82a, 82b, and 82c may be different according to the height from the road surface. For example, the saturation can be lower as the distance from the road surface increases in the height direction.

The auxiliary lines 84b, 84c on the distance wall image 82a, which is the first distance wall image, may be the same color as the auxiliary lines 84b, 84c on the other distance wall images 82b, 82c. The auxiliary lines 84a, 84b, 84c may be any color. For example, if the color of the walls of the guide wall image 80 other than the distance wall image 82a is white, the color of the auxiliary lines 84a, 84b, 84c may be white with a higher brightness or lower transmittance than these walls.

The control unit 33 may detect a vehicle 64 shown in the display area 62 of the first image as a detection target. In the example of FIG. 20, the vehicle 64 is located within the first predicted path 65. The detection target detected by the control unit 33 is not limited to a vehicle, and may be a building, a fence, an obstacle on the road, a person, an animal, etc. When a detection target is detected, the control unit 33 may synthesize a first recognition wall image 83 on the side of the recognized moving object 50 of the detection target. When a detection target is detected, the control unit 33 may change the length of the guide wall image 80 in the height direction. When a detection target is detected, the control unit 33 can shorten the length of the guide wall image 80 in the height direction. This allows the control unit 33 to make the display easier to see even when the first recognition wall image 83 is displayed, and to direct the subject 60's attention to the detection target.

The first recognition wall image 83 of the detection target located within the first predicted path 65 extends in the height direction and the horizontal direction. The first recognition wall image 83 may be displayed as an opaque or semi-transparent surface of any color. In FIG. 20, the first recognition wall image 83 is displayed as a semi-transparent surface having a rectangular shape on the moving body 50 side of the vehicle 64. The horizontal width of the first recognition wall image 83 may correspond to the width of the vehicle 64. The horizontal width of the first recognition wall image 83 may be approximately equal to the width of the vehicle 64. The height of the first recognition wall image 83 may be set to a predetermined value. The transmittance of the first recognition wall image 83 may be changed according to the height direction from the road surface. The transmittance of the first recognition wall image 83 may be set so that it becomes higher as it moves away from the road surface in the height direction. In addition, the saturation or brightness of the first recognition wall image 83 may be different according to the distance in the height direction from the road surface. For example, the saturation can be lowered as it moves away from the road surface in the height direction. The first recognition wall image 83 may include auxiliary lines extending horizontally and vertically.

The control unit 33 may move the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction within the frame of the first recognition wall image 83. The auxiliary lines may be displayed so as to appear from the lower end of the frame of the first recognition wall image 83 and disappear at the upper end. The control unit 33 may display both or either of the relative position and the relative distance change between the moving body 50 and the vehicle 64 by moving the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction. For example, when the distance between the moving body 50 and the vehicle 64 is narrowing, the control unit 33 may move the auxiliary lines extending in the horizontal direction faster in the height direction. The closer the distance between the moving body 50 and the vehicle 64 is, the faster the control unit 33 may move the auxiliary lines in the height direction. The control unit 33 may move the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in the height direction regardless of the position of the detection target and the distance from the detection target.

The color of the first recognition wall image 83 can be different depending on the distance in the depth direction from the moving object 50 to the vehicle 64. For example, as shown in FIG. 20, when the distance between the moving object 50 and the vehicle 64 is greater than a predetermined distance, the control unit 33 can display the color of the first recognition wall image 83 in blue. When the moving object 50 moves backwards and the distance to the vehicle 64 becomes shorter, the control unit 33 can change the color of the first recognition wall image 83 from blue to yellow, or from yellow to red, for example.

When the distance between the moving body 50 and the vehicle 64 is equal to or greater than a predetermined distance, the control unit 33 may not display the first recognition wall image 83. When the moving body 50 and the vehicle 64 approach each other relatively and are within a predetermined distance, the control unit 33, for example, highlights the vehicle 64 in the second image and then displays the first recognition wall image 83. Highlighting includes displaying the outline of the vehicle 64 in a thick line, blinking the outline of the vehicle 64, filling in the image of the vehicle 64, and the like. The control unit 33 may change the display mode according to a change in the relative position with the vehicle 64. For example, when the vehicle 64 is moving relatively away from the moving body 50, the first recognition wall image 83 may not be displayed.

Figure 21 is a diagram showing an example of the second image in a state in which the distance between the moving body 50 and the vehicle 64 has narrowed as a result of the moving body 50 moving backwards. The first recognition wall image 83 is displayed in red. This makes it possible to warn the subject 60 that there is a risk of colliding with the vehicle 64 if the subject 60 moves backwards any further.

When the detection object and the guide wall image 80 overlap, the control unit 33 can change the display of the guide wall image 80. For example, as shown in FIG. 21, the control unit 33 can at least partially not display the guide wall image 80 that overlaps with the vehicle 64 in the second image. More specifically, the control unit 33 does not display the guide wall image 80 that overlaps with the vehicle 64 and is located farther from the moving body 50 in the depth direction than the vehicle 64 or the first recognition wall image 83 on the display device 40. In this way, the display device 40 can display an image that is easy for the subject 60 to see and intuitively understand. The method of changing the display of the guide wall image 80 is not limited to not displaying the overlapping portion. For example, the control unit 33 can increase the transmittance of the guide wall image 80 in the portion that overlaps with the detection object.

(Modification)
The guide wall image 80 is not limited to the example shown in Figures 20 and 21. Figures 22 to 24 show modified examples of the guide wall image 80.

(First Modification)
In the example shown in FIG. 22, the horizontally extending auxiliary lines 84b of the distance wall images 82a, 82b, and 82c may display only one of the lowest positions in the height direction. The distance wall image 82a displaying only the lowest position auxiliary line 84b may facilitate recognition of the ground level. The distance wall image 82a displaying only the lowest position auxiliary line 84b may facilitate understanding of the distance from the recognition wall image. An auxiliary line 84c extending in the height direction may be displayed at both ends of the horizontal direction of the distance wall images 82a, 82b, and 82c. An auxiliary line 84b located at the lower end and an auxiliary line 84c extending in the height direction located at both ends of the horizontal direction may be displayed on the periphery of the distance wall images 82a, 82b, and 82c. The distance wall images 82a, 82b, and 82c may be displayed by an auxiliary line 84b extending in the horizontal direction at the lower end of the periphery and an auxiliary line 84c extending in the height direction at both left and right ends.

The distance wall images 82a, 82b, and 82c may be displayed including a surface having a translucent color. The distance wall images 82a, 82b, and 82c may have a higher transmittance or a lighter color density as they move away from the road surface in the height direction. The auxiliary lines 84b and 84c located on the outer periphery of the distance wall image 82a closest to the moving body 50 may be displayed in a color different from the other auxiliary lines. The different color may be, for example, red. The other auxiliary lines may be white, but is not limited to this. The first recognition wall image 83 displayed in front of the vehicle 64 may be displayed as a translucent surface of any color, and may be displayed so that the transmittance increases or the color density decreases as it moves away from the road surface in the height direction. The first recognition wall image 83 may be displayed so that the upper boundary is not visible and gradually disappears in the height direction.

In the example shown in FIG. 22, only two auxiliary lines 84a extending in the depth direction of the sidewall images 81a and 81b of the guide wall image 80 are displayed at the upper and lower ends of the sidewall images 81a and 81b. The lower auxiliary line 84a1 may be displayed more emphasized than the upper auxiliary line 84a2. Highlighting includes displaying using a thicker line, a line with a lower transmittance, a line with a higher brightness, etc. The upper auxiliary line 84a2 and the lower auxiliary line 84a1 may be displayed in the same manner. Lines in the same manner are lines that are similar in thickness, transmittance, brightness, etc., for example.

The auxiliary line 84a extending in the depth direction may be displayed using a display method in which it gradually fades and disappears at the furthest position in the depth direction. The side wall images 81a and 81b may be displayed so that the ends at the furthest position in the depth direction are visible.

Auxiliary lines 84c extending in the height direction may be displayed on the side wall images 81a and 81b at least at the ends closest to the viewer and at positions where the images intersect with the distance wall images 82a, 82b, and 82c. These auxiliary lines 84c extending in the height direction may be displayed more emphasized than the auxiliary lines 84c extending in the other height directions. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c displayed at the positions where the images intersect with the distance wall image 82a may be displayed thicker than the auxiliary lines 84c displayed at the positions where the images intersect with the distance wall images 82b and 82c.

(Second Modification)
The guide wall image 80 shown in FIG. 23 differs from the guide wall image 80 in FIG. 22 in the side wall images 81a and 81b. In FIG. 23, two or more auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 may be displayed, including the top and bottom ones. The auxiliary lines 84a extending in the depth direction may be highlighted in the lowermost auxiliary line in the height direction more than the other auxiliary lines. The multiple auxiliary lines 84a may all be displayed as lines of the same type. The guide wall image 80 in FIG. 23 may adopt the changes in transmittance, color changes, color schemes, and the like that may be adopted in the guide wall image 80 in FIG. 22.

(Third Modification)
The guide wall image 80 shown in FIG. 24 differs from the guide wall image 80 in FIG. 22 in the side wall images 81a and 81b. In FIG. 24, the auxiliary lines 84a extending in the depth direction of the side wall images 81a and 81b of the guide wall image 80 are displayed only at the lower end. The auxiliary lines 84a extending in the depth direction at the lower end of the guide wall image 80, together with the auxiliary lines 84b extending in the horizontal direction, may correspond to lines on the road surface when mapped onto the real space shown by the first video. The guide wall image 80 in FIG. 24 may adopt the changes in transmittance, color changes, color schemes, and the like that may be adopted in the guide wall image 80 in FIG. 22. The auxiliary lines 84c extending in the height direction may each have a different thickness. The auxiliary lines 84c arranged from the front side to the back side in the depth direction may be thinner as they are positioned further back.

In the guide wall images 80 shown in each of Figures 20 to 24, the display device 40 does not need to display some or all of the auxiliary lines 84a, 84b, and 84c.

(Example 7)
Fig. 25 shows a seventh example of the detection area 61 in the first video acquired by the image processing device 30 from the imaging device 20. In the example shown in Fig. 25, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located in the center of the detection area 61 in the left-right direction. The display area 62 is not limited to a part of the detection area 61, and the entire detection area 61 may be the display area 62.

In the seventh example, the moving body 50 is a vehicle that may be moving backwards toward a parking space in a parking lot. The imaging device 20 may be located behind the moving body 50 and capture an image of the rear. The control unit 33 may detect, from the detection area 61, a vehicle 64b and a pedestrian 63 that are outside the first predicted path 65 and at least partially within the display area 62, as detection targets. In the example shown in FIG. 25, the control unit 33 determines that the distance in the depth direction of the vehicle 64a that is within the first predicted path 65 of the moving body 50 is farther than a predetermined distance, and does not detect it.

26 shows an example of the second image corresponding to the display area 62 of the first image shown in FIG. 25. The control unit 33 recognizes the vehicle 64b and pedestrian 63 to be detected in the display area 62 of the first image, and displays the first recognition wall images 83a and 83b, which are virtual planes, on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. The first recognition wall images 83a and 83b may be displayed as semi-transparent surfaces parallel to the side wall images 81a and 81b extending in the height direction and the depth direction. The control unit 33 may obtain a change in the distance of the relative positions between the vehicle 64b and the pedestrian 63 and the moving body 50, and may display the side wall images 81a and 81b only when the distance is changing in a direction that shortens. Any algorithm may be used to estimate the distance between the moving body 50 and the detection target, as described above.

The control unit 33 may estimate the horizontal distance of the vehicle 64a and pedestrian 63 to be detected from the guide wall image 80, and determine the display color of the first recognition wall images 83a and 83b according to the estimated distance. Here, the distance between the detection target and the guide wall image 80 is the distance between the detection target and the virtual guide wall displayed by the guide wall image 80 when the virtual guide wall is present in the real space shown by the first image. The control unit 33 may estimate a first distance between the guide wall image 80 and the detection target from the position where the guide wall image 80 is mapped in the real space shown by the first image. The control unit 33 may set a threshold value for the first distance, and display the first recognition wall images 83a and 83b on the display device 40 using different display colors such as blue, yellow, and red in order from the longest to the shortest horizontal distance. The first recognition wall images 83a and 83b may have auxiliary lines extending in the depth direction. The control unit 33 can move the auxiliary lines extending in the depth direction of the first recognition wall images 83a and 83b in the height direction within the frame of the first recognition wall image 83, similar to the auxiliary lines extending in the horizontal direction of the first recognition wall image 83 in FIG. 20.

The control unit 33 estimates a first position as the position in the depth direction of the vehicle 64b and the pedestrian 63 to be detected. The control unit 33 may at least partially change the display of the sidewall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, in accordance with the first position. The manner of changing the display includes changing the color, brightness, saturation, etc. of the wall surface, changing the color, thickness, etc. of the frame line and/or the auxiliary line, or a combination thereof. For example, in FIG. 26, the display color of a part 81a1 of the sidewall image 81a facing the vehicle 64b and a part 81b1 of the sidewall image 81b facing the pedestrian 63 are changed. The parts of the sidewall images 81a and 81b may be regions divided by auxiliary lines 84c extending in the height direction on the sidewall images 81a and 81b. The display colors of the part 81a1 of the sidewall image 81a and the part 81b1 of the sidewall image 81b may be different depending on the first distance. The control unit 33 detects that the detection target is the side wall images 81a, 8
When the obstacle is larger than the length represented by the side wall image 81a or 81b, the display color of the entire side wall image 81a or 81b may be changed. Examples of large obstacles include large vehicles such as trailers, buildings, and the like.

When the detection target moves, the control unit 33 may estimate a second predicted path of the detection target, and estimate a first position as a position where the second predicted path intersects with the sidewall images 81a, 81b. The control unit 33 may estimate the first position as a position where the second predicted path in the real space shown by the first video intersects with the mapping of the sidewall images 81a, 81b onto the real space. The control unit 33 may at least partially change the display of the sidewall images 81a, 81b in accordance with the first position. In this case, the parts 81a1, 81b of the sidewall images 81a, 81b whose display is changed may be changed.
The display color of the first image 81a or 81b may be different depending on the first distance. When the detection target is a very large obstacle, the control unit 33 may change the display color of the entire sidewall image 81a or 81b.

As another display mode, the control unit 33 may change the display of the portions of the sidewall images 81a, 81b located further in the depth direction than the first recognition wall images 83a, 83b. As yet another display mode, the control unit 33 may change the display of the portions of the sidewall images 81a, 81b located further in the depth direction than the position where the second predicted path intersects with the sidewall images 81a, 81b.

The display of the first recognition wall images 83a and 83b may be changed depending on the height from the road surface. As with the first recognition wall image 83 in the sixth example, the display of the first recognition wall images 83a and 83b may be set to different values of transmittance, saturation, brightness, etc. depending on the height from the road surface.

As described above, the display system 10 can spatially and three-dimensionally display the distance and/or position of a detection object that may be an obstacle to reversing using the guide wall image 80 and the first recognition wall images 83a, 83b. In addition, it can three-dimensionally display positions where there is a risk of collision based on the predicted path of the detection object while moving. In this way, the display system 10 can spatially and three-dimensionally display the detection object and the warning content by displaying the guide wall image 80 and the first recognition wall images 83a, 83b as walls (or planes) with a height direction. This makes it easier for the subject 60 to understand the display and the warning content.

(Example 8)
FIG. 27 shows an eighth example of the detection area 61 in the first image acquired by the image processing device 30 from the imaging device 20. The imaging device 20 may be disposed so as to capture an image of the rear of the moving body 50, which is a vehicle. In the example shown in FIG. 27, the detection area 61 is longer in the left-right direction than in the up-down direction. The display area 62 is located at the center of the detection area 61 in the left-right direction. The control unit 33 may detect, as detection targets, a pedestrian 63 and a vehicle 64b that are reflected in a second area that is outside the display area 62 on the first image and is an area inside the detection area 61. In the eighth example, the control unit 33 determines that the distance in the depth direction of the vehicle 64a reflected in the display area 62 is farther than a predetermined distance, and does not detect the vehicle 64a.

FIG. 28 shows a moving object 50 as seen from above along the height direction in the real space shown in the first image.
6A and 6B are schematic diagrams illustrating the arrangement of a pedestrian 63 and a vehicle 64b to be detected. The imaging device 20 can capture an image of a subject within a range included in areas F1 and F2 in real space.
Regions F1 and F2 correspond to detection region 61 of the first image. Region F1 is a first region that corresponds to display region 62 of the first image. Region F2 corresponds to a second region that is inside detection region 61 and outside display region 62 of the first image.

For the purpose of explanation, FIG. 28 shows a virtual guide wall 90, which is a mapping of the guide wall image 80 displayed in the second image onto real space, by a dashed line. Virtual side walls 91a and 91b correspond to the side wall images 81a and 81b displayed in the second image. Virtual distance walls 92a, 92b, and 92c correspond to the distance wall images 82a, 82b, and 82c displayed in the second image.

28, for the sake of explanation, imaginary lines 95l1 to 95l5 and 95r1 to 95r5 are shown by dashed lines, which indicate the horizontal distance of the detection target from the imaginary side walls 91a and 91b of the moving body 50. These horizontal distances are also called second distances.
The virtual lines 95l1 to 95l5 are located on the left side of the moving object 50. The virtual lines 95r1 to 95r5 are located on the right side of the moving object 50. The intervals between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 can be set arbitrarily. The intervals between the virtual lines 95l1 to 95l5 and the virtual lines 95r1 to 95r5 may be set to equal intervals. The control unit 33 determines which of the virtual lines 95 is the detection target from the first image.
The closest to the imaginary lines 95l1-95l5 and 95r1-95r5, or
It is possible to determine whether the region is located within the region between any two of 5l5 and 95r1 to 95r5.

The second image corresponding to the first image in FIG. 27 is illustrated in FIG. 29. In the eighth example, the guide wall image 80 may adopt a display method similar to that described in the sixth and seventh examples. When the detection target is located in the second region of the first image, the control unit 33 synthesizes the second recognition wall images 89a and 89b at the end of the second image on the side where the detection target is present. The second recognition wall images 89a and 89b may include wall portions 96a and 96b extending in the height direction and the depth direction, respectively, and floor portions 97a and 97b extending in the depth direction and the horizontal direction. In FIG. 29, the wall portions 96a and 96b extend from the lower end to the upper end of the left and right ends of the second image. The wall portions 96a and 96b may extend up and down along only a part of the left and right ends of the second image, rather than the entirety of the left and right ends. Similarly, the floor portions 97a and 97b may extend in the left and right direction along the entirety or a part of the lower end of the second image. The floors 97a and 97b may be displayed between the guide wall image 80 and the bottom edge of the second image.

The control unit 33 may change the display of the second recognition wall images 89a and 89b in accordance with the second distance of the detection target. The change in the display includes a change in color, transmittance, saturation, brightness, size, etc., and a change in a dynamic display method such as blinking or movement. For example, in one of the multiple embodiments,
The change in display may be represented by a change in color. The display color may be red when the danger level is high, yellow when the danger level is medium, and blue when the danger level is low. The pedestrian 63 shown in FIG. 28 is located on the third virtual line 95r3 on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50. In this case, the control unit 33 may determine that the danger level is medium and set the color of the second recognition wall image 89b on the right side to yellow. The vehicle 64b shown in FIG. 28 is located farther away from the fifth virtual line 95l5 on the left side in the horizontal direction from the virtual side wall 91a of the moving body 50. In this case, the control unit 33 may determine that the danger level is low and set the color of the second recognition wall image 89a on the left side to blue. In one of the multiple embodiments, the change in display may include a horizontal movement of the vertical auxiliary lines 96a1 and 96b1 displayed on the walls 96a and 96b. The control unit 33 may change the speed of movement of the walls 96a, 96b depending on the length of the second distance.

The second recognition wall images 89a and 89b may be semi-transparent. The wall portions 96a and 96b of the second recognition wall images 89a and 89b may change their display characteristics in the vertical direction. For example, the wall portions 96a and 96b of the second recognition wall images 89a and 89b may have a lower transmittance toward the lower portion. Also, the second recognition wall images 89a and 89b may have a higher saturation toward the lower portion. The floor portions 97a and 97b of the second recognition wall image 89a may change their display characteristics in the horizontal direction. For example, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower transmittance toward the left and right ends and a higher transmittance toward the center of the second image. Also, the floor portions 97a and 97b of the second recognition wall image 89a may have a lower saturation from the left and right ends toward the center of the second image. The saturation of an image can also be expressed as the color intensity of the image.

The control unit 33 determines the type of the detection target, and displays an icon image 99a,
99b may be displayed on the floors 97a, 97b of the second recognition wall images 89a, 89b. The type of the detection target can be determined by a known method based on an image of the detection target detected from the first video. For example, the control unit 33 can determine the object type by matching the contour shape of the detection target with a model pattern. For example, a pedestrian 63 and a vehicle 64
For icon images 99a and 99b, icons indicating a person and a vehicle, respectively, may be selected. The control unit 33 may estimate a second distance, which is a horizontal distance between virtual side walls 91a and 91b obtained by mapping the side wall images 81a and 81b onto the real space, and the detection target in the real space. The second distance may be determined from the relationship between each of the lines obtained by mapping the virtual lines 95l1 to 95l5 and 95r1 to 95r5 in the real space onto the detection area 61 of the first image, and the position of the image of each detection target. The control unit 33 may change the position at which the icon images 99a and 99b are displayed in accordance with the second distance.

In the display example of the second image shown in FIG. 29, virtual lines 95l1 to 95l5, 95
The first distance discrimination lines 98l1 to 98l5, 98r1 to 98r5 correspond to the first distance discrimination lines 98l1 to 98l5, 98r1 to 98r5,
8r5 are provided. The first distance identification lines 98l1 to 98l5 are located on the left side of the sidewall image 81a, and are numbered in the order of 98l1, 98l2, 98l3, 98l4, and 98l5 from the center to the left side. 98r1 to 98r5 are located on the right side of the sidewall image 81b, and are numbered in the order of 98r1, 98r2, 98r3, 98r4, and 98r5 from the center to the right side. The first distance identification lines 98l1 to 98l5 and 98r1 to 98r5 indicate the positions where the icon images 99a and 99b are displayed, and do not have to be displayed in the second video.

In FIG. 28, the pedestrian 63 is located on the third virtual line 95r3 on the right side in the horizontal direction from the virtual side wall 91b of the moving body 50, so the control unit 33 may place the icon image 99b of the pedestrian 63 on the third first distance identification line 98r3 on the right side in the second image in FIG. 29. As shown in FIG. 29, the icon image 99b may be displayed three-dimensionally as a surface having lengths in the depth direction and height direction on the first distance identification line 98r3 extending in the depth direction in the second image. When viewed stereoscopically, the icon image 99b may be displayed as a surface parallel to the side wall images 81a, 81b and the second recognition wall images 89a, 89b. The display mode of the icon image 99b, such as color, saturation, brightness, etc., may be changed according to the second distance, similar to the second recognition wall image 89b. For example, the icon image 99b may be displayed in yellow.

In FIG. 28, the vehicle 64b is located farther away from the imaginary side wall 91a of the moving body 50 than the fifth imaginary line 9515 on the left side in the horizontal direction from the imaginary side wall 91a. In this case, the control unit 33
29, the icon image 99a of the vehicle 64b may be disposed on the first distance identification line 9815 on the left side.
The icon image 9 may be arranged so as to appear flush with the wall portion 96a of the recognition wall image 89a.
Similar to the icon image 99b, the icon image 99a can be displayed in a different manner depending on the second distance. For example, the icon image 99a can be displayed in blue.

When the relative position of the detection target as viewed from the moving body 50 changes due to the movement of both or either one of the moving body 50 and the detection target, the positions and display manner of the second recognition wall images 89a, 89b and the icon images 99a, 99b change. Furthermore, when the moving body 50 enters the display area 62, the detection target may be displayed in the manner described in the seventh example. An example of the change in display when the vehicle 64b moves from area F2 to area F1 in the real space shown in FIG. 28 will be described.

First, in the real space of Fig. 28, the vehicle 64b is located on the left side of the virtual line 9515. At this time, as described above, the second recognition wall image 89a is displayed in the second video of Fig. 29, and an icon image 99a is displayed on the leftmost side of the floor portion 98a of the second recognition wall image 89a.
At this time, the second recognition wall image 89a and the icon image 99a can be displayed in blue.

In the real space of FIG. 28, as the vehicle 64b approaches the virtual guide wall 90, the second recognition wall image 89a displayed in the second image of FIG. 29 changes its display mode. For example, the color of the second recognition wall image 89a changes from blue to yellow to red in sequence. At the same time, the icon image 99a moves its display position from above the first distance identification line 98l5 to above the first distance identification line 98l1. The color of the icon image 99a may change from blue to yellow to red in sequence. This allows the subject 60 to recognize that the vehicle 64b is approaching outside, which is not displayed in the second image. In addition, by using a stereoscopic display method, the subject 60 can spatially grasp the approach of the vehicle 64b, and the risk of overlooking information can be reduced.

When the vehicle 64b is approaching the moving object 50, the icon image 99a may move toward the center of the screen. The subject 60 may recognize the direction of movement of the vehicle 64b via the movement of the icon image 99a.

Furthermore, when the vehicle 64b enters the area F1 from the area F2 in the real space of FIG.
Since vehicle 64b is within the display target area of the first image, it is included in the second image and is displayed on display device 40. At the same time, the display of second recognition wall image 89a disappears, and vehicle 64b is displayed as shown in FIG.
The first recognition wall image 83a is displayed on the guide wall image 80 side of the vehicle 64b. When the second recognition wall image 89a disappears, the control unit 33 may blink the second recognition wall image 89a multiple times to call the attention of the subject 60. The control unit 33 displays the first recognition wall image 83a on the moving body 50 side of the vehicle 64b. Furthermore, the control unit 33 estimates a second predicted course of the vehicle 64b, and estimates a first position as a position where the second predicted course intersects with the side wall images 81a, 81b. The control unit 33
The control unit 33 may change the display of a portion 81a1 of the sidewall image 81a including the position of the vehicle 64b. For example, the control unit 33 may change the color of the portion 81a1 of the sidewall image 81a to blue, yellow, and red in sequence as the vehicle 64b approaches. This allows the subject 60 to know that the vehicle 64b is approaching and that the second predicted path of the vehicle 64b is in line with the first predicted path 65 of the moving body 50.
It becomes easy to understand that there is an overlap with

As described above, according to one of the multiple embodiments of the present disclosure, the spatial understanding of the position of the detection target is facilitated by the three-dimensional display of the guide wall image, the first and second recognition wall images, the icon image, etc. Furthermore, according to one of the multiple embodiments of the present disclosure, information such as cautions and warnings can be provided in an easily understandable manner by changing the display mode of each wall surface. In addition, the use of three-dimensional display makes it possible to reduce the risk of information being overlooked.

(Modification)
Fig. 30 shows another example of the second image corresponding to the first image in Fig. 27. In Fig. 30, view display areas 101, 102 showing the selection status between "Normal View Display" and "Wide View Display" are added to the lower left. The "Normal View Display" and "Wide View Display" will be described later. In Fig. 30, the display manner of the guide wall image 80, the second recognition wall images 89a, 89b, and the icon images 99a, 99b may be partially different from that in Fig. 29. For example, in Fig. 30, the guide wall image 80 in the same display manner as in Fig. 22 is displayed.

The wall portions 96a, 96b of the second recognition wall images 89a, 89b may have one or more auxiliary lines 96a1, 96b1 extending in the height direction. In Fig. 30, the wall portions 96a, 96b have a plurality of auxiliary lines 96a1, 96b1, but only one of them is indicated by a reference symbol. The wall portions 96a, 96b of the second recognition wall images 89a, 89b may indicate the distance to the detection target by color, and the transmittance may vary from the front to the back in the depth direction.
The transmittance of b may gradually increase in the height direction, or the darkness may gradually decrease, and may disappear at a predetermined position in the height direction.

The control unit 33 controls the transmittance or color density of the walls 96a and 96b, and the auxiliary line 96a1
, 96a2, it is possible to indicate the presence of an object within a predetermined distance range, the distance to the detected object, or a change in the distance to the detected object.
The change in the transmittance or color density of the light can be called a change in gradation. For example,
The control unit 33 can indicate that the detection target is within a predetermined distance by moving the auxiliary lines 96a1, 96a2 of the walls 96a, 96b. The control unit 33 can indicate that the detection target is within a predetermined distance by making the color of the front side of the walls 96a, 96b the darkest and changing the gradation in the depth direction.
The control unit 33 can control the wall portions 96a and 96b not to display the auxiliary lines 96a1 and 96b1.
It is possible to prevent the change in transmittance (gradation) from occurring.

In addition to the first distance identification lines 98l, 98r, one or more second distance identification lines 100l, 100r are provided on the floors 97a, 97b of the second recognition wall images 89a, 89b. In FIG. 30, there are multiple second distance identification lines 100l, 100r, but only one of each is indicated by a reference symbol. The first distance identification lines 98l, 98r and the second distance identification lines 100l, 100r do not have to be displayed on the display. The first distance identification lines 98l, 98r correspond to the horizontal distance from the moving body 50 to the detection target, whereas the second distance identification lines 100l, 100r can correspond to the distance in the depth direction. Each of the multiple second distance identification lines 100l, 100r may correspond to the distance in the depth direction when the auxiliary lines 84c extending in the height direction on the side wall images 81a, 81b are mapped onto the real space.

The subject 60 can grasp the horizontal and depth positions of the detection target according to the horizontal and depth positions of the icon images 99a, 99b on the floors 97a, 97b. As the icon images 99a, 99b, three-dimensional images according to the type of detection target are used. The floors 97a, 97b may be positioned closer to the front end of the sidewall images 81a, 81b and inside the line extending from the sidewall images 81a, 81b. The positional relationship between the second recognition wall images 89a, 89b and the guide wall image 80 is not limited to this and can be changed as appropriate.

In FIG. 30, the display device 40 has view display areas 101 and 102 in the lower right portion, which show the selection status of "Normal View Display" and "Wide View Display". The "Normal View Display" and "Wide View Display" have different angles of view of the images displayed. In one of the embodiments, the "Normal View Display" corresponds to the angle of view of the display area 62 of the second image shown in FIG. 4. In one of the embodiments, the "Wide View Display" corresponds to the angle of view of the detection area 61 of the first image shown in FIG. 4. The "Wide View Display" does not need to be the entire image of the detection area 61 of the first image, but may be an image with an angle of view including the "second image" cut out from the detection area 61. The second images shown in FIG. 30 and those shown before it are both normal view displays. Whether the image being displayed is "Normal View Display" or "Wide View Display" is shown by changing the display of the view display areas 101 and 102 showing the selection status. For example, changing the display of the area showing the selection status includes changing the color of the view display areas 101 and 102, changing the darkness, changing the color of the text, etc.

The control unit 33 can switch between "normal view display" and "wide view display" depending on the result of processing the image of the first video, or upon receiving input from the subject 60 via the display device 40 or the like. For example, the control unit 33 may switch the displayed video to "wide view display" when it detects that a detection target is approaching in the "normal view display" state. The control unit 33 may switch the displayed video to "normal view display" when it detects that a detection target is not present within a predetermined range in the "wide view display" state. If the display device 40 has a touch panel screen, the control unit 33 may switch the display upon receiving a signal from the display device 40 indicating that the subject 60 has touched and selected a portion of the view display area 101 or 102 on the screen.

Figure 31 is a diagram showing an example of an image that has been switched from the "normal view display" shown in Figure 30 to the "wide view display." The angle of view of the "wide view display" image in Figure 31 is wider horizontally than the angle of view of the "normal view display" image in Figure 30. Figure 31 shows a pedestrian 63 and a vehicle 64b that are located outside the display range of the "normal view display" shown in Figure 30. Also, because the horizontal angle of view of the image displayed in Figure 31 is wider than that of the "normal view display" in Figure 30, the distance between the left and right side wall images 81a and 81b of the guide wall image 80 is displayed relatively narrower.

In FIG. 31, the second recognition wall images 89a and 89b disappear because the pedestrian 63 and the vehicle 64b have entered the display range. The control unit 33 recognizes the vehicle 64b and the pedestrian 63 to be detected in the detection area 61 of the first image, and can display the first recognition wall images 83a and 83b, which are virtual planes, on the moving body 50 side of the vehicle 64b and the pedestrian 63, respectively. The control unit 33 estimates the positions of the vehicle 64b and the pedestrian 63 to be detected in the depth direction. The control unit 33 may change the display of the side wall images 81a and 81b facing the vehicle 64b and the pedestrian 63, respectively, based on the estimated positions. In FIG. 31, the display of a part 81b1 of the side wall image 81b and a part 81a1 of the side wall image 81a are changed in correspondence with the pedestrian 63 and the vehicle 64b, respectively. The change in display includes various modes. For example, the change in display includes a change in color, a change in the thickness of the auxiliary lines surrounding the periphery, a change in the line type, a change in the start and stop of blinking, a change in the blinking cycle, etc. The first recognition wall images 83a, 83b and the parts 81a1, 81b1 of the side wall images 81a, 81b may adopt a display method and a display mode similar to the example of FIG.

30 and 31, according to one of the embodiments, the image captured by the imaging device 20 can be displayed on the display device 40 by switching between "normal view display" and "wide view display" with different angles of view. In the "normal view display", the subject 60 can easily grasp the presence and position of a detection target located outside the range of the displayed image by the second recognition wall images 89a, 89b and the icon images 99a, 99b. By automatically or manually switching the display device 40 from the "normal view display" to the "wide view display", the subject 60 can confirm the detection target detected outside the range of the image of the "normal view display" by the image. In the "wide view display", the display of the guide wall image 80 and the first recognition wall image 83b makes it possible to grasp the positional relationship with the detection target in a three-dimensional manner. In this way, the display system 10 in one of the embodiments can make it easy for the subject to grasp the surrounding situation and sense danger.

Although the present invention has been described based on the drawings and embodiments, it should be noted that a person skilled in the art would be able to easily make various modifications and corrections based on this disclosure. Therefore, it should be noted that these modifications and corrections are included in the scope of the present invention. For example, the functions included in each means, step, etc. can be rearranged so as not to cause logical inconsistencies, and multiple means, steps, etc. can be combined into one or divided.

For example, the components and functions of the display system 10 according to the above-described embodiment may be rearranged. For example, some or all of the configuration and functions of the image processing device 30 may be included in at least one of the imaging device 20 and the display device 40.

Some of the components of the display system 10 according to the above-described embodiment may be located outside the mobile body 50. For example, the image processing device 30 and the like may be realized as a communication device such as a mobile phone or an external server, and may be connected to other components of the display system 10 by wire or wirelessly.

10 Display system 20 Imaging device 21 Imaging optical system 22 Imaging element 23 Communication unit 24 Control unit 30 Image processing device 31 Communication unit 32 Storage unit 33 Control unit 40 Display device 50 Moving object 51 Network 60 Target person 61 Detection area 62 Display area 63, 63a, 63b, 63c, 63d Pedestrian 64, 64a, 64b Vehicle 65 First predicted path 66 Guide line 67 Icon image 68 Third marker 69 Contour line 70 Area 71 First marker 72 Second marker 73 Fifth marker 73a Icon image 73b Band image 73c Right end area 74 Obstacle image 75a, 75b, 75c Sixth marker 80 Guide wall image 81a, 81b Side wall image 82a, 82b, 82c Distance wall image 83, 83a, 83b First recognition wall images 84a, 84b, 84c Auxiliary lines 89a, 89b Second recognition wall image 90 Virtual guide walls 91a, 91b Virtual side walls 92a, 92b, 92c Virtual distance walls 95l1 to 95l5, 95r1 to 95r5 Virtual lines 96a, 96b Walls 97a, 97b Floors 98l1 to 98l5, 98r1 to 98r5 First distance identification lines 99a, 99b Icon images 100l, 100r Second distance identification lines 101, 102 View display area

Claims (8)

A display system mounted on a vehicle, comprising:
an imaging device for imaging a rear area of the vehicle;
a display device that displays a second image that is a part of a first image captured by the imaging device of an imaging area;
a control unit that, when an object is detected outside the second image within the imaging area to the right rear or left rear of the vehicle, and when it is determined that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, and when the distance between the vehicle and the object is decreasing and the vehicle is moving at a moving speed less than a reference value, displays a marker in the second image displayed on the display device in a display manner that corresponds to the object and allows the user to recognize that the object is approaching the vehicle,
The control unit is capable of acquiring a moving speed of the vehicle.
Display system. The display system according to claim 1, wherein the control unit changes the width of the marker when displaying it. The display system according to claim 1 or 2, wherein the control unit changes the width of the marker depending on the distance between the object and the vehicle. The display system according to any one of claims 1 to 3, wherein the control unit causes the marker to be displayed at the right edge or the left edge of the second image depending on the position of the object. The control unit extends a display area of the second image displayed on the display device in a left-right direction in response to a user operation.
A display system according to any one of claims 1 to 4. A moving object equipped with a display system according to claims 1 to 5. An image processing device mounted on a vehicle, the image processing device displaying, on a display device, a second image that is a part of a first image captured by an imaging device that captures an image of a rear area of the vehicle, the second image being a part of a first image captured by the imaging device, the image processing device comprising:
a control unit that acquires the first image and extracts the second image from the first image;
The control unit detects an object outside the second image within the imaging area at a right rear or left rear of the vehicle,
when it is determined that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance , and when the distance between the vehicle and the object is decreasing and the vehicle is moving at a moving speed less than a reference value, the control unit displays a marker in the second image displayed on the display device in a display manner that corresponds to the detected object and allows the user to recognize that the object is approaching the vehicle;
The control unit is capable of acquiring a moving speed of the vehicle.
Image processing device. A program for an image processing device mounted on a vehicle, the program causing a display device to display a second image, which is a part of a first image captured by an imaging device that captures an image of a rear area of the vehicle,
an acquisition step in which a control unit mounted in the image processing device acquires the first image captured by the imaging device;
The control unit extracts the second image from the first image;
a detection step of the control unit detecting an object outside the second image within the imaging area at a right rear or left rear of the vehicle;
a display step of displaying a marker in the second image displayed on the display device in a display manner that corresponds to the object and enables the user to recognize that the object is approaching the vehicle, when the control unit has detected the object in the detection step and determined that there is a possibility of contact between the vehicle and the object because the distance between the vehicle and the object is less than a predetermined distance, and when the distance between the vehicle and the object has decreased and the vehicle is moving at a moving speed less than a reference value,
The control unit is capable of acquiring a moving speed of the vehicle.
program.