JP7066271B2 - Image processing device, image processing method and head-up display system - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method and a head-up display system.

Conventionally, there has been known a projection technique in which a head-up display system is mounted on a vehicle and information notified to the driver is visually recognized as a virtual image in front of the vehicle. According to the head-up display, the driver can visually recognize the information without significantly moving his / her line of sight while driving.

Recently, by applying such projection technology, an object called an AR (Augmented Reality) marker is superimposed on a real scene that the driver sees through the front window (for example, another vehicle traveling in front of the own vehicle). Image processing techniques have also been proposed.

Japanese Patent Application Laid-Open No. 2015-221651 JP-A-2015-226304 Japanese Patent Application Laid-Open No. 2015-09677 International Publication No. 2015/128985

However, the relative positional relationship between the other vehicle traveling ahead and the own vehicle varies greatly depending on the traveling environment such as unevenness of the road surface and the presence or absence of banks. It is difficult to make the AR marker follow such fluctuations in real time, and for example, the AR marker may be displaced with respect to another vehicle traveling ahead. Such misalignment gives the driver a sense of discomfort.

The present invention has been made in view of the above problems, and an object of the present invention is to reduce a sense of discomfort given to a driver in a head-up display system.

According to one aspect, the image processing apparatus has the following configuration. That is,
An image processing device that outputs a projected image in which an object located in front of the vehicle is visually recognized as being surrounded by an object by projecting it onto the windshield of the vehicle.
A detection unit that detects the movement in the height direction and the movement in the width direction of the vehicle,
When the detection unit detects the movement in the height direction, the side or surface in the direction substantially orthogonal to the height direction is detected while leaving the side or surface in the height direction of the object included in the projected image. When the detection unit detects an operation in the width direction by hiding or making it difficult to see, it is abbreviated as the width direction while leaving the side or surface in the width direction of the object included in the projected image. It is characterized by having a correction unit that hides sides or faces in orthogonal directions or makes it difficult to see.

In the head-up display system, it is possible to reduce the discomfort given to the driver.

It is a figure which shows an example of the system configuration of the head-up display system which concerns on 1st Embodiment. It is a figure which shows the arrangement example of each apparatus which constitutes the head-up display system which concerns on 1st Embodiment. It is the first figure which shows the projection example of the projection image which includes an AR marker. It is a figure which shows an example of the hardware composition of an image processing apparatus. It is a figure which shows the detail of the functional structure of the AR marker part. FIG. 1 is a first diagram showing a specific example of processing by the AR marker target detection unit and the AR marker generation unit. It is a figure which shows the specific example of the processing by a vehicle motion detection part. FIG. 1 is a first diagram showing a specific example of processing by the AR marker correction unit. It is a flowchart which shows the flow of AR marker processing by an AR marker part. It is the first figure which shows the projection example in the correction time zone of the projection image which includes an AR marker. It is a 2nd figure which shows the projection example in the correction time zone of the projection image including an AR marker. FIG. 3 is a third diagram showing an example of projection in a correction time zone of a projected image including an AR marker. It is a figure which shows the arrangement example of each apparatus which constitutes the head-up display system which concerns on 2nd Embodiment. It is the 2nd figure which shows the projection example of the projection image which includes an AR marker. FIG. 2 is a second diagram showing a specific example of processing by the AR marker target detection unit and the AR marker generation unit. FIG. 2 is a second diagram showing a specific example of processing by the AR marker correction unit. FIG. 4 is a fourth diagram showing an example of projection in a correction time zone of a projected image including an AR marker.

Hereinafter, each embodiment will be described with reference to the attached drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate explanations.

[First Embodiment]
<System configuration of head-up display system>
First, the system configuration of the head-up display system mounted on the vehicle will be described. FIG. 1 is a diagram showing an example of a system configuration of a head-up display system according to the first embodiment. As shown in FIG. 1, the head-up display system 100 includes an image pickup device 110, a vehicle motion detection sensor 120, an image processing device 130, and an optical member 150. In the present embodiment, the head-up display system 100 is started when the ignition switch of the vehicle is turned on, and is stopped when the ignition switch of the vehicle is turned off. However, the timing of starting and stopping the head-up display system 100 is not limited to this, and for example, after the ignition switch of the vehicle is turned on, the head-up display system 100 is started and stopped based on the instruction of the driver of the vehicle. It may be configured.

The image pickup apparatus 110 photographs the front at the center position in the width direction of the vehicle, and transmits the captured image to the image processing apparatus 130. The vehicle motion detection sensor 120 detects the motion of the vehicle in a predetermined direction as a displacement signal indicating the displacement from the reference position, and transmits the motion to the image processing device 130. In this embodiment, a 6-axis gyro sensor is used as the vehicle motion detection sensor 120.

An AR marker program is installed in the image processing device 130, and when the program is executed, the image processing device 130 functions as an AR marker unit 140. The AR marker unit 140 detects an object (AR marker object) on which the AR marker (object) is superimposed from the captured image. Further, the AR marker unit 140 generates and outputs a projected image (projected image including the AR marker) projected on the windshield of the vehicle according to the position and size of the detected AR marker object.

Further, the AR marker unit 140 monitors the displacement signal transmitted from the vehicle motion detection sensor 120, and when the motion in the predetermined direction of the vehicle is detected, a part of the AR marker included in the projected image is moved in the motion direction. Correct accordingly. As a result, according to the AR marker unit 140, even if the AR marker visually recognized by the driver of the vehicle is displaced with respect to the AR marker object due to the movement of the vehicle in a predetermined direction, a sense of discomfort is given to the driver. Can be reduced.

The optical member 150 projects the projected image output from the image processing device 130 onto the windshield of the vehicle. The optical member 150 is configured so that the projected image projected on the windshield of the vehicle can be visually recognized by the driver as a virtual image in front of the vehicle. As a result, the AR marker object in the actual scene is visually recognized by the driver as if it is surrounded by the AR marker.

<Arrangement example of each device constituting the head-up display system>
Next, an arrangement example of each device constituting the head-up display system 100 will be described. FIG. 2 is a diagram showing an arrangement example of each device constituting the head-up display system according to the first embodiment.

As shown in FIG. 2, the image pickup apparatus 110 is arranged in the vicinity of the windshield 160 of the vehicle, on the ceiling portion at the center position in the width direction of the vehicle, toward the front of the vehicle. As a result, the image pickup apparatus 110 can photograph the front at the center position in the width direction of the vehicle. Further, the vehicle motion detection sensor 120 and the image processing device 130 are arranged behind the center console of the vehicle or under the floor.

As shown in FIG. 2, the projected image output from the image processing device 130 is drawn on the screen 151 constituting the optical member 150 and projected onto the windshield 160 via the reflection mirrors 152 and 153. As a result, the driver can visually recognize the AR marker included in the projected image at the position of the virtual image shown in FIG. Therefore, by controlling the position and size of the AR marker included in the projected image according to the position and size of the AR marker object, the driver can make the AR marker object surrounded by the AR marker. Can be visually recognized. As shown in FIG. 2, in the present embodiment, the height direction of the vehicle is the y-axis direction, and the front-rear direction of the vehicle is the z-axis direction.

<Projection example of projected image>
Next, an example of projection of a projected image will be described. FIG. 3 is a first diagram showing a projection example of a projected image including an AR marker. As shown in FIG. 3, the driver can visually recognize the actual view in front of the own vehicle through the windshield 160 of the vehicle. The actual scene visually recognized by the driver includes another vehicle 300 traveling in front of the own vehicle. In this embodiment, the other vehicle 300 is detected as an AR marker object.

Therefore, the projected image includes an AR marker configured to surround the other vehicle 300. Specifically, as shown in FIG. 3, when the projected image is projected on the windshield 160, the AR marker 310 visually recognized by the driver surrounds the other vehicle 300, so that the AR marker in the projected image is displayed. The position coordinates, the length in the width direction, and the length in the height direction are calculated. As shown in FIG. 3, in the present embodiment, the width direction of the vehicle is set to the x-axis.

<Hardware configuration of image processing device>
Next, the hardware configuration of the image processing apparatus 130 will be described. FIG. 4 is a diagram showing an example of the hardware configuration of the image processing device. As shown in FIG. 4, the image processing device 130 includes a CPU (Central Processing Unit) 401, a ROM (Read Only Memory) 402, and a RAM (Random Access Memory) 403. Further, the image processing device 130 includes a GPU (Graphic Processing Unit) 404 and an I / F (Interface) device 405. Each part of the image processing device 130 is connected to each other via a bus 406.

The CPU 401 is a device that executes various programs (for example, an AR marker program) installed in the ROM 402. The ROM 402 is a non-volatile memory and functions as a main storage device for storing various programs executed by the CPU 401. The RAM 403 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). The RAM 403 functions as a main storage device that provides a work area that is expanded when various programs installed in the ROM 402 are executed by the CPU 401.

The GPU 404 is a dedicated integrated circuit that performs image processing, and in the first embodiment, a projected image is generated based on an instruction from the CPU 401 that executes the AR marker program. The I / F device 405 is a connection device for connecting the image pickup device 110, the vehicle motion detection sensor 120, and the image processing device 130.

<Functional configuration of the AR marker section of the image processing device>
Next, the details of the functional configuration of the AR marker unit 140 of the image processing apparatus 130 will be described. FIG. 5 is a diagram showing details of the functional configuration of the AR marker unit. As shown in FIG. 5, the AR marker unit 140 includes an AR marker target detection unit 501, an AR marker generation unit 502, a vehicle motion detection unit 503, an AR marker correction unit 504, and an AR marker output unit 505.

The AR marker target detection unit 501 acquires a captured image from the image pickup device 110 and detects an AR marker target included in the captured image. Further, the AR marker target detection unit 501 notifies the AR marker generation unit 502 of the position coordinates, the width direction length, and the height direction length of the detected AR marker target object.

The AR marker generation unit 502 calculates the position and size of the AR marker in the projected image based on the position coordinates, the length in the width direction, and the length in the height direction notified by the AR marker target detection unit 501. Generate a projected image containing an AR marker. Further, the AR marker generation unit 502 notifies the AR marker correction unit 504 of the generated projection image.

The vehicle motion detection unit 503 is an example of the detection unit, monitors the displacement signal acquired from the vehicle motion detection sensor, and detects the motion of the vehicle in a predetermined direction. Further, when the vehicle motion detection unit 503 detects the motion of the vehicle in a predetermined direction, the vehicle motion detection unit 503 notifies the AR marker correction unit 504 of the detection result.

The AR marker correction unit 504 is an example of a correction unit. When the AR marker correction unit 504 receives the detection result indicating that the motion in a predetermined direction is detected from the vehicle motion detection unit 503, the AR marker correction unit 504 is at least a part of the AR marker included in the projection image generated by the AR marker generation unit 502. Is corrected according to the operating direction. Further, the AR marker correction unit 504 notifies the AR marker output unit 505 of the projected image including the corrected AR marker whose at least a part has been corrected.

If the AR marker correction unit 504 does not receive the detection result indicating that the motion in the predetermined direction is detected from the vehicle motion detection unit 503, the AR marker correction unit 504 includes the AR included in the projection image generated by the AR marker generation unit 502. Notify the AR marker output unit 505 without correcting the marker.

The AR marker output unit 505 outputs the projected image notified by the AR marker correction unit 504. As a result, a projected image including the AR marker is projected on the windshield 160 of the vehicle.

<Specific example of processing of each part of AR marker part>
Next, specific examples of processing of each part of the AR marker unit 140 (excluding the AR marker output unit 505, the AR marker target detection unit 501, the AR marker generation unit 502, the vehicle motion detection unit 503, and the AR marker correction unit 504) will be described. do.

(1) Specific Example of Processing by AR Marker Target Detection Unit and AR Marker Generation Unit First, a specific example of processing by the AR marker target detection unit 501 and the AR marker generation unit 502 will be described. FIG. 6 is a first diagram showing a specific example of processing by the AR marker target detection unit and the AR marker generation unit.

Of these, FIG. 6A shows a photographed image 600 acquired by the AR marker target detection unit 501 from the image pickup apparatus 110. As shown in FIG. 6A, the captured image 600 includes another vehicle 300 traveling in front of the own vehicle.

FIG. 6B shows a state in which another vehicle 300 is detected as an AR marker target object included in the captured image 600 by the AR marker target detection unit 501. In the example of FIG. 6B, the AR marker target detection unit 501 has the position coordinates (x _p , y _p ), the width direction length (w _p ), and the height direction length in the captured image 600 of the other vehicle 300. It shows how the ( _hp ) was calculated.

FIG. 6C shows that the AR marker generation unit 502 uses the AR marker 611 in the projected image 610 based on the position coordinates, the width direction length, and the height direction length of the other vehicle 300 in the captured image 600.
-Position coordinates (x _d , y _d ),
・ Length in the width direction ( _wd )
・ Length in the height direction ( _hd )
Is shown.

(2) Specific Example of Processing by Vehicle Motion Detection Unit Next, a specific example of processing by the vehicle motion detection unit 503 will be described. FIG. 7 is a diagram showing a specific example of processing by the vehicle motion detection unit. Of these, FIG. 7A is a diagram showing the displacement 710, the displacement signal 711, and the displacement signal processing result 712 from the reference position of the movement of the vehicle in the y-axis direction by the vehicle motion detection unit 503. be.

As shown in FIG. 7A, the vehicle motion detection unit 503 detects the displacement 710 as the motion in the y-axis direction of the vehicle. In the displacement signal 711, the horizontal axis represents time and the vertical axis represents the displacement 710 in the y-axis direction. Further, the displacement signal processing result 712 is a processing result obtained by converting the displacement signal 711 into an amplitude value of pp (peak to peak) every unit time.

The vehicle motion detection unit 503 notifies the AR marker correction unit 504 of the detection result that the motion in the y-axis direction of the vehicle is detected when the displacement signal processing result 712 exceeds the threshold value 713. In the example of FIG. 7A, the AR marker correction unit 504 determines that the vehicle motion detection unit 503 has detected the movement of the vehicle in the y-axis direction in the time zone (correction time zone) indicated by the arrow 714. It shows how it was notified to. In the processing result 712 shown in FIG. 7A, for example, after the vehicle travels on a paved road, it temporarily travels on an unpaved gravel road or the like, and then travels on the paved road again. It occurs when you do.

FIG. 7B is a diagram showing the displacement 720 from the reference position of the movement of the vehicle in the x-axis direction by the vehicle motion detection unit 503 and the displacement signal 722. As shown in FIG. 7B, the vehicle motion detection unit 503 detects the displacement 720 as the motion in the x-axis direction of the vehicle. In the displacement signal 722, the horizontal axis represents time and the vertical axis represents displacement 720 in the x-axis direction.

When the displacement signal 722 exceeds the threshold value 723, the vehicle motion detection unit 503 notifies the AR marker correction unit 504 of the detection result that the motion in the x-axis direction of the vehicle is detected. In the example of FIG. 7B, the AR marker correction unit 504 determines that the vehicle motion detection unit 503 has detected the movement of the vehicle in the x-axis direction in the time zone (correction time zone) indicated by the arrow 724. It shows how it was notified to. The displacement signal 722 shown in FIG. 7B, for example, indicates that the vehicle travels on a straight road, temporarily travels on a sharp curve to the right, and then travels on the straight road again. Occurs in some cases.

FIG. 7C is a diagram showing the displacement 730 from the reference position of the motion around the z-axis of the vehicle and the displacement signal 732 by the vehicle motion detection unit 503. As shown in FIG. 7 (c), the vehicle motion detection unit 503 detects the displacement 730 as the motion around the z-axis of the vehicle. In the displacement signal 732, the horizontal axis represents time and the vertical axis represents displacement 730 around the z-axis.

The vehicle motion detection unit 503 notifies the AR marker correction unit 504 of the detection result that the motion around the z-axis of the vehicle is detected when the displacement signal 732 exceeds the threshold value 733. In the example of FIG. 7C, the AR marker correction unit 504 detects the detection result that the vehicle motion detection unit 503 has detected the movement of the vehicle around the z-axis in the time zone (correction time zone) indicated by the arrow 734. It shows how it was notified to. The displacement signal 732 shown in FIG. 7 (c) is, for example, a vehicle traveling on a flat road, temporarily traveling on a left curve with a bank, and then traveling on the flat road again. Occurs in some cases.

(3) Specific Example of Processing by AR Marker Correction Unit Next, a specific example of processing by the AR marker correction unit 504 will be described. FIG. 8 is a first diagram showing a specific example of processing by the AR marker correction unit. In FIG. 8, "before correction" indicates the AR marker included in the projected image notified by the AR marker generation unit 502. As shown in FIG. 8, the AR marker included in the projected image notified by the AR marker generation unit 502 has a rectangular shape.

On the other hand, in FIG. 8, "after correction" indicates the AR marker after being corrected by the AR marker correction unit 504. Of these, "y-axis direction" indicates the corrected AR marker when the vehicle motion detection unit 503 notifies the detection result that the motion in the y-axis direction of the vehicle is detected. As shown in FIG. 8, when the detection result indicating that the movement of the vehicle in the y-axis direction is detected is notified, the AR marker correction unit 504 substantially orthogonal to the y-axis direction among the four sides included in the AR marker. Make corrections such as hiding the side to be used or making it difficult to see. As a result, the side of the AR marker visually recognized by projecting the projected image, which is substantially orthogonal to the y-axis direction, is changed. The correction that makes it difficult to see is, for example, a correction such as lowering the brightness or changing the color.

Further, "x-axis direction" indicates a corrected AR marker when the vehicle motion detection unit 503 notifies the detection result that the motion in the x-axis direction of the vehicle is detected. As shown in FIG. 8, when the detection result that the movement of the vehicle in the x-axis direction is detected is notified, the AR marker correction unit 504 substantially orthogonal to the x-axis direction among the four sides included in the AR marker. Make corrections such as hiding the side to be used or making it difficult to see. As a result, the side of the AR marker visually recognized by projecting the projected image, which is substantially orthogonal to the x-axis direction, is changed.

Further, "around the z-axis" indicates a corrected AR marker when the vehicle motion detection unit 503 notifies the detection result that the motion around the z-axis of the vehicle is detected. As shown in FIG. 8, when the detection result that the movement around the z-axis of the vehicle is detected is notified, the AR marker correction unit 504 makes the four corners of the AR marker curved or makes the AR marker elliptical. Make corrections such as making it into a shape. As a result, the side of the AR marker that is visually recognized by projecting the projected image is changed to a curved line.

As described above, when the AR marker correction unit 504 detects the movement of the vehicle in a predetermined direction, at least a part of the AR markers included in the projection image generated by the AR marker generation unit 502 is replaced with the vehicle motion detection unit 503. Correct according to the more notified operation direction.

<Flow of AR marker processing by the AR marker section>
Next, the flow of AR marker processing by the AR marker unit 140 will be described. FIG. 9 is a flowchart showing the flow of AR marker processing by the AR marker unit. When the head-up display system 100 is activated, any one of the flowcharts shown in FIG. 9 is executed.

Of these, FIG. 9A is a flowchart showing a flow of AR marker processing for correcting an AR marker based on an operation in the y-axis direction of the vehicle. In step S901, the AR marker target detection unit 501 acquires a captured image from the image pickup device 110. In step S902, the vehicle motion detection unit 503 acquires a displacement signal in the y-axis direction from the vehicle motion detection sensor 120.

In step S903, the AR marker target detection unit 501 determines whether or not the AR marker target has been detected from the acquired captured image. If it is determined in step S903 that the AR marker object has not been detected (No in step S903), the process returns to step S901. On the other hand, if it is determined in step S903 that the AR marker object has been detected (Yes in step S903), the process proceeds to step S904.

In step S904, the AR marker generation unit 502 determines the position coordinates and width of the AR marker in the projected image based on the position coordinates, the width direction length, and the height direction length of the AR marker object detected in step S903. Calculate the directional length and the height directional length. Further, the AR marker generation unit 502 generates a projection image including the AR marker based on the calculation result.

In step S905, the vehicle motion detection unit 503 calculates the pp amplitude value based on the acquired displacement signal in the y-axis direction, and whether or not the calculated pp amplitude value exceeds a predetermined threshold value 713. Is determined. If it is determined in step S905 that the threshold value 713 is not exceeded (No in step S905), the process proceeds to step S907.

On the other hand, if it is determined in step S905 that the threshold value 713 has been exceeded (yes in step S905), the process proceeds to step S906. In step S906, the AR marker correction unit 504 corrects two sides substantially orthogonal to the y-axis direction among the four sides constituting the AR marker in the projection image generated in step S904.

In step S907, the AR marker output unit 505 outputs either the projection image generated by the AR marker generation unit 502 or the projection image corrected by the AR marker correction unit 504.

In step S908, the AR marker target detection unit 501 determines whether or not to end the AR marker processing. If it is determined in step S908 that the AR marker processing is to be continued (if No in step S908), the process returns to step S901. On the other hand, if it is determined in step S908 that the AR marker processing is to be completed (yes in step S908), the AR marker processing is terminated.

FIG. 9B is a flowchart showing a flow of AR marker processing for correcting an AR marker based on an operation in the x-axis direction of the vehicle. Since steps S911, S913, S914, S917, and S918 in FIG. 9B correspond to steps S901, S903, S904, S907, and S908 in FIG. 9A, respectively, here, step S912, S915 and S916 will be described.

In step S912, the vehicle motion detection unit 503 acquires a displacement signal in the x-axis direction from the vehicle motion detection sensor 120. In step S915, the vehicle motion detection unit 503 determines whether or not the acquired displacement signal in the x-axis direction exceeds a predetermined threshold value 723. If it is determined in step S915 that the threshold value does not exceed 723 (if No in step S915), the process proceeds to step S917.

On the other hand, if it is determined in step S915 that the threshold value 723 has been exceeded (yes in step S915), the process proceeds to step S916. In step S916, the AR marker correction unit 504 corrects two sides that are substantially orthogonal to the x-axis direction among the four sides constituting the AR marker in the projection image generated in step S914.

FIG. 9C is a flowchart showing the flow of AR marker processing for correcting AR markers based on the movement of the vehicle around the z-axis. Since steps S921, S923, S924, S927, and S928 in FIG. 9C correspond to steps S901, S903, S904, S907, and S908 in FIG. 9A, respectively, here, step S922, S925 and S926 will be described.

In step S922, the vehicle motion detection unit 503 acquires a displacement signal around the z-axis from the vehicle motion detection sensor 120. In step S925, the vehicle motion detection unit 503 determines whether or not the acquired displacement signal around the z-axis exceeds a predetermined threshold value 733. If it is determined in step S925 that the threshold value does not exceed 733 (if No in step S925), the process proceeds to step S927.

On the other hand, if it is determined in step S925 that the threshold value has been exceeded (yes in step S925), the process proceeds to step S926. In step S926, the AR marker correction unit 504 corrects the four corners of the AR marker to be curved or the AR marker to be elliptical in the projected image generated in step S924.

In the above description, it is assumed that one of the flowcharts shown in FIG. 9 is executed by activating the head-up display system 100. However, the flowchart to be executed is not limited to any one, and any two flowcharts or three flowcharts may be executed in parallel. However, in this case, it is assumed that any one of the steps is executed for the steps that overlap between the flowcharts. Further, regarding steps S906, S916, and S926 (correction of the AR marker), for example, it is assumed that any one of the processes is executed according to the priority of steps S906> S916> S926.

<Projection example in correction time zone>
Next, an example of projection in the correction time zone will be described. FIG. 10A is a first diagram showing an example of projection in a correction time zone of a projected image including an AR marker. As shown in FIG. 10A, when the vehicle travels on an unpaved gravel road or the like, the AR marker correction unit 504 is notified of the detection result that the movement in the y-axis direction is detected. Therefore, the AR marker included in the projected image is corrected, and the projected image including the corrected AR marker is projected on the front window of the vehicle. As a result, the AR marker 1000a shown in FIG. 10A is visually recognized.

In the case of the AR marker 1000a, even if the relative positional relationship in the y-axis direction fluctuates between the other vehicle 300 traveling in front and the own vehicle and a positional shift in the y-axis direction occurs, it is omitted in the y-axis direction. The orthogonal sides have been changed. Therefore, it is possible to reduce the sense of discomfort caused by the positional deviation in the y-axis direction, which is given to the driver.

FIG. 10B is a second diagram showing an example of projection in a correction time zone of a projected image including an AR marker. As shown in FIG. 10B, when the vehicle travels on a sharp curve to the right, the AR marker correction unit 504 is notified of the detection result that the movement in the x-axis direction is detected. Therefore, the AR marker included in the projected image is corrected, and the projected image including the corrected AR marker is projected on the front window of the vehicle. As a result, the AR marker 1000b shown in FIG. 10B is visually recognized.

In the case of the AR marker 1000b, even if the relative positional relationship in the x-axis direction fluctuates between the other vehicle 300 traveling in front and the own vehicle and a positional shift in the x-axis direction occurs, it is abbreviated in the x-axis direction. The orthogonal sides have been changed. Therefore, it is possible to reduce the discomfort caused by the positional deviation in the x-axis direction given to the driver.

FIG. 10C is a third diagram showing an example of projection in a correction time zone of a projected image including an AR marker. As shown in FIG. 10C, when the vehicle travels on a left curve with a bank, the AR marker correction unit 504 is notified of the detection result that the movement around the z-axis is detected. Therefore, the AR marker included in the projected image is corrected, and the projected image including the corrected AR marker is projected on the front window of the vehicle. As a result, the AR marker 1000c shown in FIG. 10C is visually recognized.

In the case of the AR marker 1000c, the shape becomes elliptical even if the relative positional relationship around the z-axis fluctuates between the other vehicle 300 traveling in front and the own vehicle, causing a positional shift around the z-axis. has been changed. Therefore, it is possible to reduce the discomfort caused by the positional deviation around the z-axis given to the driver.

<Summary>
As is clear from the above description, in the image processing apparatus 130 according to the first embodiment, the AR marker object located in front of the vehicle is surrounded by the AR marker by projecting onto the windshield of the vehicle. When outputting a projected image that is visually recognized as
・ Monitor the displacement signal of the vehicle in the specified direction and monitor it.
-When the movement of the vehicle in a predetermined direction is detected, at least a part of the AR marker included in the projected image is corrected according to the movement direction.

Thereby, according to the first embodiment, even if the AR marker is displaced with respect to the object of the AR marker, the discomfort given to the driver can be reduced.

[Second Embodiment]
In the first embodiment described above, a case where the driver is made to visually recognize the two-dimensional AR marker has been described. On the other hand, in the second embodiment, a case where the driver is made to visually recognize the three-dimensional AR marker will be described. Hereinafter, the differences from the first embodiment will be mainly described.

<Arrangement example of each device constituting the head-up display system>
First, an arrangement example of each device constituting a head-up display system that allows the driver to visually recognize a three-dimensional AR marker will be described. FIG. 11 is a diagram showing an arrangement example of each device constituting the head-up display system according to the second embodiment.

The difference from FIG. 2 is that the screen 1110 constituting the optical member 1100 is movable. As the screen 1110 moves in this way, the optical path length up to the windshield 160 fluctuates. This makes it possible to move the position where the virtual image is visually recognized in the z-axis direction. As a result, according to the head-up display system according to the second embodiment, the AR marker corresponding to each position of the virtual image is included in the projected image while the position where the virtual image is visually recognized is moved back and forth once in the z-axis direction. Then, the driver can visually recognize the three-dimensional AR marker.

<Projection example of projected image>
Next, an example of projection of a projected image will be described. FIG. 12 is a second diagram showing a projection example of a projected image including an AR marker. As shown in FIG. 12, by including the AR marker at each position of the virtual image in the projected image with respect to the other vehicle 300, the driver can visually recognize the three-dimensional AR marker 1210.

<Specific example of processing by the AR marker target detection unit and the AR marker generation unit>
Next, a specific example of the processing by the AR marker target detection unit 501 and the AR marker generation unit 502 in the second embodiment will be described. FIG. 13 is a second diagram showing a specific example of processing by the AR marker target detection unit and the AR marker generation unit.

Of these, FIG. 13A shows a captured image 1300 acquired by the AR marker target detection unit 501 from the image pickup apparatus 110. As shown in FIG. 13A, the captured image 1300 includes another vehicle 300 traveling in front of the own vehicle.

FIG. 13B shows a state in which another vehicle 300 is detected as an AR marker target object included in the captured image 1300 by the AR marker target detection unit 501. In the example of FIG. 13B, the AR marker target detection unit 501 calculates the position coordinates (x _pf , y _pf ) of the tip of the other vehicle 300 to the position coordinates (x _pr , y _pr ) of the rear end. Shows. Further, the AR marker target detection unit 501 has a width direction length (w _pf ) of the tip of the other vehicle 300 to a width direction length (w _pr ) of the rear end, and a height direction length (h) of the tip of the other vehicle 300. It shows how the length (h _pr ) in the height direction from _pf ) to the rear end is calculated.

FIG. 13C shows a plurality of AR marker generation units 502 corresponding to each position of the virtual image based on the position coordinates of the front and rear ends of the other vehicle 300 in the captured image 1300, the length in the width direction, and the length in the height direction. Of the AR marker in each projected image of
-It shows how the position coordinates, the length in the width direction, and the length in the height direction are calculated. The example of FIG. 13C shows how the position coordinates, the length in the width direction, and the length in the height direction of the AR markers 1330 to 1335 in each of the six projected images 1320 to 1325 are calculated.

For example, the AR marker generation unit 502 calculates (x _df , y _df ) as the position coordinates of the AR marker 1330 in the projected image 1320, and calculates (w _df ) as the width direction length of the AR marker 1330. (H _df ) is calculated as the length in the height direction. Similarly, the AR marker generation unit 502 calculates (x _dr , y _dr ) as the position coordinates of the AR marker 1335 in the projected image 1325, and calculates (w _dr ) as the width direction length of the AR marker 1330. , Calculate (h _dr ) as the length in the height direction.

<Specific example of processing by the AR marker correction unit>
Next, a specific example of the processing by the AR marker correction unit 504 in the second embodiment will be described. FIG. 14 is a second diagram showing a specific example of processing by the AR marker correction unit. In FIG. 14, “before correction” indicates a case where the AR marker included in the projected image is visually recognized by projecting the projected image notified by the AR marker generation unit 502. As shown in FIG. 14, the AR marker visually recognized by projecting the projected image notified by the AR marker generation unit 502 has a rectangular parallelepiped shape.

On the other hand, in FIG. 14, "after correction" indicates a case where the AR marker corrected by the AR marker correction unit 504 is visually recognized. Of these, "y-axis direction" indicates the case where the corrected AR marker is visually recognized when the vehicle motion detection unit 503 notifies the detection result that the motion in the y-axis direction of the vehicle is detected. There is.

As shown in FIG. 14, when the detection result indicating that the movement of the vehicle in the y-axis direction is notified is notified, the AR marker correction unit 504 substantially orthogonal to the y-axis direction among the four sides included in the AR marker. Make corrections such as hiding the surface to be used or making it difficult to see. As a result, the plane of the AR marker visually recognized by projecting the projected image, which is substantially orthogonal to the y-axis direction, is changed.

Further, "x-axis direction" indicates a corrected AR marker when the vehicle motion detection unit 503 notifies the detection result that the motion in the x-axis direction of the vehicle is detected. As shown in FIG. 14, when the detection result indicating that the movement of the vehicle in the x-axis direction is detected is notified, the AR marker correction unit 504 substantially orthogonal to the x-axis direction among the four sides included in the AR marker. Make corrections such as hiding the surface to be used or making it difficult to see. As a result, the plane of the AR marker visually recognized by projecting the projected image, which is substantially orthogonal to the x-axis direction, is changed.

Further, "around the z-axis" indicates a case where the vehicle motion detection unit 503 notifies the detection result that the motion around the z-axis of the vehicle is detected. As shown in FIG. 14, when the detection result that the movement around the z-axis of the vehicle is detected is notified, the AR marker correction unit 504 makes corrections such as making the AR marker circular or elliptical. As a result, the AR marker visually recognized by projecting the projected image is changed to a cylindrical shape (a curved surface on the side surface and a circular or elliptical cross section).

As described above, when the AR marker correction unit 504 detects the movement of the vehicle in a predetermined direction, at least a part of the AR markers included in the projection image generated by the AR marker generation unit 502 is replaced with the vehicle motion detection unit 503. Correct according to the more notified operation direction.

<Projection example in correction time zone>
Next, an example of projection in the correction time zone will be described. FIG. 15 is a fourth diagram showing an example of projection in a correction time zone of a projected image including an AR marker. As shown in FIG. 15, when the vehicle travels on an unpaved gravel road or the like, the AR marker correction unit 504 is notified of the detection result that the movement in the y-axis direction is detected. Therefore, the AR marker included in the projected image is corrected, and the projected image including the corrected AR marker is projected on the front window of the vehicle. As a result, the AR marker 1500 shown in FIG. 15 will be visually recognized.

In the case of the AR marker 1500, even if the relative positional relationship in the y-axis direction fluctuates between the other vehicle 300 traveling in front and the own vehicle and a positional shift in the y-axis direction occurs, it is abbreviated in the y-axis direction. The orthogonal planes have been changed. Therefore, it is possible to reduce the sense of discomfort caused by the positional deviation in the y-axis direction, which is given to the driver.

<Summary>
As is clear from the above description, in the image processing apparatus 130 according to the second embodiment, by projecting onto the front window of the vehicle, the object located in front of the vehicle is surrounded by the three-dimensional AR marker. When outputting a projected image that is visually recognized as
・ Monitor the displacement signal of the vehicle in the specified direction and monitor it.
-When the movement of the vehicle in a predetermined direction is detected, at least a part of the AR marker included in the projected image is corrected according to the movement direction.

Thereby, according to the second embodiment, even if the position of the AR marker is displaced with respect to the object of the AR marker, the discomfort given to the driver can be reduced.

[Other embodiments]
In the first and second embodiments, the case where the AR marker target detection unit 501 detects another vehicle 300 traveling in front of the own vehicle as the AR marker target is described, but the AR marker target detection unit 501 has been described. The AR marker object detected by is not limited to other vehicles. For example, a pedestrian walking ahead may be detected. Further, the object detected by the AR marker object detection unit 501 is not limited to an animal body, and may be a stationary object.

Further, in the first and second embodiments, it has been described that the motion in a predetermined direction is detected based on the displacement signal from the vehicle motion detection sensor. However, the method for detecting the movement in a predetermined direction is not limited to this. For example, the operation in a predetermined direction may be detected based on the signal indicating the steering angle of the steering device.

Further, in the second embodiment, the optical path length up to the windshield 160 is varied by moving the screen 1110, but the configuration for varying the optical path length is not limited to this. For example, the optical path length up to the windshield 160 may be changed by moving the reflection mirror.

The present invention is not limited to the configurations shown here, such as combinations with other elements in the configurations and the like described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form thereof.

100: Head-up display system 110: Image pickup device 120: Vehicle motion detection sensor 130: Image processing device 140: AR marker unit 150: Optical member 501: AR marker target detection unit 502: AR marker generation unit 503: Vehicle motion detection unit 504 : AR marker correction unit 505: AR marker output unit 610: Projected image 611: AR marker 1000: AR marker 1100: Optical member 1320 to 1325: Projected image 1330 to 1335: AR marker 1500: AR marker

Claims (3)

An image processing device that outputs a projected image in which an object located in front of the vehicle is visually recognized as being surrounded by an object by projecting it onto the windshield of the vehicle.
A detection unit that detects the movement in the height direction and the movement in the width direction of the vehicle,
When the detection unit detects the movement in the height direction, the side or surface in the direction substantially orthogonal to the height direction is detected while leaving the side or surface in the height direction of the object included in the projected image. When the detection unit detects an operation in the width direction by hiding or making it difficult to see, it is abbreviated as the width direction while leaving the side or surface in the width direction of the object included in the projected image. An image processing device characterized by having a correction unit that hides sides or surfaces in orthogonal directions or makes them difficult to see. It is an image processing method that outputs a projected image in which an object located in front of the vehicle is visually recognized as being surrounded by an object by projecting it on the windshield of the vehicle.
A detection step for detecting the movement in the height direction and the movement in the width direction of the vehicle,
When the movement in the height direction is detected in the detection step, the side or surface in the direction substantially orthogonal to the height direction while leaving the side or surface in the height direction of the object included in the projected image. Is hidden or made difficult to see, and when the movement in the width direction is detected in the detection step, the width direction is left while the side or surface in the width direction of the object included in the projected image is left. An image processing method characterized by having a correction step of hiding or making it difficult to see a side or a surface in a direction substantially orthogonal to the above . An image pickup device that shoots the front of the vehicle and outputs the shot image,
An image processing device that generates and outputs a projected image based on the captured image, in which an object located in front of the vehicle is visually recognized as being surrounded by an object by projecting it onto the windshield of the vehicle. ,
A head-up display system comprising an optical member that projects the output projected image onto the windshield of the vehicle.
The image processing device is
A detection unit that detects the movement in the height direction and the movement in the width direction of the vehicle,
When the detection unit detects the movement in the height direction, the side or surface in the direction substantially orthogonal to the height direction is detected while leaving the side or surface in the height direction of the object included in the projected image. When the detection unit detects an operation in the width direction by hiding or making it difficult to see, it is abbreviated as the width direction while leaving the side or surface in the width direction of the object included in the projected image. A head-up display system characterized by having a compensator that hides or obscures sides or faces in orthogonal directions .

Images