JP7655202B2 - Image display device - Google Patents

Description

This specification discloses an image display device that displays a target image superimposed on the field of view of a vehicle occupant.

Conventionally, there is known a technology that superimposes a specific image on the user's field of vision, thereby allowing the user to perceive the virtual object represented by the image as if it actually exists. For example, Patent Document 1 discloses a technology in which a driver wears smart glasses, which are eyeglass-type displays, and an image of a preceding vehicle leading the vehicle in which the driver is riding is displayed on the smart glasses. In Patent Document 1, the preceding vehicle represented by the image moves so as to lead the vehicle in which the driver is riding to the destination. Therefore, the driver can move to the destination by performing driving operations to follow the preceding vehicle.

JP 2017-129406 A

In order to make a virtual object appear as if it actually exists, the display position of the image displayed on the display (hereafter referred to as the "target image") must be determined based on the position in real space of the virtual object represented by the target image, and the position of the display in real space.

Patent Document 1 shows that in order to identify the position of the display in real space, the shape of the front window is detected based on an image obtained by a camera, and these are used as markers. The camera then captures an image of the scenery including the markers, and the position of the display in real space is identified based on the obtained image.

There is a demand for markers to be easy to recognize in images captured by a camera. However, there are some images that are easy to recognize but are not suitable as markers.

The image display device disclosed in this specification comprises a display that is worn on the head of a user who is a vehicle occupant and displays a target image superimposed on the user's field of vision, a slam camera that is fixed to the display and obtains a slam image showing the area around the display, an extraction means that performs image recognition processing on the obtained slam image and extracts a characteristic shape as a characteristic part, a determination means that uses stored information about the shape of the steering wheel to determine whether the extracted characteristic part belongs to the steering wheel, an exclusion means that excludes the characteristic part if the extracted characteristic part belongs to the steering wheel, a registration means that registers the characteristic part that does not belong to the steering wheel as an AR marker, and a display position determination means that determines the display position of the target image using the registered AR marker.

The object on the rear side of the steering wheel that can be seen through the steering wheel is the target for extracting the feature part.

The technology disclosed in this specification makes it possible to more appropriately determine the display position of the target image.

FIG. 1 is a block diagram showing a configuration of an image display device. FIG. 1 is a diagram showing a state in which a user is wearing a wearable device. FIG. 2 is a diagram showing a schematic view of a driver who is a user. FIG. 13 is an image diagram for explaining a space-fixed display and a device-fixed display. FIG. 2 is a diagram illustrating a user's field of view when a target image is displayed. 13 is a flowchart showing a process of capturing the shape of an interior part as an AR marker in the device controller. FIG. 13 is a diagram for explaining area specification when excluding a steering wheel. 10 is a flowchart showing a process for displaying a target image while driving.

The configuration of the image display device will be described below with reference to the drawings. In the description, specific aspects are shown to facilitate understanding, but these are merely examples and can be modified as appropriate. Figure 1 is a block diagram showing the configuration of an image display device 10. In this embodiment, the image display device 10 is configured by a wearable device 12.

The wearable device 12 is a device that is worn on the head of a vehicle occupant (e.g., the driver), and is, for example, a glasses- or goggle-type device. The wearable device 12 has a display 14, a slam camera 16, a pupil position sensor 18, and a device controller 20 to function as an image display device 10.

This wearable device 12 will be described with reference to FIG. 2. FIG. 2 is a diagram showing a state in which the wearable device 12 is worn by a user 100 who is a vehicle occupant. The wearable device 12 is a device formed in the shape of glasses, and is called smart glasses or AR glasses. The wearable device 12 has temples 26, which are straight frames for hanging on the ears, and rims 24, which are frames formed in a shape that surrounds the eyes and can be hung on the nose.

The display 14 displays an image in the field of view of the user 100 wearing the wearable device 12. In this example, the display 14 is an organic EL display or liquid crystal display having a display area 22 arranged inside the rim 24, and displays an image in part or all of this display area 22. The display area 22 has high transparency. Therefore, when no image is displayed in the display area 22, the user 100 (i.e., the passenger) can see the scenery ahead through the display area 22. Also, when an image is displayed only in a part of the display area 22, the user 100 can see the scenery in the field of view ahead and the displayed image at the same time. In this case, the image may be opaque or translucent. In the following description, the image displayed on the display 14 is called a "target image" to distinguish it from other images. Also, the object virtually represented by the target image is called a "target object."

The slum camera 16 is a camera fixed to the display 14, and captures the surroundings of the display 14. The slum camera 16 is fixed facing forward, for example, near the front end of the temple 26, and captures an area similar to the field of view of the user 100. Hereinafter, the image captured by this slum camera 16 will be referred to as the "slum image." The device controller 20, which will be described later, identifies the position and orientation of the display 14 in real space based on the AR marker captured in the slum image, which will be described later.

The pupil position sensor 18 is a sensor that detects the positions of the pupils of the right and left eyes of the user 100, and is fixed, for example, near the center of the rim 24. This pupil position sensor 18 can be formed, for example, using a camera, etc.

The device controller 20 controls the operation of the wearable device 12. The device controller 20 acquires the images and position information acquired by the slam camera 16 and the pupil position sensor 18, processes this information, and displays the target image on the display 14.

The device controller 20 is physically a computer having a processor 20a, memory 20b, and a communication I/F 20c. This "computer" also includes a microcontroller in which a computer system is incorporated into a single integrated circuit. The processor 20a refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPU: Central Processing Unit, etc.) and dedicated processors (e.g., GPU: Graphics Processing Unit, ASIC: Application Specific Integrated Circuit, FPGA: Field Programmable Gate Array, programmable logic device, etc.).

The memory 20b holds digital data necessary for the computer to process. This memory 20b includes at least one of a main memory connected to the processor 20a via a memory bus, and a secondary storage device accessed by the processor 20a via an input/output channel. Such memory 20b can be composed of semiconductor memory (e.g., RAM, ROM, solid-state drive, etc.).

The communication I/F 20c is wirelessly connected to other electronic devices, specifically the in-vehicle system 28, and can also be connected to various sites via the Internet. In particular, it is capable of communicating with an information center 30 that provides vehicle information. For example, the communication I/F 20c may also transmit and receive data to and from the in-vehicle system 28 via short-range wireless communication such as CAN (Controller Area Network) communication, Bluetooth (registered trademark), Wi-Fi (registered trademark), and infrared communication.

The functions of the device controller 20 may be realized by an external system such as a computer in the in-vehicle system 28, a computer in the information center 30, or another portable computer (such as a smartphone). In this case, the device controller 20 sends information from the slam camera 16 and pupil position sensor 18 to an external system such as the in-vehicle system 28, receives image data as a result of processing, and displays it on the display 14. In addition, some of these processes can also be performed in the external system.

The in-vehicle system 28 is a system installed in the vehicle and controls various in-vehicle devices. As shown in FIG. 3, the in-vehicle system 28 includes interior parts such as a meter display 40a provided on the instrument panel, a multi-display 40b provided on the center console, and an electronic inner mirror 40c provided on the inside of the upper part of the windshield. In this embodiment, these interior parts are used as the AR marker 60.

Figure 3 is a diagram that shows a schematic view of the driver, who is the user 100. The meter display 40a is a display that displays information about the vehicle's state, such as vehicle speed and fuel efficiency. As shown in Figure 3, the meter display 40a is located behind the steering wheel 56 as seen from the driver, and the driver can see the display area of the meter display 40a through the steering wheel 56.

The multi-display 40b is a display that displays information about in-vehicle electronic devices (e.g., a navigation device or an audio device). As shown in FIG. 3, the multi-display 40b is located in the center of the instrument panel in the vehicle width direction, that is, in a location generally called the center console.

The electronic inner mirror 40c is a display that shows an image of the rear of the vehicle captured by a backup camera (not shown). This electronic inner mirror 40c is used in place of a rearview mirror (room mirror) that shows the rear of the vehicle by optical reflection. The electronic inner mirror 40c may be switchable between a digital mode that shows an image and a mirror mode that shows the rear by optical reflection. As shown in FIG. 3, the electronic inner mirror 40c is placed in the same position as a general rearview mirror, that is, near the top edge of the windshield glass. Note that a general room mirror may be used instead of the electronic inner mirror.

As described above, the device controller 20 generates data for the target image to be displayed on the display 14. Here, the display form of the target image on the display 14 can be either "device-fixed display" or "space-fixed display", but in this embodiment, "space-fixed display" is used. This space-fixed display is a display form in which a target image representing a specific object is displayed as if it exists in real space.

For example, as shown in Figure 4, consider a case where a user 100 is looking at a real space in which a table 80 actually exists through the display area 22 of the display device 14. In this case, as shown in state S1 in Figure 4, when a target image 50 representing a "ball" is displayed in the display area 22, naturally, the real space including the table 80 and the target image 50 of the "ball" are simultaneously reflected in the field of view of the user 100.

In the case of device-fixed display, the display position of the target object 72 (a "sphere" in the example of FIG. 4) indicated by the target image 50 is determined regardless of the real space. Therefore, in device-fixed display, as shown in state S2 in FIG. 4, even if the viewpoint of the user 100 moves, the display position, size, and shape of the target image 50 within the display area 22 do not change.

On the other hand, in a spatially fixed display, the location of the target object 72 (a "ball" in the example of FIG. 4) shown by the target image 50 in real space is identified, and the target image 50 is displayed as if it were actually located at the identified location. For example, in a spatially fixed display, consider a case in which the "ball" target object 72 is assumed to be located on a table 80 in real space. In this case, as shown in state S3, the display position, size, and shape of the "ball" in the display area 22 are changed so that the "ball" appears to be located on the table 80 even if the viewpoint of the user 100 moves.

In this way, by displaying the target image 50 in a spatially fixed display, the user 100 is given the illusion that the target object 72 shown in the target image 50 actually exists. In other words, by displaying the target image 50 in a spatially fixed display, it is possible to add, delete, emphasize, or attenuate information in the real environment, thereby expanding the real world as seen by humans. This technology is generally called "augmented reality" or "AR."

Next, an example of the display of the target image 50 in this example will be described. FIG. 5 is a diagram showing a schematic view of the user 100 (the driver in this example) when the target images 50a and 50b are displayed. In the example of FIG. 5, the target image 50a showing the traveling direction of the vehicle and the target image 50b showing a message to the driver are displayed in a fixed spatial manner. These target images 50a and 50b are displayed on the display 14 in the same position and size as when the target object shown by the image actually exists. For example, the target image 50a is displayed in the display area 22 in the same position and size as when the arrow-shaped object represented by the target image 50a actually exists on the road surface in front of the vehicle that actually exists. In addition, the target image 50b is displayed in the display area 22 in the same position and size as when the text object represented by the target image 50b actually exists, to the upper right of the steering wheel 56 that actually exists. Therefore, if the line of sight of the user 100 moves, the display position and size of these target images 50a and 50b in the display area 22 also change.

In this way, in the case of spatially fixed display, the target image 50 can be displayed taking into consideration the arrangement of real objects, so that it is possible to reliably prevent the target image 50 from interfering with driving operations. Furthermore, in the case of spatially fixed display, the target image 50 can be displayed in a position that is correlated with a real object (e.g., a pedestrian), so that the attention of the user 100 can be effectively directed to that object.

In order to perform spatially fixed display, it is necessary to accurately detect the position of the pupil relative to the display 14, and the position and orientation of the display 14 in real space. The device controller 20 determines the position of the target image 50 within the display area 22, etc., based on the position and orientation of the target object in real space, the position and orientation of the display 14 in real space, and the position of the pupil relative to the display 14. Of these, the position of the pupil relative to the display 14 is detected by the pupil position sensor 18, as described above.

The position and orientation of the display 14 in the real space are calculated by the device controller 20 performing visual simultaneous localization and mapping (visual SLAM) based on the slum image obtained by the slum camera 16. Visual SLAM is a technology that simultaneously estimates three-dimensional information of the environment and the position and orientation of the camera from an image captured by a camera. To perform this visual SLAM, the characteristic shapes of multiple interior parts in the vehicle are recognized as AR markers 60 (see Figure 3). The device controller 20 extracts multiple AR markers 60 from the slum image captured by the slum camera 16, and can calculate the position and orientation of the display 14 in the real space from the positional relationship in these slum images. It can also calculate the position and orientation of the display 14 in the real space based on the coordinates, size, distortion, etc. of one AR marker 60 in the slum image.

To perform this visual slam, an AR marker 60 (see FIG. 3) is present inside the vehicle. In this embodiment, the AR marker 60 is identified by image recognition based on the shape of the interior part. The shape, size, and installation position of the AR marker 60 are pre-registered by initial setting. Therefore, the device controller 20 extracts an image of the AR marker 60 from the slam image captured by the slam camera 16. Then, based on the coordinates, size, distortion, etc. of the image of the AR marker 60 in the slam image, the position and orientation of the display 14 in real space can be calculated. In addition, by performing image recognition on multiple AR markers 60 and referring to the installation positions of the multiple AR markers 60, the position and orientation of the display 14 can be easily calculated. Note that, instead of an interior part, a marker with a predetermined shape and size and installed at a predetermined position inside the vehicle can also be used as the AR marker 60. Furthermore, these different types of AR markers can be used together.

<AR marker registration>
6 is a flowchart showing a process for acquiring the shape of an interior part of a vehicle as the AR marker 60 in the device controller 20.

First, the wearable device 12 is brought into the vehicle, and, for example, the user puts the wearable device 12 on their head in the driver's seat, turns the power on, and sets the AR marker registration mode. To set the AR marker registration mode, the wearable device 12 may have an operation button or the like, or may be operable from a terminal such as a smartphone. When the AR marker registration mode is set, an AR marker recognition program is executed in the device controller 20.

In the AR marker registration mode, first an image is captured from the slam camera 16 (S11). This image is the same as the image seen by the driver in the driver's seat. At this time, it is advisable to have the user stand still in a position looking forward in the driver's seat, and then move their head to obtain images and videos from different positions.

The area in which feature parts are detected is limited from the captured image (S12). For example, if a candidate to be used as the AR marker 60 is known in advance, the recognition area is limited to a recognition area 70 including the location of that candidate (see FIG. 3). In this example, the area is limited to the upper center area where the electronic inner mirror 40c is located, and the instrument panel area including the meter display 40a and multi-display 40b. Also, if there is an AR marker 60 that has been recognized and registered in advance, it is advisable to use that data to limit the target area. However, such limitations may not be imposed and the entire captured image may be subject to feature recognition.

Next, image recognition processing is performed on the target image to detect characteristic shapes, i.e., characteristic parts (S13). This can be done by detecting contours and extracting shapes using edge detection or the like. Note that if the target is a display frame or the like, recognition may be limited to shapes close to that. Furthermore, characteristic parts may be recognized using information from previously registered AR markers.

As described above, when the wearable device 12 is moving and acquiring images, it is advisable to exclude feature parts outside the vehicle by detecting the movement of the detected feature parts or by comparing two images of the wearable device 12 taken in different positions to calculate the distance to the feature parts.

Then, one characteristic part is extracted (S14) and it is determined whether or not it belongs to the steering wheel 56 (S15). The steering wheel 56 is basically a circular ring shape and has characteristics different from others, so it can be easily identified. Note that information about the shape of the steering wheel 56 may be stored and used.

Then, the characteristic parts that do not belong to the steering wheel 56 are registered as AR markers (S16).

On the other hand, if the target feature part belongs to the steering wheel 56 in S15, S15 is skipped and it is determined whether extraction of the feature part has been completed (S17). However, if the small extraction of the feature part has not been completed and there are still feature parts remaining, the process returns to S14 and the next feature part is processed, and this is repeated until there are no more feature parts.

Figure 7 is a diagram explaining the area specification when excluding the steering wheel 56. In the figure, the area hatched with diagonal lines is the excluded area that is excluded as the steering wheel 56 belongs to. In this way, the upper half of the steering wheel 56 where the back is visible is included in the target area. As a result, the meter display 40a located on the back side of the steering wheel 56 becomes the extraction target, and can be registered as the AR marker 60.

In this way, the device controller 20 functions as an exclusion means for excluding the steering wheel from the characteristic parts, an extraction means for extracting the characteristic parts, and a display position determination means for determining the display position of the target image 50.

<Display of target image>
FIG. 8 is a flowchart showing a process for displaying a target image while driving.

First, an image is captured by the slam camera 16 (S21). Information about the registered AR marker 60 is used to detect the AR marker 60 from within the image (S22). Then, the display position of the target image 50 is determined using the position information of the recognized AR marker 60 (S23), and the target image 50 is displayed at the determined position (S24).

In this way, according to this embodiment, the steering wheel 56 is excluded from the interior parts that are registered as AR markers 60.

Because the steering wheel 56 moves or is hidden by the hands, excluding it reduces the processing burden when recognizing the AR marker 60 and also reduces the possibility that the AR marker 60 will not be recognized.

10 Image display device, 12 Wearable device, 14 Display, 16 Slam camera, 18 Pupil position sensor, 20 Device controller, 20a Processor, 20b Memory, 20c Communication I/F, 22 Display area, 24 Rim, 26 Temple, 28 In-vehicle system, 30 Information center, 40a Meter display, 40b Multi-display, 40c Electronic inner mirror, 50 Target image, 56 Steering wheel, 60 AR marker, 70 Recognition area, 72 Target object, 80 Table, 100 User.

Claims (2)

A display device that is mounted on the head of a vehicle occupant and displays a target image in a superimposed manner in the user's field of vision;
a slum camera fixed to the display and configured to capture a slum image of the periphery of the display;
An extraction means for performing image recognition processing on the obtained slam image and extracting a characteristic shape as a characteristic portion;
a determination means for determining whether the extracted feature portion belongs to a steering wheel;
an exclusion means for excluding the extracted feature portion when the extracted feature portion belongs to a steering wheel;
A registration means for registering a characteristic portion not belonging to the steering wheel as an AR marker;
a display position determination means for determining a display position of the target image by using the registered AR marker;
An image display device comprising: The image display device according to claim 1 ,
An object on the rear side of the steering wheel that can be seen through the steering wheel is set as a target for extracting the characteristic portion.
Image display device.