JP7838596B2 - In-vehicle device and recording method - Google Patents

Description

This invention relates to an in-vehicle device and a recording method.

A technology has been proposed that extracts matters of interest from the behavior of vehicle occupants and presents information related to those matters of interest. Patent Document 1 discloses a technology that, after detecting an unusual driving pattern, determines whether the driver is showing interest based on changes in behavior such as the movement of the driver's eyes, and then presents information about the facilities that the driver is showing interest in.

Patent Document 2 presents a technology for extracting the interests of bus passengers and presenting related information. Here, passenger interests are extracted from dynamic information such as eye movements and conversations, as well as customer information pre-registered in the passenger's account.

The technologies described in Patent Documents 1 and 2 make it possible to present information desired by the driver or other party, triggered by a change in behavior. On the other hand, it is difficult to use these technologies to record things the driver suddenly remembers while driving the vehicle, or to extract matters of interest that the driver is not consciously aware of.

Japanese Patent Publication No. 2009-168473 Japanese Patent Publication No. 2020-57066

As described above, when a driver remembers something while driving, they cannot take their hands off the wheel, making it difficult to record information related to the remembered fact.
Furthermore, while driving, drivers must think about many things, such as vehicle operation at traffic lights and curves, avoiding close proximity to other vehicles, and choosing a driving route. As a result, there is a problem that drivers forget what they remembered when they finish driving. Therefore, this invention provides an in-vehicle device and recording method that can record information related to a fact that the driver remembers while driving.

The present invention provides an in-vehicle device comprising: a gaze detection unit for detecting the direction of the vehicle driver's gaze; an estimation unit for determining whether the driver is looking at a specific object in the direction of their gaze, based at least on the speed of the driver's gaze movement, and for estimating the object; an information processing unit for recording information related to the object as memo information; and an output unit for outputting the memo information recorded during driving when the vehicle's operation is completed.

The present invention provides a recording method comprising: a gaze detection step for detecting the direction of a vehicle driver's gaze; an estimation step for determining whether the driver is looking at a specific object in the direction of their gaze, based at least on the speed of the driver's gaze movement, and estimating the object; an information processing step for recording information related to the object as memo information; and an output step for outputting the memo information recorded during driving when the vehicle driving is completed.

According to the present invention, it is possible to provide an in-vehicle device and recording method that can record information related to a fact that a driver remembers while driving.

This is a block diagram showing the configuration of the in-vehicle device according to Embodiment 1. This flowchart shows the operation of the in-vehicle device according to Embodiment 1. This is a block diagram showing the configuration of the in-vehicle device according to Embodiment 2.

(Embodiment 1)
First, the configuration of the in-vehicle device 100 according to Embodiment 1 will be explained using Figure 1. The in-vehicle device 100 is, for example, a drive recorder. The in-vehicle device 100 includes an image acquisition unit 1
The system comprises 10, a gaze detection unit 120, an estimation unit 130, a storage unit 140, a determination unit 150, an information processing unit 160, and an information presentation unit 170.

The image acquisition unit 110 acquires images of the vehicle driver and the exterior of the vehicle from the camera 10. The image acquisition unit 110 may acquire images of the driver and the exterior of the vehicle simultaneously using an ultra-wide-angle lens (fisheye, 360 lens, etc.), or it may acquire multiple images of the driver and the exterior of the vehicle taken by separate cameras. The camera 10 may be an infrared camera. The acquisition of images of the exterior of the vehicle by the image acquisition unit 110 is not necessarily an essential configuration and may be omitted.

The gaze detection unit 120 detects the direction of the driver's gaze from the driver's facial image. The direction of the gaze is detected as one of nine quadrants, for example, up, center, down, left, center, or right. It is preferable that the direction of the gaze be detected with fine precision. Furthermore, the gaze detection unit 120 may also detect the period during which the driver is looking in a certain direction, or it may detect the movement of the gaze associated with the vehicle's movement.
The system may also detect changes in gaze direction caused by looking at a moving object. The gaze detection unit 120 outputs the detected gaze direction to the estimation unit 130.

The gaze detection unit 120 can detect the line of sight based on the positional relationship between the inner corner of the eye and the iris, and the positional relationship between the corneal reflection and the pupil. The gaze detection unit 120 may also detect the direction the driver is looking based on the orientation of the face and use that as the direction of the line of sight. Known technologies may be used as the line of sight detection technology.

The estimation unit 130 estimates the object the driver is looking at, at least based on the direction of their line of sight.
For example, if the estimation unit 130 detects that the driver's line of sight is slightly below the center in the left-right direction, it estimates that the object the driver is looking at is the road or another vehicle traveling in front of the vehicle. The object the driver is looking at may also be estimated based on the line of sight and the surrounding conditions of the vehicle. The surrounding conditions of the vehicle are determined from images of the outside of the vehicle taken by the image acquisition unit 110, or from map information obtained from the internet or other sources via a car navigation system (not shown) or a communication unit (not shown). The estimation unit 130 outputs the estimation result to the determination unit 150.

Here, the estimation unit 130 first determines whether the driver is looking at the entire direction of their gaze or looking at (gazing at) a specific object in that direction, based on the movement of the driver's pupils, changes in the direction of their face, and the duration for which the driver is looking in a certain direction. For example, if the estimation unit 130 determines that the driver is often looking at the center of the left-right direction and that their gaze movements are small,
It is assumed that the driver is looking at the entire direction of their line of sight. The speed and direction of the driver's eye movement,
The system may determine whether there is a correlation with the vehicle's speed and whether the driver is looking at a specific object in the direction of their line of sight. If the driver is looking at a specific object, the estimation unit 130 will
The system estimates what the driver is looking at. The object may be estimated as a single facility, such as "○○ Post Office," or as a collective term for multiple stores, such as "○○ Burger."
It may be considered a genre, like "ramen shop." The subject is not limited to facilities; it could also include fields, trees, etc.

The estimation unit 130 may use image recognition technology to estimate the object based on the direction of gaze and the image of the vehicle's exterior. For example, if the driver is looking to the upper left and the image of the vehicle's exterior recognizes that what is there is a post office sign, the estimation unit 130 can estimate that the object the driver is looking at is a post office. The estimation unit 130 may also use image recognition to estimate that the object the driver is looking at is a road, other vehicles, traffic lights, signs, etc.

Furthermore, the estimation unit 130 may estimate what the driver is looking at based on the direction of gaze and map information. For example, if the driver is looking slightly to the left in front, the estimation unit 130 may obtain from the map information that a facility located to the left in front of the vehicle's current position is a post office, and estimate that what the driver is looking at is a post office. The estimation unit 130 may also obtain from the map information the presence of traffic signals, signs, etc., and estimate that these are what the driver is looking at.

Furthermore, the estimation unit 130 may estimate what the driver is looking at based on the direction of gaze and the vehicle's driving information. For example, if the vehicle turns left after the driver looks directly to the left, the estimation unit 130 may estimate that what the driver is looking at is the left side mirror.

The estimation unit 130 may estimate that the object the driver is looking at is the accident site, based on the direction of the line of sight and information indicating the location of the accident site. Information indicating the location of the accident site may be obtained from the internet or the like via a communication unit (not shown).

The storage unit 140 is a storage device or storage medium such as a hard disk, flash memory, or SD card. The storage unit 140 may also include a volatile storage device such as RAM (Random Access Memory), which is a storage area for temporarily holding information. The storage unit 140 is,
The history 141 and memo information 142 are stored. The storage unit 140 may further store images captured by the camera 10.

History 141 is a database that manages the history of the estimation results of the estimation unit 130. History 1
41 may store only objects unrelated to the driving of the vehicle. Also, history 141 may store the estimated object in association with the time the driver saw the object. Furthermore, history 1
41 may store the length of time the driver viewed the object. Memo information 142 is information recorded by the information processing unit 160, which will be described later. Note that history 141 and memo information 142
This data may be stored on a server outside the in-vehicle device 100.

If the object estimated by the estimation unit 130 is not related to driving the vehicle, the determination unit 150 determines, based on the history 141, how often the driver visually observes the object, that is, whether or not they normally see it. The determination result by the determination unit 150 is sent to the information processing unit 160 or the information presentation unit 1
Output to 70.

Here, the determination unit 150 first determines whether the object the driver is looking at is related to the operation of the vehicle. Things related to the operation of the vehicle include roads, other vehicles, traffic lights, signs, mirrors, etc. The accident scene may also be included in things related to the operation of the vehicle.

Next, the determination unit 150 determines whether the driver normally looks at the object.
The determination unit 150 may determine whether the frequency of visually inspecting the target is greater than or equal to a predetermined value of 1 or less, depending on whether the target is included in the history 141 or more, or whether the frequency of visually inspecting the target is greater than or equal to a predetermined number, depending on whether the target is included in the history 141 in a predetermined number or more. The determination unit 150 may also determine the frequency of visually inspecting the target by referring to the history 141 for the most recent period, such as whether the target has been seen three or more times in a week, or by comparing it with the driving history, such as whether the target has been seen once or more when passing through a road adjacent to the target twice.

Here, the determination unit 150 may make a determination by taking into account the amount of time the driver is looking at the object (gaze time). For example, the determination unit 150 may determine that the driver is normally looking at the object if the number of times the driver has looked at the object for one second or more is greater than a predetermined number.

Here, the determination unit 150 may refer to the history 141, taking into account conditions such as time of day and day of the week. For example, the determination unit 150 may determine whether the driver usually sees the subject, taking into account that restaurants tend to be seen at night and on holidays.

The information processing unit 160 records information related to the object as memo information 142 if the driver does not normally look at the object. Therefore, the information processing unit 160 records memo information 142 when the object is not related to driving and is not something the driver normally looks at. The information processing unit 160 may record the name of the object estimated by the estimation unit 130 as memo information 142, or it may acquire an image of the object from the image acquisition unit 110 and record it as memo information 142. The information processing unit 160 may also capture map information of the object estimated by the estimation unit 130, for example, the screen of a car navigation system (not shown), and record it as memo information 142.

Furthermore, the information processing unit 160 creates a message that includes the name of the target as part of it, and memo information 1
It may be recorded as 42. The message may be, for example, "Do you have any business at the post office?" if the object the driver is looking at is a post office. The information processing unit may create the message as an audio memo or an image memo and record it as memo information 142.

Furthermore, the memo information 142 may be voiced by the driver. The information processing unit 160 may notify the driver of a message including the name of the target (for example, "Do you have any business at the post office?"), and then record the driver's voice using the microphone 20 as memo information 142. The information processing unit 160 may output the message from the speaker 30 or to a monitor. This allows the driver to recall that they had business at the post office and record the information by voice or other means ("I had business at the post office").

Here, the information processing unit 160 may record a video of the period including both the time the subject was filmed and the period during which the driver spoke as memo information 142. Alternatively, the information processing unit 160 may record the driver's voice and the external video of the vehicle during the period in which the voice was spoken as memo information 142. In other words, the information processing unit 160 may detect the driver's voice and record the external video for the period until the end of the voice.

Furthermore, it is preferable to associate the estimated name of the target (for example, "○○ Post Office") with the created file. Therefore, the information processing unit 160 may record the memo information 142 in association with the name of the target. For example, the information processing unit 160 may use the name "○○ Post Office" as the file name of the memo information 142, and a voice memo synthesized from that name,
The memo information 142 may be combined with the image memo, the name may be superimposed on the image memo as subtitle data, or the name may be embedded in the memo information 142 as metadata.

The information display unit 170 displays information related to the object to the driver when the driver is normally looking at the object. Therefore, the information display unit 170 displays information when the object is not related to driving and is something the driver normally looks at. The information display unit 170 may output the displayed information to the speaker 30 or to the monitor.

The information display unit 170 may output information related to the name of the subject, such as, "You often look at XX Ramen." This allows the driver to recognize their potential interest.

Furthermore, the information display unit 170 may obtain information related to the name of the target from the Internet or other sources via a communication unit (not shown) and output the obtained information. For example, the information display unit 170 can show the driver whether or not "○○ Ramen" is running a campaign, or whether or not it has a good reputation.

Furthermore, the information display unit 170 may transmit information related to the name of the target to a car navigation system (not shown) so that it can be searched and registered on the car navigation system.

Furthermore, the information display unit 170 may display information if the subject is related to driving and is something the driver does not normally see. For example, if the driver sees a no-parking sign that they do not normally see, it can be determined that the driver is looking for a parking space, and the information display unit 170 may display parking information found through an internet search or other means to the driver.
For example, if the driver is looking in the rearview mirror more than usual, it can be determined that they are concerned about the vehicle behind them, and the information display unit 170 can then display information to encourage them to focus their attention on the road ahead.

Next, the operation flow of the in-vehicle device 100 will be explained using Figure 2. First, the gaze detection unit 120 detects the driver's gaze direction from the driver's face image in the image acquired by the image acquisition unit 110 (step S101). Next, the estimation unit 130 determines whether the driver is looking at the entire gaze direction or is looking at (gazing at) a specific object in the gaze direction (step S102). If the driver is not looking at a specific object (step S
If the driver is looking at a specific object (Yes in step S102), the in-vehicle device 100 returns to the process of step S101. If the driver is looking at a specific object (Yes in step S102), the estimation unit 130 estimates the object in the direction of the driver's line of sight (step S103).

Next, the determination unit 150 determines whether the object estimated in step S103 is directly related to driving (step S104). If the object is related to driving the vehicle (Yes in step S104), the in-vehicle device 100 returns to the process in step S101.

If the subject is not related to the operation of a vehicle (No. in step S104), the determination unit 15
A value of 0 determines whether the driver normally sees the object (step S105). If the driver does not normally see the object (No. in step S105), the information processing unit 160 determines that the driver has remembered the object (step S106) and records an image memo or voice memo as memo information 142 (step S107). At this point, the information processing unit 160 may also notify the driver of a message such as, "Do you have any business at the post office?"

If the driver regularly observes the object (Yes in step S105), the information display unit 170 determines that the driver is interested in the object (step S108) and presents information about the object to the driver (step S109). Finally, the estimation result history 141 is updated (step S110).

The in-vehicle device 100 according to Embodiment 1 distinguishes between objects the driver is looking at and objects the driver normally looks at and objects the driver does not normally look at.
00 can present information of interest to the driver. Furthermore, it can automatically record information the driver recalls as memo information while driving, even when the hands are occupied.

(Embodiment 2)
The in-vehicle device 100a according to Embodiment 2 has an information processing unit 16 when the vehicle operation is finished.
0 outputs the memo information recorded. Figure 3 is a block diagram showing an example configuration of the in-vehicle device 100a. The differences from Embodiment 1 will be described below.

The in-vehicle device 100a is connected to the driver's communication terminal 40 via the network 200. The network 200 includes the internet, intranet, mobile phone network,
This refers to a communication network such as a LAN (Local Area Network). Communication terminal 40
This includes mobile phones, smartphones, tablet devices, and PCs (Personal Com).
(Putter, etc.) Note that the in-vehicle device 100a may output the memo information 142 to a speaker 30, a monitor screen (not shown), etc., instead of the communication terminal 40.

In addition to the configuration shown in Figure 1, the in-vehicle device 100a includes a communication unit 180 and an output unit 190. The communication unit 180 is a communication interface with the network 200.
If 00a outputs memo information 142 to the speaker 30 or the like, the in-vehicle device 100a does not need to be equipped with a communication unit 180.

The output unit 190 determines whether or not the operation of the vehicle on which the in-vehicle device 100a is installed has ended. The output unit 190 works in conjunction with a car navigation system (not shown) installed in the vehicle.
The system may determine that the drive has ended when the vehicle arrives near the destination or when it is a predetermined distance from the destination (e.g., 100m). The destination may be determined not only by the destination set in the car navigation system, but also by the registered home address or the current location being a parking lot. In addition, the output unit 190 acquires the vehicle's gear information and determines when the vehicle has entered a parking operation,
The system may determine that driving has ended when it is determined from the driving information that the parking operation has been completed. The output unit 190 may also determine that driving has ended by detecting that the engine has been turned off.

The output unit 190 outputs memo information 142 when the vehicle operation is completed.
90 may output the voice memo as sound from the speaker 30, or display the image memo on a monitor (not shown). The output unit 190 may output the memo information 142 via the communication unit 180 to a communication terminal 40 that has been pre-registered as a recipient. The output unit 190 may also output the memo information 142 via the communication unit 180 to a pre-registered email address, telephone number, SNS (social networking service), etc.

The in-vehicle device 100a according to Embodiment 2 outputs memo information recorded when driving is completed. Therefore, the driver can check the memo information after driving is finished and perform any tasks that come to mind. In addition, the driver can check the memo information output from the speaker, monitor, etc., and take notes in a notebook or similar.

Furthermore, the aforementioned program is not compatible with various types of non-temporary computer-readable media (no
It can be stored using n-transitor computer-readable medium and supplied to a computer. Non-transitor computer-readable medium can be various types of tangible storage media.
Includes (um). Non-temporary computer-readable media include, for example, magnetic recording media, magneto-optical recording media, CD-ROM (Read Only Memory), CD-R, CD-R/W,
This includes semiconductor memory. Magnetic recording media include, for example, flexible disks, magnetic tapes, and hard disk drives. Magneto-optical recording media include, for example, magneto-optical disks. Semiconductor memory includes, for example, mask ROM and PROM (Programmable ROM).
M), EPROM (Erasable EPROM), Flash ROM, RAM (Ran
These include DOM Access Memory, etc. Also, programs are stored in various types of temporary computer-readable media (transitor computer reads).
The computer may be supplied with a temporary computer-readable medium. Examples of temporary computer-readable mediums include electrical signals, optical signals, and electromagnetic waves. Temporary computer-readable mediums can supply programs to a computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels.

The in-vehicle device according to the above embodiment can be applied to the field of driver monitoring systems. Furthermore, devices other than in-vehicle devices can also detect a person's gaze and estimate what that person is looking at. For example, a similar system can be realized by attaching a gaze detection device and a camera that captures the surrounding scenery to a glasses-type device. This allows,
This makes it possible to investigate a person's potential interests not only while they are driving, but also in their everyday life.

10 Camera 20 Microphone 30 Speaker 40 Communication terminal 100, 100a In-vehicle device 110 Image acquisition unit 120 Eye-gaze detection unit 130 Estimation unit 140 Storage unit 141 History 142 Memo information 150 Judgment unit 160 Information processing unit 170 Information presentation unit 180 Communication unit 190 Output unit 200 Network

Claims (2)

A gaze detection unit that detects the direction of the vehicle driver's gaze,
An estimation unit that determines, at least based on the speed of the driver's eye movement, that the driver is looking at a specific object in the direction of their gaze, and estimates the object;
A database for managing the history of the estimation results of the estimation unit,
If the number of times the driver has viewed the object for a predetermined period of time or longer is not included in the history at a predetermined number or more, an information processing unit records information related to the object as memo information.
An output unit outputs the memo information recorded during operation when the operation of the vehicle ends,
A vehicle-mounted device equipped with this technology. A gaze detection step that detects the direction of the vehicle driver's gaze,
An estimation step of determining, at least based on the speed of the driver's eye movement, that the driver is looking at a specific object in the direction of their gaze, and estimating the object;
A database for managing the history of estimation results in the estimation step ,
If the number of times the driver has viewed the object for a predetermined period of time or longer is not included in the history at a predetermined number or more, the information processing step includes recording information related to the object as memo information.
An output step in which, when the operation of the vehicle ends, the memo information recorded during operation is output,
A recording method equipped with [a specific feature/feature].