JP7619146B2 - Mobile terminal identification system - Google Patents

Description

This disclosure relates to a system that identifies a driver's mobile device.

Conventionally, systems have been proposed that provide services to drivers by connecting a mobile terminal carried by the driver to a vehicle (see, for example, Patent Document 1). One example of such a service is a service that determines whether the driver is awake.

JP 2014-115983 A

A mobile terminal and a vehicle are usually connected via wireless communication. However, a vehicle may have passengers other than the driver. If the passenger carries a mobile terminal, the passenger's mobile terminal may be connected to the vehicle via wireless communication. In this way, when multiple mobile terminals are connected to the vehicle, the system needs to determine (identify) which mobile terminal is the driver's mobile terminal. If the system cannot identify the driver's mobile terminal, it cannot provide services to the driver.

One of the objectives of this disclosure is to provide technology that can identify a driver's mobile device in a system in which a mobile device and a vehicle are connected.

In one or more embodiments, the mobile terminal determination system includes:
A first imaging unit (75) mounted on a vehicle (VA) and configured to image a driver of the vehicle;
A mobile terminal (110) including a second imaging unit (202);
a control device (10) configured to execute a comparison process for comparing a first image (500) acquired by the first imaging unit with a second image (600) acquired by the second imaging unit, thereby determining whether the mobile terminal is a mobile terminal of the driver;
Equipped with.

The second image acquired by the second imaging unit of the portable device carried by the driver includes the driver. On the other hand, the second image acquired by the second imaging unit of the portable device carried by a passenger other than the driver does not include the driver. Therefore, the control device having the above configuration can determine which portable device is the driver's portable device by comparing the first image and the second image.

In one or more embodiments, the comparison process comprises:
comparing the faces recognized in the first image with the faces recognized in the second image;
A process of comparing the behavior of the occupant recognized in the first image with the behavior of the occupant recognized in the second image; and
comparing a change in luminance in the first image with a change in luminance in the second image;
At least one of the following is included.

In one or more embodiments, the mobile terminal determination system includes:
A first communication unit mounted on a vehicle (VA);
A mobile terminal (110) including a second communication unit (204);
A control device (10) configured to determine whether the mobile terminal is a mobile terminal of a driver based on a communication state between the first communication unit and the second communication unit;
Equipped with.

A control device having the above configuration can determine which mobile device is the driver's mobile device based on the communication status between the first communication unit and the second communication unit.

In one or more embodiments, the first communication unit is a short-range communication unit (401) installed near the driver's seat.

For example, the second communication unit of the mobile device carried by the driver can establish communication with the short-range communication unit, but the second communication unit of the mobile device carried by a passenger other than the driver cannot establish communication with the short-range communication unit. Taking such a situation into account, the control device having the above configuration can determine which mobile device is the driver's mobile device.

In one or more embodiments,
The first communication unit includes a plurality of short-range communication units installed separately within the vehicle.
Each of the plurality of short-range communication units measures a receiving strength of a signal transmitted from the second communication unit.
The control device is configured to determine whether the mobile terminal is a driver's mobile terminal based on the difference in reception strength among the multiple short-range communication units.

For example, assume that a first short-range communication unit of the multiple short-range communication units is installed near the driver's seat, and a second short-range communication unit of the multiple short-range communication units is installed in the back seat. When the first short-range communication unit and the second short-range communication unit each receive a signal from the second communication unit of a mobile terminal carried by the driver, the reception strength of the signal received by the first short-range communication unit is high, and the reception strength of the signal received by the second short-range communication unit is low. A control device having the above configuration can determine which mobile terminal is the driver's mobile terminal based on this difference in reception strength.

In one or more embodiments, the control device may be implemented by at least one processor programmed to perform one or more functions described herein. In one or more embodiments, the control device may be implemented in whole or in part by hardware configured by one or more application specific integrated circuits, i.e., ASICs. In the above description, the names and/or symbols used in the embodiments are added in parentheses to components corresponding to one or more embodiments described below. However, each component is not limited to the embodiment defined by the name and/or symbol.

1 is a schematic configuration diagram of a mobile terminal determination system according to a first embodiment. FIG. 1 is an external view of a mobile terminal according to a first embodiment. FIG. 1 is a schematic configuration diagram of a mobile terminal according to a first embodiment. FIG. 13 is a diagram showing a situation in which a driver operates a mobile terminal while automatic driving control is being executed. 1 is an example of an image (first image) captured by a driver monitor camera. 11 is an example of an image (second image) captured by the mobile terminal. FIG. 2 is a sequence diagram of an automatic driving device (automatic driving ECU) and mobile terminals (first mobile terminal and second mobile terminal).

First Embodiment
As shown in FIG. 1 , the mobile terminal determination system includes an automatic driving device 100 mounted (applied) to a vehicle VA, and a mobile terminal 110.

(Configuration of the automatic driving device)
The autonomous driving device 100 includes an autonomous driving ECU 10, an engine ECU 20, a brake ECU 30, a steering ECU 40, a meter ECU 50, and a navigation ECU 60. These ECUs are connected to each other via a controller area network (CAN) 90 so as to be able to transmit and receive data to and from each other.

ECU is an abbreviation for Electronic Control Unit, and is an electronic control circuit that has a microcomputer as its main component. The microcomputer includes a CPU, ROM, RAM, an interface, and non-volatile memory. The CPU realizes various functions by executing instructions (routines, programs) stored in the ROM. For example, an autonomous driving ECU 10 includes a microcomputer that includes a CPU 10a, ROM 10b, RAM 10c, an interface (I/F) 10d, and non-volatile memory 10e.

The engine ECU 20 is connected to the engine actuators 21. The engine actuators 21 include a throttle valve actuator that changes the opening of the throttle valve of the internal combustion engine 22. The engine ECU 20 can change the torque generated by the internal combustion engine 22 by driving the engine actuators 21. The torque generated by the internal combustion engine 22 is transmitted to the driving wheels via the transmission 23 and a driving force transmission mechanism (not shown). Therefore, the engine ECU 20 can control the driving force of the vehicle VA by controlling the engine actuators 21.

If the vehicle VA is a hybrid vehicle, the engine ECU 20 can control the driving force generated by either or both of the "internal combustion engine and electric motor" as the driving source. Furthermore, if the vehicle VA is an electric vehicle, the engine ECU 20 can control the driving force generated by the electric motor as the driving source.

The brake ECU 30 is connected to the brake actuator 31. The brake actuator 31 includes a known hydraulic circuit. The hydraulic circuit includes a reservoir, an oil pump, various valve devices, and the like. The brake actuator 31 adjusts the hydraulic pressure (i.e., braking pressure) supplied to the wheel cylinder 32 in response to instructions from the brake ECU 30. The frictional braking force generated on the wheel changes in response to the braking pressure. Therefore, the brake ECU 30 can control the braking force of the vehicle VA by controlling the brake actuator 31.

The steering ECU 40 is connected to an assist motor (M) 41. The assist motor 41 is incorporated in a steering mechanism 42. The steering mechanism 42 is a mechanism for steering the wheels by rotating the steering wheel SW. The steering mechanism 42 includes a steering wheel SW, a steering shaft US connected to the steering wheel SW, and a steering gear mechanism (not shown). The steering ECU 40 detects the steering torque input to the steering wheel SW by the driver using a steering torque sensor (not shown) provided on the steering shaft US, and drives the assist motor 41 based on this steering torque. The steering ECU 40 applies a steering torque (steering assist torque) to the steering mechanism 42 by driving the assist motor 41, thereby assisting the steering operation of the driver.

In addition, the steering ECU 40 drives the assist motor 41 according to a steering command (including the automatic steering torque described later) from the automatic driving ECU 10 while the automatic driving control described below is being executed. This automatic steering torque is different from the steering assist torque described above, and represents a torque applied to the steering mechanism 42 by a steering command from the automatic driving ECU 10 without requiring the driver to perform a steering operation. The automatic steering torque changes the steering angle of the steered wheels of the vehicle VA.

The meter ECU 50 is connected to an alarm device 51. The alarm device 51 includes a buzzer and a display device. The display device is, for example, a multi-information display provided in front of the driver's seat. In response to an instruction from the automatic driving ECU 10, the meter ECU 50 can sound a buzzer to alert the driver or display an alert mark (warning lamp) on the display device.

The navigation ECU 60 is connected to a GPS receiver 61 and a map storage unit 62. The GPS receiver 61 receives a GPS signal to detect the "latitude and longitude" of the location where the vehicle VA is located. The map storage unit 62 stores map information. The map information includes road information. The road information includes the road type, road width, road curvature, etc. In this example, the road type is either a motorway (expressway) or a general road. The navigation ECU 60 performs various calculations based on the latitude and longitude of the location where the vehicle VA is located and the map information, etc., and displays the position of the vehicle VA on the map on a touch panel (not shown).

The vehicle speed sensor 71 detects the traveling speed (vehicle speed) SPD of the vehicle VA and outputs a signal representing the vehicle speed SPD to the automatic driving ECU 10.

The surrounding sensor 72 is adapted to acquire information about the road surrounding the vehicle VA (including the lane in which the vehicle VA is traveling) and about three-dimensional objects present on the road. Three-dimensional objects include, for example, moving objects such as pedestrians, four-wheeled vehicles and two-wheeled vehicles, as well as fixed objects such as guardrails and fences. Hereinafter, these three-dimensional objects will be referred to simply as "objects." The surrounding sensor 72 is equipped with a radar sensor 73 and a camera sensor 74.

The radar sensor 73, for example, emits millimeter wave band radio waves (hereinafter referred to as "millimeter waves") to the surrounding area including at least the area in front of the vehicle VA, and receives millimeter waves (i.e., reflected waves) reflected by objects present within the emission range. The radar sensor 73 then determines whether an object is present and calculates information indicating the relative relationship between the vehicle VA and the object. The information indicating the relative relationship between the vehicle VA and the object includes the distance between the vehicle VA and the object, the orientation (or position) of the object relative to the vehicle VA, and the relative speed between the vehicle VA and the object. Hereinafter, the information indicating the relative relationship between the vehicle VA and the object will be simply referred to as "object information".

The camera sensor 74 photographs the scenery ahead of the vehicle VA to obtain image data. Based on the image data, the camera sensor 74 recognizes the left and right dividing lines (e.g., left and right white lines) that define the driving lane, and calculates parameters that indicate the shape of the driving lane (e.g., curvature) and parameters that indicate the positional relationship between the vehicle VA and the driving lane. The parameter that indicates the positional relationship between the vehicle VA and the driving lane is, for example, the distance from the left or right dividing line to the center position of the vehicle VA in the vehicle width direction. The information obtained by the camera sensor 74 is referred to as "lane information." The camera sensor 74 may be configured to determine the presence or absence of an object based on the image data and calculate object information.

The surrounding sensor 72 outputs information about the surrounding conditions of the vehicle VA, including "object information and lane information," to the autonomous driving ECU 10 as "vehicle surrounding information."

The driver monitor camera (first imaging unit) 75 is disposed above the column cover (so-called steering column cover) of the steering wheel SW. The driver monitor camera 75 captures an image of the driver to obtain image data. Hereinafter, the image data obtained by the driver monitor camera 75 is referred to as the "first image." The position and orientation of the driver monitor camera 75 are set so that at least the driver's face is included in the first image. The driver monitor camera 75 outputs the first image to the autonomous driving ECU 10.

The driver monitor camera 75 may be set in another position within the vehicle VA as long as it can capture the driver's face. For example, the driver monitor camera 75 may be installed near the dashboard or windshield.

Furthermore, the autonomous driving ECU 10 is connected to a communication unit 80. The communication unit 80 is configured to be capable of transmitting and receiving information to and from the mobile terminal 110 via short-range wireless communication, which will be described later.

The automatic driving switch 81 is a switch that is operated by the driver. By operating the automatic driving switch 81, the driver can set the operating state of the automatic driving control, which will be described later.

(Autonomous Driving Control Overview)
Hereinafter, the automatic driving ECU 10 is simply referred to as "ECU 10". The ECU 10 is configured to execute automatic driving control. The level of automatic driving control (so-called automatic driving level) is classified into a plurality of levels 0 to 5 by the SAE (Society of Automotive Engineers). In this example, the automatic driving level is level 3. Level 3 is a level at which all driving operations are executed by automatic driving control under a specific environment, and when a request for changing driving operations (hereinafter simply referred to as "driving operation change request") occurs, the driver is required to perform driving operations. Therefore, automatic driving control at level 3 may be referred to as "conditional automatic driving control". Automatic driving control includes speed control and steering control.

An example of speed control is adaptive cruise control (hereinafter simply referred to as "ACC"). ACC itself is publicly known (see, for example, JP 2014-148293 A, JP 2006-315491 A, and the specification of Japanese Patent No. 4172434, etc.).

ACC includes two types of control: constant speed cruise control and preceding vehicle following control. Constant speed cruise control is a control that adjusts the speed of vehicle VA so that the traveling speed of vehicle VA matches the target speed Vset without the need to operate the accelerator pedal or brake pedal. Preceding vehicle following control is a control that makes vehicle VA follow the preceding vehicle (vehicle to be followed) while maintaining the inter-vehicle distance Dset between vehicle VA and the preceding vehicle (vehicle to be followed) without the need to operate the accelerator pedal or brake pedal. The vehicle to be followed is a vehicle traveling in the area ahead of vehicle VA and immediately before vehicle VA.

One example of steering control is lane keeping assist control (hereinafter simply referred to as "LKA"). LKA itself is publicly known (see, for example, JP 2008-195402 A, JP 2009-190464 A, JP 2010-6279 A, and Japanese Patent No. 4349210, etc.).

LKA is a control that applies automatic steering torque to the steering mechanism 42 to change the steering angle of the steering wheels of the vehicle VA so that the vehicle VA travels along a target driving line that is set using either the marking line or the driving trajectory of the preceding vehicle (i.e., the preceding vehicle trajectory), or both.

The ECU 10 can execute the automatic driving control under a specific situation. The ECU 10 executes the automatic driving control when a predetermined automatic driving execution condition is satisfied. For example, the automatic driving execution condition is satisfied when all of the following conditions A1 to A3 are satisfied.
(Condition A1): The vehicle VA is traveling on a highway.
(Condition A2): The vehicle speed SPD is equal to or less than a predetermined speed threshold SPDth.
(Condition A3): The driver presses the automatic driving switch 81 to set the operation state of the automatic driving control from the OFF state to the ON state.

The speed threshold SPDth may be set to a relatively small value. In this case, the situation in which the autonomous driving execution condition is met means that the vehicle VA is traveling in a congested section. In such a situation, the ECU 10 may execute autonomous driving control. Note that the autonomous driving execution condition is not limited to the above example, and may be other conditions.

When a predetermined autonomous driving end condition is satisfied, the ECU 10 ends the autonomous driving control. For example, the autonomous driving end condition is satisfied when at least one of the following conditions B1 to B4 is satisfied.
(Condition B1): The vehicle VA has arrived at the exit of a highway.
(Condition B2): The vehicle speed SPD becomes greater than a speed threshold SPDth.
(Condition B3): The driver presses the automatic driving switch 81 to change the operation state of the automatic driving control from the on state to the off state.
(Condition B4): An event (e.g., a system error) has occurred that makes it impossible to continue automatic driving control.
The autonomous driving end condition is not limited to the above example, and may be another condition.

(Mobile terminal configuration)
The mobile terminal 110 is a device carried by the occupant, and in this example, is a smartphone as shown in Fig. 2. The mobile terminal 110 may be another device (for example, a tablet terminal or a wearable terminal) as long as it is carried by the occupant.

The mobile terminal 110 includes a display 201. The display 201 is a touch panel display, and is capable of detecting touch operations on the display surface (operation surface) of the display 201.

The mobile terminal 110 is provided with a front camera (second imaging unit) 202 on the surface facing the display 201. The mobile terminal 110 may also be provided with a rear camera on the surface opposite the display 201 (i.e., the back surface). Furthermore, the mobile terminal 110 is provided with an illumination unit 203 (e.g., an LED) on the surface facing the display 201. The illumination unit 203 is configured to emit light.

Furthermore, the mobile terminal 110 includes a communication unit 204 therein. The communication unit 204 is a communication module for communicating with an external device via short-range wireless communication. Short-range wireless communication includes various wireless communication such as Bluetooth (registered trademark), wireless ID tags (RFID: Radio Frequency Identification), near-field wireless communication (NFC: Near Field Communication), and WiFi.

As shown in FIG. 3, the mobile terminal 110 includes a CPU 301, a memory 302, and a non-volatile memory 303. The CPU 301, the memory 302, the non-volatile memory 303, the display 201, the front camera 202, the illumination unit 203, and the communication unit 204 are connected to each other via an internal bus 310.

The CPU 301 includes at least one processor or circuit. The memory 302 includes, for example, a RAM. The non-volatile memory 303 includes, for example, a flash memory and a ROM. The CPU 301 uses the memory 302 as a working memory to execute instructions (routines, programs) stored in the non-volatile memory 303.

(Outline of wakefulness state determination process)
In level 3 autonomous driving control, when the ECU 10 cannot continue the autonomous driving control (when a request to take over driving operation occurs), the driver must take over the driving operation. Therefore, the driver is not allowed to be dozing off. On the other hand, the driver is allowed to be distracted. Therefore, while the ECU 10 is executing level 3 autonomous driving control, the driver can perform various actions (such as operating a mobile device or reading a book).

For example, as shown in FIG. 4, there are cases where the driver DR operates the mobile terminal 110 while the ECU 10 is executing the automatic driving control. In such a situation, the ECU 10 cooperates with the mobile terminal 110 to provide a specific service. In this example, the ECU 10 provides a service of determining whether the driver DR is in an awake state so that the driver can immediately take over driving operations when a request for a change of driving operations occurs. If the driver DR is not in an awake state, the ECU 10 issues a warning to the driver. This allows the driver DR to return to an awake state.

Specifically, while the automatic driving control is being executed, the ECU 10 uses information acquired by the mobile terminal 110 to determine whether the driver DR is in an awake state. Hereinafter, such processing is referred to as "awake state determination processing." Furthermore, the information acquired by the mobile terminal 110 is referred to as "terminal side information."

For the above configuration, as shown in FIG. 3, the non-volatile memory 303 of the mobile terminal 110 stores a driver monitoring application 311. The driver monitoring application 311 is a program for monitoring the state of the driver DR.

The CPU 301 executes the driver monitoring application 311 to acquire information to be used in the wakefulness determination process, and transmits the acquired information to the automated driving device 100 as terminal-side information. Specifically, the CPU 301 acquires image data (hereinafter referred to as the "second image") from the front camera 202. The second image includes the face of the driver DR. The CPU 301 transmits the second image to the automated driving device 100 as terminal-side information.

The ECU 10 receives the terminal side information via the communication unit 80. The ECU 10 executes an awakening state determination process based on the second image. Specifically, the ECU 10 determines the degree of awakening of the driver DR according to a known method. In this example, the degree of awakening includes a first degree, a second degree, and a third degree. The first degree indicates that the driver DR is in an awakening state. The second degree indicates that the driver DR is not in a drowsy state, but is feeling drowsy. The third degree indicates that the driver DR is in a drowsy state.

For example, the degree of wakefulness is determined based on at least one of the following: eye closure rate, blinking frequency, eye open state (distance between upper and lower eyelids), eye movement, and the presence or absence of yawning behavior. The eye closure rate is briefly explained below.

The ECU 10 extracts the eyes of the driver DR from the second image by inputting the second image to a classifier for eye detection or by performing template matching on the second image. The ECU 10 determines whether the eyes are closed, for example, based on the aspect ratio of the extracted eye portion. The ECU 10 calculates the proportion of time that the eyes are closed within a specified time as the eye closure rate. If the eye closure rate is equal to or less than a first threshold, the ECU 10 determines that the driver DR has a first degree of wakefulness. If the eye closure rate is greater than the first threshold and equal to or less than a second threshold, the ECU 10 determines that the driver DR has a second degree of wakefulness. If the eye closure rate is greater than the second threshold, the ECU 10 determines that the driver DR has a third degree of wakefulness.

When the ECU 10 determines that the driver DR is at the third level of wakefulness, it causes the notification device 51 to execute a predetermined attention-calling process. Specifically, the ECU 10 uses the meter ECU 50 to sound a buzzer and cause the display device to display a warning lamp.

The terminal-side information used in the wakefulness state determination process is not limited to the second image. The CPU 301 may obtain operation information on the mobile terminal 110 (touch operations on the display 201 and/or operations on various buttons) and/or acceleration information on the mobile terminal 110 as terminal-side information and transmit it to the automatic driving device 100. The ECU 10 may use this information to execute the wakefulness state determination process.

(Overview of operation)
There may be passengers other than the driver DR in the vehicle VA. For example, assume that there is a passenger in the back seat. Hereinafter, the mobile terminal 110 carried by the driver DR may be referred to as the "first mobile terminal 110A," and the mobile terminal 110 carried by the passenger in the back seat may be referred to as the "second mobile terminal 110B." When there is no need to distinguish between the first mobile terminal 110A and the second mobile terminal 110B, they will be collectively referred to as the "mobile terminal 110."

It is assumed that the first mobile terminal 110A and the second mobile terminal 110B are each connected to the ECU 10 via wireless communication. When the ECU 10 executes the wakefulness state determination process in this situation, the ECU 10 needs to determine (identify) which mobile terminal is the first mobile terminal 110A of the driver DR.

Therefore, the ECU 10 executes a comparison process that compares the first image acquired by the driver monitor camera 75 with the second image acquired by the front camera 202 of the mobile terminal 110, thereby determining which mobile terminal is the first mobile terminal 110A of the driver DR. Hereinafter, this type of process is referred to as the "terminal determination process."

In this example, the comparison process is a process of comparing a face recognized in the first image with a face recognized in the second image.

For the above configuration, as shown in FIG. 3, the non-volatile memory 303 of the mobile terminal 110 stores a terminal determination application 312. The terminal determination application 312 is a program for acquiring the second image.

The ECU 10 acquires a first image from the driver monitor camera 75. Meanwhile, the CPU 301 of the mobile terminal 110 executes the terminal determination application 312 and transmits the second image to the automated driving device 100 as terminal-side information. The ECU 10 receives the second image as terminal-side information.

The ECU 10 executes the terminal determination process as follows. The ECU 10 extracts characteristic elements (feature points) of the face included in the first image. In this example, the characteristic elements include the positional relationship of the center points and/or end points of the parts that make up the face. The parts that make up the face include the eyes, eyebrows, nose, mouth, etc., and are hereinafter simply referred to as "facial parts". In another example, the characteristic elements may include the size of the facial part and/or the aspect ratio of the facial part. In yet another example, the characteristic element may be the length of a straight line connecting the center points and/or end points of the facial part, or the angle formed by two straight lines extracted from those straight lines. Similarly, the ECU 10 extracts characteristic elements of the face included in the second image.

As a comparison process, the ECU 10 compares the characteristic element extracted from the first image (hereinafter referred to as the "first characteristic element") with the characteristic element extracted from the second image (hereinafter referred to as the "second characteristic element"). If the degree of match between the first characteristic element and the second characteristic element is greater than a predetermined degree, the ECU 10 determines that the mobile terminal that transmitted the second image is the first mobile terminal 110A of the driver DR.

For example, assume that the first image is the first image 500 shown in FIG. 5, and the second image is the second image 600 shown in FIG. 6. The second image 600 is an image acquired by the first mobile terminal 110A. In this case, since the degree of match between the first characteristic element and the second characteristic element is greater than a predetermined degree, the ECU 10 determines that the mobile terminal that transmitted the second image 600 is the first mobile terminal 110A of the driver DR. After determining the first mobile terminal 110A of the driver DR in this manner, the ECU 10 continues to acquire terminal side information (second image) from the first mobile terminal 110A and executes the wakefulness state determination process.

On the other hand, assume that the second mobile terminal 110B of the passenger in the back seat transmits a second image to the automatic driving device 100. In this case, the degree of match between the first characteristic element and the second characteristic element is below a predetermined level, so the ECU 10 determines that the mobile terminal that transmitted the second image is not the first mobile terminal 110A of the driver DR. After this determination, the ECU 10 does not acquire terminal side information (second image) from the second mobile terminal 110B.

(Example of operation)
A specific process flow will be described with reference to Fig. 7. As in the above, an example in which a driver is present in the driver's seat and passengers are present in the rear seats will be described.

The driver DR sits in the driver's seat of the vehicle VA, and a passenger sits in the back seat of the vehicle VA. Short-range wireless communication is started between the ECU 10 and the first mobile terminal 110A. In response, the first mobile terminal 110A launches the terminal determination application 312 (step 701). Furthermore, short-range wireless communication is started between the ECU 10 and the second mobile terminal 110B. In response, the second mobile terminal 110B launches the terminal determination application 312 (step 701).

The first mobile terminal 110A transmits the terminal side information (second image) to the ECU 10 (step 702). Furthermore, the second mobile terminal 110B transmits the terminal side information (second image) to the ECU 10 (step 703).

The ECU 10 executes the terminal determination process as described above (step 704). Specifically, the ECU 10 acquires a first image from the driver monitor camera 75. The ECU 10 executes a comparison process to compare the first characteristic element extracted from the first image with the second characteristic element extracted from the second image of the first mobile terminal 110A. Furthermore, the ECU 10 executes a comparison process to compare the first characteristic element extracted from the first image with the second characteristic element extracted from the second image of the second mobile terminal 110B. As a result, the ECU 10 determines that the first mobile terminal 110A is the mobile terminal of the driver DR. On the other hand, the ECU 10 determines that the second mobile terminal 110B is not the mobile terminal of the driver DR.

Then, the driver DR drives the vehicle VA. The vehicle VA is traveling on a motorway at a vehicle speed SPD equal to or less than the speed threshold SPDth. In this situation, the driver DR presses the automatic driving switch 81. The ECU 10 determines that the automatic driving execution conditions are met and starts automatic driving control (level 3) (step 705). The ECU 10 transmits a start instruction signal only to the first mobile terminal 110A (step 706). The start instruction signal is a signal that instructs the start of the driver monitoring application 311. Since the second mobile terminal 110B is not the mobile terminal of the driver DR, the ECU 10 does not transmit the start instruction signal to the second mobile terminal 110B. In response to the start instruction signal, the first mobile terminal 110A starts the driver monitoring application 311 (step 707).

The first mobile terminal 110A transmits the terminal side information (second image) to the ECU 10 every time the predetermined time dT elapses (step 708). The ECU 10 executes the wakefulness state determination process using the terminal side information (second image) every time the predetermined time dT elapses (step 709). The processes of steps 708 and 709 are repeatedly executed until the automatic driving control ends.

Then, the autonomous driving termination condition described above is satisfied. The ECU 10 terminates the autonomous driving control (step 710). Then, the ECU 10 transmits an end instruction signal to the first mobile terminal 110A (step 711). The end instruction signal is a signal that instructs the driver monitoring application 311 to terminate. In response to the end instruction signal, the first mobile terminal 110A terminates the driver monitoring application 311 (step 712).

The mobile terminal determination system having the above configuration can determine (identify) which mobile terminal is the first mobile terminal 110A of the driver DR. This allows the ECU 10 to provide various services (awake state determination processing and attention calling processing) to the driver DR.

The comparison process may be a process of comparing the occupant's behavior recognized in the first image with the occupant's behavior recognized in the second image. Here, "occupant's behavior" refers to the orientation and/or movement of the occupant's upper body. "Occupant's behavior" includes the orientation and movement of the face, as well as the orientation and movement of facial parts. An example in which "facial orientation" is used as occupant's behavior will be described. If the facial orientation of the occupant recognized in the first image matches the facial orientation of the occupant recognized in the second image, the ECU 10 determines that the mobile terminal that transmitted the second image is the first mobile terminal 110A of the driver DR.

The comparison process may be a process of comparing the change in luminance in the first image with the change in luminance in the second image. In this configuration, an illumination unit 76 (e.g., an LED, see FIG. 4) is provided near the driver monitor camera 75 and facing the driver's seat. The illumination unit 76 is configured to emit light toward the driver's seat. When the ECU 10 starts the terminal determination process, it activates the illumination unit 76. This causes the face of the driver DR to be illuminated by the light. As the luminance in the first image increases, the luminance in the second image of the first mobile terminal 110A also increases. Then, both the luminance in the first image and the luminance in the second image of the first mobile terminal 110A fall within a predetermined luminance range. Taking this into consideration, when both the luminance in the first image and the luminance in the second image fall within a predetermined luminance range, the ECU 10 determines that the mobile terminal that transmitted the second image is the first mobile terminal 110A of the driver DR.

On the other hand, the passenger in the back seat carrying the second mobile terminal 110B is not illuminated by light, and so the luminance in the second image of the second mobile terminal 110B is low. That is, although the luminance in the first image is within the predetermined luminance range, the luminance in the second image of the second mobile terminal 110B is lower than the predetermined luminance range. Taking this into consideration, if the luminance in the second image is outside the predetermined luminance range, the ECU 10 determines that the mobile terminal that transmitted the second image is not the first mobile terminal 110A of the driver DR.

In addition, in the terminal determination process, the lighting unit 203 of the first mobile terminal 110A and the lighting unit 203 of the second mobile terminal 110B may be activated. In this case, the ECU 10 transmits instructions to the first mobile terminal 110A and the second mobile terminal 110B so that the light intensity of the lighting unit 203 of the first mobile terminal 110A differs from the light intensity of the lighting unit 203 of the second mobile terminal 110B. This allows the ECU 10 to determine which mobile terminal is the first mobile terminal 110A of the driver DR based on the difference between the change in luminance in the first image and the change in luminance in the second image.

Two or more of the above comparison processes may be combined as appropriate.

The comparison process may be a process of comparing the second image with a preset reference image. The reference image is the first image acquired by the driver monitor camera 75 at any timing before the terminal determination process is executed.

A combination of multiple occupants using the vehicle VA and an identifier that identifies each occupant's mobile device may be registered in the non-volatile memory 10e. In this case, the non-volatile memory 10e may further include information on the driving schedule of each occupant. The ECU 10 may determine which occupant is sitting in the driver's seat according to the driving schedule, and determine that the mobile device corresponding to that occupant is the driver's mobile device.

Second Embodiment
Next, a mobile terminal determination system according to the second embodiment will be described. In this embodiment, the terminal determination application 312 acquires information on the communication status between a first communication unit mounted on the vehicle VA and the communication unit 204 of the mobile terminal 110. The mobile terminal 110 transmits the information on the communication status to the ECU 10. The ECU 10 determines which mobile terminal is the first mobile terminal 110A of the driver DR based on the information on the communication status.

In this example, the first communication unit is a short-range communication unit 401 (see FIG. 4) installed near the driver's seat. The short-range communication unit 401 is built into the steering wheel SW. The short-range communication unit 401 is a device that can perform wireless communication by NFC with a mobile terminal. NFC is a wireless communication technology that uses a frequency band of 13.56 MHz, and its communication range is about 10 cm. It is considered that the first mobile terminal 110A of the driver DR is highly likely to be within the communication range of NFC. The mobile terminal 110 transmits information about the communication status between the short-range communication unit 401 and the communication unit 204 (i.e., information indicating whether wireless communication is established) to the ECU 10. The ECU 10 determines that the mobile terminal with which wireless communication is established with the short-range communication unit 401 is the first mobile terminal 110A of the driver DR.

On the other hand, the second mobile terminal 110B carried by the passenger in the back seat is not within the communication distance range of NFC. The ECU 10 can determine that a mobile terminal with which wireless communication has not been established with the short-range communication unit 401 is not the first mobile terminal 110A of the driver DR.

The mobile terminal determination system having the above configuration can determine (identify) which mobile terminal is the first mobile terminal 110A of the driver DR based on information about the communication status between the first communication unit (short-range communication unit 401) and the mobile terminal 110.

The first communication unit may include multiple short-range communication units installed at a distance within the vehicle VA. In this case, the multiple short-range communication units include at least a first short-range communication unit and a second short-range communication unit. The first short-range communication unit and the second short-range communication unit are each devices capable of wireless communication with the mobile terminal 110 using Bluetooth (registered trademark). The first short-range communication unit is installed near the driver's seat, and the second short-range communication unit is installed in the rear seat. The first short-range communication unit and the second short-range communication unit each measure the reception strength of a signal transmitted from the communication unit 204 of the mobile terminal 110. The first short-range communication unit and the second short-range communication unit each transmit information regarding the reception strength to the ECU 10. The ECU 10 determines which mobile terminal is the first mobile terminal 110A of the driver DR based on the difference in reception strength. For example, when the first short-range communication unit and the second short-range communication unit each receive a signal from the communication unit 204 of the first mobile terminal 110A, the reception strength of the signal received at the first short-range communication unit is high and the reception strength of the signal received at the second short-range communication unit is low. On the other hand, when the first short-range communication unit and the second short-range communication unit each receive a signal from the communication unit 204 of the second mobile terminal 110B, the reception strength of the signal received at the first short-range communication unit is low and the reception strength of the signal received at the second short-range communication unit is high. Based on the difference in reception strength as described above, the ECU 10 can determine which mobile terminal is the first mobile terminal 110A of the driver DR.

Note that this disclosure is not limited to the above-described embodiments, and various modifications may be made within the scope of this disclosure.

Another ECU may be disposed between the ECU 10 and the mobile terminal 110. The ECU may execute the terminal determination process and the wakefulness state determination process.

The ECU 10 may transmit the first image to the first mobile terminal 110A and the second mobile terminal 110B. In this case, each of the first mobile terminal 110A and the second mobile terminal 110B executes a terminal determination process. Each of the first mobile terminal 110A and the second mobile terminal 110B may transmit a determination result of the terminal determination process to the autonomous driving device 100 (i.e., the ECU 10).

The terminal determination process may be applied to services other than the awake state determination process. Therefore, the timing at which the terminal determination process is executed is not limited to before the execution of the autonomous driving control, but may be before the start of the service to be implemented.

10...Automatic driving ECU, 20...Engine ECU, 30...Brake ECU, 40...Steering ECU, 50...Meter ECU, 60...Navigation ECU, 100...Automatic driving device, 110...Mobile terminal.

Claims (1)

A first imaging unit mounted in a vehicle and configured to capture an image of a driver of the vehicle;
A portable terminal including a second imaging unit;
a control device configured to execute a comparison process of comparing a first image acquired by the first imaging unit with a second image acquired by the second imaging unit, thereby determining whether the mobile terminal is a mobile terminal of the driver; and
A mobile terminal determination system comprising:
Further, a lighting unit that emits light toward a driver's seat of the vehicle is provided,
The control device includes:
Activating the lighting unit to illuminate the face of the driver;
The comparison process is configured to compare a luminance in the first image with a luminance in the second image, and determine whether or not both the luminance in the first image and the luminance in the second image are within a predetermined luminance range, thereby determining whether or not the mobile terminal is a mobile terminal of the driver.
Mobile terminal identification system.

Images