JP7717519B2 - Agent Device - Google Patents

Description

The present invention relates to an agent device capable of providing notifications using personified characters.

BACKGROUND ART Conventionally, various methods have been proposed as methods for alerting occupants (particularly the driver) in a driving environment of a vehicle such as an automobile.
As one such technique, an agent device or driving assistance device has been proposed in which an occupant and an anthropomorphized character (a so-called agent) interact with each other to provide information on driving assistance in response to the occupant's requests.

For example, in conventional driving assistance devices, it is thought that notifications of danger avoidance actions using voice or other means are not intuitively conveyed, so one proposal is to place a small robot on the dashboard as an anthropomorphized agent, and have this agent simulate the danger avoidance actions that the driver should take (see Patent Document 1).

JP 2018-032204 A

However, conventional agents were impersonal or appeared suddenly to draw attention, making it difficult to build a relationship of trust between the agent and the driver. Also, even if the agent suddenly took the same action as an evasive action, it was difficult for the driver to take that action immediately if they did not understand the cause of the evasive action, and there was a problem that evasive action would not be taken unless a relationship of trust was established.
Furthermore, despite the advances in the development of autonomous driving in recent years, it is difficult to tell whether the autonomous driving system is operating or not. For example, there are some systems that have a small light that comes on during autonomous driving and goes off when the vehicle switches to manual driving. However, this can often be overlooked, and if the driver shifts their gaze to this small light, they may become distracted from the road ahead. Furthermore, in the case of autonomous driving at level -2 or other levels, in an emergency, driving authority is suddenly handed over from the autonomous driving system to the driver. However, many drivers honestly feel confused when they are suddenly asked to drive.

The present invention was made to solve these conventional problems, and aims to provide an agent device that can clearly communicate whether the vehicle is being driven manually or automatically, and support safety while driving.

The agent device of the present invention comprises a shift position detection unit that detects the shift position; a surrounding condition acquisition unit that acquires the surrounding conditions; a driving state switching unit that switches between manual driving and autonomous driving based on the surrounding conditions acquired by the surrounding condition acquisition unit or on instructions from the driver; an autonomous driving determination unit that determines whether the vehicle is in an autonomous driving state; and an agent control unit that controls a personified agent, wherein the agent control unit switches the agent between a standing position and a seated position when switching between manual driving and autonomous driving is performed.

The present invention provides an agent device that can clearly communicate whether the vehicle is being driven manually or automatically, and support safety while driving.

1 is a perspective view showing an interior of a vehicle according to an embodiment of the present invention; FIG. 2 is a front view showing the interior of the vehicle in front of the driver's seat. FIG. 10 is a front view showing the placement of agents. FIG. 2 is a schematic diagram showing a configuration of an agent according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an agent device. 10 is a main flowchart of an agent control process performed by a control unit of the agent device. 10 is a flowchart showing a process performed by a control unit of the agent device during driving. 10 is a flowchart showing a process performed by a control unit of the agent device when switching to automatic driving. 10A and 10B are diagrams illustrating examples of the standing and sitting postures of an agent. 10 is a flowchart showing a process during automatic driving performed by a control unit of the agent device. FIG. 10 is an operational diagram showing an example of a deceleration instruction operation by an agent. 10A and 10B are diagrams illustrating an example of an agent's guidance operation for guiding an approaching object or the like. 10A and 10B are diagrams illustrating an example of an agent's guidance operation around a corner or the like. 10 is a flowchart showing an autonomous driving end process performed by a control unit of the agent device.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
In the following description, the same reference numerals in different drawings indicate parts with the same functions, and duplicated descriptions in each drawing will be omitted as appropriate.

(Vehicle interior configuration)
1 and 2, a vehicle 1 has an instrument panel 3 provided in front of a driver's seat 2, and a steering wheel 4 disposed between the driver's seat 2 and the instrument panel 3. The steering wheel 4 is rotatably attached to a steering column (not shown) via a steering shaft (not shown). An airbag 8 is stored inside the steering wheel 4 and is deployed toward the driver H in the event of a collision or the like of the vehicle 1.
In the present embodiment, the agent device 100 (described later) and the agent 200 (described later) are described based on dialogue and gestures (movements) with the driver H, but similar actions can also be taken with other passengers.

As shown in FIG. 1, two display panels, a first display panel 30 and a second display panel 40, are provided inside the vehicle 1. The first display panel 30 is located on the instrument panel 3 in front of the driver's seat 2, and the second display panel 40 is located on the instrument panel 3 to the left and front of the driver's seat 2.

(First display panel 30)
1 and 2, the first display panel 30 is configured to include a pointer-type meter configured like an analog clock and a so-called liquid crystal display device in which a liquid crystal panel and a backlight are integrally provided. The driver H can view various information displayed in the first display area 30a of the first display panel 30 through the space 4a above the steering wheel 4.

As shown in FIG. 2, the first display panel 30 has a first display area 30a. On the left and right sides of this first display area 30a are two pointer-type meters that display information such as the vehicle 1's traveling speed (speedometer) and the engine's revolutions per unit time (tachometer), and between the two pointer-type meters, in the center of the first display panel 30, is a small liquid crystal display device that displays images showing general vehicle information. Note that the first display panel 30 may not have a pointer-type meter and may instead be configured entirely with a single liquid crystal display device.

(Second display panel 40)
1, the second display panel 40 is configured as a so-called liquid crystal display device, in which a liquid crystal panel and a backlight are integrally provided, for example. Note that the second display panel 40 displays map information and the like, and serves as a so-called car navigation system.

The liquid crystal display devices configured in the first display panel 30 and the second display panel 40 may also be configured as self-luminous display devices such as plasma displays or organic EL displays, or display devices such as projection projectors.

(Agent 200)
As shown in FIG. 3, an agent 200 is disposed on the upper part of the instrument panel 3 in front of the driver's seat 2 .
The agent 200 may be located anywhere as long as it is within the peripheral vision of the driver H, and does not necessarily have to be located directly in front of the driver's seat. It is desirable that the agent 200 be located within the peripheral vision of the driver H, but in a position that does not obstruct the driver H's view of the outside of the vehicle. The location of the agent 200 may also be made changeable using a predetermined operating device or voice instructions, etc.

Agent 200 is an anthropomorphized character, a three-dimensional object with a humanoid external shape used for notification and interaction. In this embodiment, agent 200 is a real three-dimensional object in the shape of a humanoid, but it is not limited to this. It may also be a character displayed on a display device such as a monitor, a three-dimensional object represented by a hologram, a virtual three-dimensional object, etc.

As shown in FIG. 4, the agent 200 includes a head 211, a right eye 212R, a left eye 212L, a neck 213, a chest 214, a waist 215, a buttocks 216, and a right shoulder 2. 21R, left shoulder 221L, right upper arm 222R, left upper arm 222L, right elbow 223R, left elbow 223L, right forearm 224R, left forearm 224L, right wrist 22 5R, left wrist 225L, right hand 226R, left hand 226L, right hip 231R, left hip 231L, right thigh 232R, left thigh 232L, right knee 2 33R, a left knee part 233L, a right lower leg part 234R, a left lower leg part 234L, a right ankle part 235R, a left ankle part 235L, a right foot part 236R, and a left foot part 236L.

The agent 200 is also provided with a support connector 240, which supports the agent 200 so that it can be placed upright on the instrument panel 3.
Incidentally, ears are provided on both sides of the head 211 of the agent 200. This makes it appear as if the agent 200 is listening carefully to what the driver H is saying. Even if these ears are not actually provided, as will be described later, when the agent 200 is listening to what the driver H is saying, the agent 200 can make it appear as if he is listening with his hands on his ears by placing his right hand 226R or left hand 226L on the side of the head 211 of the agent 200, thereby deepening the relationship of trust between the agent 200 and the driver H.

Note that Figure 4(a) is a diagram showing the basic posture of the agent 200 when it is facing the driver H (the rear side of the vehicle 1), and Figure 4(b) is a diagram showing the basic posture of the agent 200 when it is facing away from the driver H (facing the front side of the vehicle 1).
In addition, in the basic posture, the agent 200 stands with the right palm portion 226Ra of the right hand portion 226R and the left palm portion 226La of the left hand portion 226L facing toward the inside of the agent 200, and the right back portion 226Rb of the right hand portion 226R and the left back portion 226Lb of the left hand portion 226L facing toward the outside of the agent 200.
As described above, the agent 200 can freely rotate between a front posture facing the driver H and a rear posture facing away from the driver H. In other words, the agent 200 may be one in which the agent 200 itself rotates around the support connector 240, or the agent 200 may be placed on a rotating body and rotated from the front posture to the rear posture and from the rear posture to the front posture by rotating the rotating body.
The agent 200 can also take a standing posture as shown in Figure 4 and Figure 9(a) and a sitting posture as shown in Figure 9(b). The details of the sitting posture will be described later.

The agent 200 also includes an agent control unit 190, which will be described later, that controls and operates each joint of the agent 200, allowing the agent 200 to assume a predetermined posture. The agent control unit 190 is wirelessly connected to a control unit 110 (ECU), which will be described later, and operates the agent 200 under the control of the control unit 110. Note that in this embodiment, the agent control unit 190 and the control unit 110 are connected wirelessly, but they may also be connected by wire, for example, by passing a communication line through the support connector 240.

An agent speaker 191, which will be described later, is provided in the head 211 of the agent 200, and audio is output under the control of the agent control unit 190. Note that this agent speaker 191 is not limited to being located in the head 211 of the agent 200, but may be provided in another part of the agent 200 or in another location on the vehicle 1. However, providing the agent speaker 191 in the head 211 of the agent 200 allows the driver H to perceive the speech from the agent 200 more realistically, makes it easier for the driver to empathize with the agent, and greatly contributes to building the driver H's trust in the agent 200.

Next, the configuration of the agent device 100 of the present invention will be described with reference to the block diagram in Figure 5.

As shown in FIG. 5, the agent device 100 includes a control unit 110, a surrounding environment recognition unit 120, an occupant state recognition unit 130, a vehicle state recognition unit 140, a speaker 150, a microphone 160, a memory unit 170, a transceiver 180, an agent control unit 190, an agent speaker 191, and a driving state switching operation unit 310.

Note that the agent device 100 shown in FIG. 5 is merely an example, and the components of the agent device 100 can be modified as appropriate. For example, the agent device 100 of the present invention can be realized with only either the speaker 150 or the agent speaker 191. Furthermore, an agent microphone may be provided within the agent 200 in addition to the microphone 160, or the agent device 100 of the present invention can be realized with only an agent microphone provided within the agent 200 instead of the microphone 160. Furthermore, in this embodiment, both the control unit 110 and the agent control unit 190 are provided, but this is not limiting, and it is also possible to have only one of the control unit 110 or the agent control unit 190 and have it perform the functions of both.

(Control unit 110)
The control unit 110 is equipped with a CPU, ROM, RAM (e.g., a ring buffer), EEPROM, input/output ports, etc., which are not shown in the figure, and for example, when information is input through the input port, it controls various devices via the output port based on a control program read from the ROM.

The control unit 110 is also configured to switch the driving state between manual driving and automatic driving based on information input from the surrounding environment recognition unit 120 and the vehicle state recognition unit 140. The control unit 110 may also cancel the switch to automatic driving depending on information input from the occupant state recognition unit 130. The control unit 110 is also configured to switch the driving state between manual driving and automatic driving by a switching operation input to the driving state switching operation unit 310.
Here, when switched to manual driving, the control unit 110 controls the driving of the vehicle 1 based on the input operation of the driver H, and when switched to automatic driving, the control unit 110 controls the driving of the vehicle 1 based on information input from the surrounding environment recognition unit 120 and the vehicle state recognition unit 140, etc.

The ROM of the control unit 110 stores a data table (not shown) of the actions and lines to be performed by the agent 200. Furthermore, the agent 200 may be equipped with a learning function such as AI, allowing it to perform actions and lines other than those stored. The CPU of the control unit 110 determines the actions and lines to be performed by the agent 200 from the data table based on information acquired from each recognition unit (described below) and information acquired from the microphone 160. For example, when an utterance from driver H is acquired from the microphone 160, a "nod" gesture is determined from the data table, and when driver H has finished speaking, a line such as "I understand" is determined. The data table of the actions and lines to be performed by the agent 200 may be stored in the ROM of the agent control unit 190.

(Surrounding environment recognition unit 120)
The surrounding environment recognition unit 120 is provided to recognize the surrounding environment of the vehicle 1 (host vehicle). The surrounding environment recognition unit 120 also includes an exterior camera 120a, a radar 120b, a temperature sensor 120c, a weather sensor 120d, and a navigation device 121, and these devices enable the surrounding environment and surrounding conditions of the vehicle 1 to be recognized. For example, the surrounding environment recognition unit 120 can detect the presence of corners or turning points in the traveling direction, the presence or absence of approaching objects, etc.
In this embodiment, the navigation device 121 is configured to be part of the surrounding environment recognition unit 120, but this is not limited to this, and the navigation device 121 may be configured as a standalone device separate from the surrounding environment recognition unit 120.

(Exterior camera 120a)
The exterior camera 120a is attached to, for example, a rearview mirror (not shown) and is capable of capturing images of the area in front of and behind the vehicle 1. The captured image information is input to the control unit 110, which then stores the image information in RAM. This allows the control unit 110 to recognize the conditions in front of and behind the vehicle 1 in real time and after the fact.

(Radar 120b)
The radar 120b is, for example, a millimeter-wave radar that detects obstacles and the like by emitting radio waves. The millimeter-wave radar is attached to the front bumper or rear bumper of the vehicle 1, and is capable of monitoring the area ahead of the vehicle 1, the areas in front of the vehicle 1, and the areas behind the vehicle 1. The monitoring information is input to the control unit 110, which stores the monitoring information in RAM. This allows the control unit 110 to recognize the situation ahead of the vehicle 1, the areas in front of the vehicle 1, and the areas behind the vehicle 1 in real time and after the fact. Note that although a millimeter-wave radar is used in this embodiment, other radars may also be used. For example, infrared radar may be used.

(Air temperature sensor 120c)
The air temperature sensor 120c is attached to, for example, the rear of the vehicle 1 and is capable of detecting the air temperature outside the vehicle 1. The detected air temperature is input to the control unit 110, which then stores the air temperature in RAM. This allows the control unit 110 to recognize the air temperature outside the vehicle 1.

(Weather sensor 120d)
The weather sensor 120d is attached, for example, to the top of the vehicle 1 and is capable of detecting raindrops and the like falling on the vehicle 1. The detected water droplet information is input to the control unit 110, which then stores the water droplet information in RAM. This allows the control unit 110 to recognize the weather conditions outside the vehicle 1.

(Navigation device 121)
The navigation device 121 provides route guidance according to the destination, waypoints, etc. The navigation device 121 also has map information, a GPS, a VICS receiver, etc., and is capable of acquiring map information on the current position of the vehicle 1, surrounding areas, traffic congestion information, road regulation information, parking lot information, etc. Therefore, the control unit 110 can recognize the surrounding conditions of the vehicle 1 in real time based on information acquired by the navigation device 121, the exterior camera 120a, the radar 120b, etc.
The navigation device 121 can also receive map information at any time, enabling it to acquire the latest map information. The transmitter/receiver of the navigation device 121 may also serve as the transceiver 180, which will be described later.

In this way, the control unit 110 can recognize the surrounding environment and circumstances of the vehicle 1 in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the surrounding environment and circumstances of the vehicle 1, and notify the driver H of information about the surrounding environment and circumstances of the vehicle 1. For example, if there is a fallen object in front of the vehicle 1, the control unit 110 can notify the driver H by gesture and voice, such as "There is a fallen object ahead." This can improve safety.

In this embodiment, the surrounding environment recognition unit 120 includes an exterior vehicle camera 120a, radar 120b, temperature sensor 120c, weather sensor 120d, and navigation device 121, but these are merely examples, and other devices may of course be used.

(Occupant state recognition unit 130)
The occupant state recognition unit 130 is provided to recognize the state of the driver H. The occupant state recognition unit 130 also includes an occupant image capturing camera 130a and a vital sign sensor 130b, and is able to recognize the state of the driver H using these devices.

(Occupant photographing camera 130a)
The occupant photographing camera 130a is attached to, for example, the instrument panel 3, and is capable of photographing the driver H. The photographed image information is input to the control unit 110, which stores the image information in RAM. This allows the control unit 110 to recognize the state of the driver H in real time and after the fact. Note that the state of the driver H referred to here specifically includes the state of the driver H's eyelids, the number of blinks, the direction of the driver's gaze, the direction of the face, whether the driver is crouched, or whether the driver is unable to move, etc.

(Vital sensor 130b)
The vital sensor 130b is attached, for example, to a part of the steering wheel 4 that is gripped by the driver H, and is capable of acquiring vital information such as the heart rate and blood pressure of the driver H. The acquired vital information is then input to the control unit 110, which then stores the vital information in RAM. This allows the control unit 110 to recognize the condition of the driver H in real time and after the fact.

In this way, the control unit 110 can recognize the state of driver H in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of driver H's state, and notify driver H of specified information. For example, it can notify driver H by image and sound, such as "Your eyelids are drooping. Would you like to take a break?" or "Your heart rate is faster than usual. Would you like to take a break?" This leads to improved safety.

The occupant state recognition unit 130 can recognize driver H's thoughts and emotions to a certain extent based on information acquired from the occupant image capture camera 130a and vital sensor 130b, and information input from the microphone 160. For example, it can acquire driver H's facial expression from the occupant image capture camera 130a, acquire driver H's heart rate and blood pressure from the vital sensor 130b, and acquire voice volume and input content from the microphone 160, and from this acquired information it can recognize whether driver H is thinking or feeling normal thoughts and emotions or abnormal thoughts and emotions (for example, surprised, angry, etc.).

In this embodiment, the occupant state recognition unit 130 is described as comprising an occupant image capturing camera 130a and a vital sign sensor 130b, but this is merely an example, and other devices may of course also be used.

(Vehicle state recognition unit 140)
The vehicle state recognition unit 140 is provided to recognize the state of the vehicle 1. The vehicle state recognition unit 140 also includes a vehicle speed sensor 140a, a steering wheel angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f, and is able to recognize the state of the vehicle 1 using these devices.

(Vehicle speed sensor 140a)
The vehicle speed sensor 140a is a sensor for detecting the vehicle speed of the vehicle 1. The detected vehicle speed is input as a vehicle speed signal to the control unit 110, and the control unit 110 stores the vehicle speed information in RAM. This allows the control unit 110 to recognize the vehicle speed of the vehicle 1 in real time and after the fact.

(Steering wheel angle sensor 140b)
The steering wheel angle sensor 140b is a sensor for detecting the steering wheel angle (angle of the steering wheel 4) of the vehicle 1. The detected steering wheel angle is input as an angle signal to the control unit 110, which then stores the angle information in RAM. This allows the control unit 110 to recognize the steering wheel angle (angle of the steering wheel 4) of the vehicle 1 in real time and afterwards.

(Accelerator pedal sensor 140c)
The accelerator pedal sensor 140c is a sensor for detecting the depression amount of an accelerator pedal (not shown), and the detected depression amount is input as a depression amount signal to the control unit 110, which then stores the depression amount information in RAM. This allows the control unit 110 to recognize the depression amount of the accelerator pedal of the vehicle 1 in real time and after the fact.

(Brake pedal sensor 140d)
The brake pedal sensor 140d is a sensor for detecting the depression amount of a brake pedal (not shown), and the detected depression amount is input as a depression amount signal to the control unit 110, which then stores the depression amount information in RAM. This allows the control unit 110 to recognize the depression amount of the brake pedal of the vehicle 1 in real time and after the fact.

(G sensor 140e)
The G sensor 140e is a sensor for detecting the acceleration, deceleration, and inclination of the vehicle 1. When acceleration is detected, the acceleration amount is input as an acceleration amount signal, when deceleration is detected, the deceleration amount is input as a deceleration amount signal, and when inclination is detected, the inclination angle amount is input as an inclination angle signal to the control unit 110, and the control unit 110 stores the acceleration information, deceleration information, and inclination information in RAM. This allows the control unit 110 to recognize the acceleration, deceleration, and inclination of the vehicle 1 in real time and after the fact.

(Shift position sensor 140f)
The shift position sensor 140f is a sensor for detecting the shift range (shift position) of the vehicle 1. That is, the shift position sensor 140f detects the position to which the shift range is set, such as parking range (stopped position), drive range (driving position), reverse range (backing position), neutral range (neutral position), etc. The shift range detected by the shift position sensor 140f is input to the control unit 110, which then stores the current shift range in RAM. This allows the control unit 110 to recognize the position to which the shift range is set.

In this way, the control unit 110 can recognize the state of the vehicle 1 in real time and after the fact. As a result, the control unit 110 can control the agent 200 and the agent speaker 191 based on the recognition of the state of the vehicle 1 to notify the driver H of information about the state of the vehicle 1. For example, if the vehicle is traveling at an appropriate speed, the control unit 110 can notify the driver H by gestures and voice, such as "You are traveling at an appropriate speed." This leads to improved safety.

In this embodiment, the vehicle state recognition unit 140 includes a vehicle speed sensor 140a, a steering wheel angle sensor 140b, an accelerator pedal sensor 140c, a brake pedal sensor 140d, a G sensor 140e, and a shift position sensor 140f, but these are merely examples, and other devices may of course be used.

(Speaker 150)
The speaker 150 is attached to, for example, the instrument panel 3, and outputs sounds other than the sound emitted from the agent 200, warning sounds, etc. Note that an audio speaker built into the vehicle 1 may be used instead of providing the speaker 150. Furthermore, as will be described later, the speaker 150 may also serve as the agent speaker 191.

(Mike 160)
The microphone 160 is attached to, for example, the instrument panel 3, and receives input of voices emitted by the driver H and other passengers.

(Storage unit 170)
The storage unit 170 is capable of storing information acquired from each of the recognition units described above, dialogues held between the driver H and the control unit 110, and the like. By storing this information in the storage unit 170, the control unit 110 can recognize the driving tendencies of the driver H (for example, what kind of driving style the driver is), the hobbies and preferences of the driver H (for example, what kind of background music the driver likes), and the like. Furthermore, by recognizing these, it is also possible for the agent device 100 (agent 200) to actively hold a dialogue that is tailored to the driving tendencies of the driver H and the hobbies and preferences of the driver H.

(Transceiver 180)
The transceiver 180 can, for example, acquire information using an in-vehicle wireless LAN, acquire location information using a satellite positioning system, etc. Based on this acquired information and the information stored in the storage unit 170, the control unit 110 can also actively carry out a dialogue from the agent device 100 side (agent 200) that is tailored to the driving tendencies and hobbies and preferences of the driver H.

(Agent control unit 190)
The agent control unit 190 is equipped with a CPU, ROM, RAM, input/output ports, etc. (not shown), and when information is input from the control unit 110, for example, it can operate each part of the agent 200 and cause the agent speaker 191 to output a specified voice.

(Agent Speaker 191)
As described above, the agent speaker 191 is provided inside the head 211 of the agent 200, and outputs sounds and the like emitted from the agent 200. Incidentally, by providing the agent speaker 191 inside the head 211 of the agent 200, particularly in the area corresponding to the mouth, as in this embodiment, it becomes more realistic and contributes to building a relationship of trust, but in cases where the agent 200 is small and it is difficult to provide the agent speaker 191 inside the head 211, the above-mentioned speaker 150 may be used instead, or the agent speaker 191 or a speaker that serves as a substitute may be provided in another part inside the vehicle 1.

(Operating state switching operation unit 310)
The driving state switching operation unit 310 inputs an instruction to switch the driving state between manual driving and automatic driving, and depending on the input operation, inputs an instruction to switch from automatic driving to manual driving or an instruction to switch from manual driving to automatic driving to the control unit 110. This allows the driver H to switch between manual driving and automatic driving at his/her will.

Next, the operation and dialogue of the agent 200 in the agent device 100 will be described. The operation and dialogue of the agent 200 are performed by control processing by the control unit 110, and the agent control unit 190 operates each unit of the agent 200, while audio is output from the agent speaker 191. The control flow for performing the control processing is stored in the ROM of the control unit 110, and the CPU of the control unit 110 reads it from the ROM and performs various processes.

The main processing of the agent control process performed by the CPU of the control unit 110 will be explained below using Figure 6.

(Step S10)
In step S10, when the ignition is turned on, the CPU of the control unit 110 starts the main processing of the agent control processing. Specifically, the CPU of the control unit 110 reads the main processing of the agent control processing from the ROM, starts the processing, and then proceeds to step S100.

(Step S100)
In step S100, the CPU of the control unit 110 performs a boarding process. The boarding process is a process that starts when the driver H gets into the vehicle 1. For example, in the boarding process, the CPU of the control unit 110 performs a process of making the agent 200 face the driver H. The CPU of the control unit 110 also performs processes such as having a conversation with the driver H and having the agent 200 provide boarding explanations, such as weather, remaining fuel information, and important vehicle maintenance information, through speech and movement. Note that when boarding, the agent 200 is in a standing position. Then, when the boarding process is completed, the process proceeds to step S200.

(Step S200)
In step S200, the CPU of the control unit 110 performs pre-driving processing. In the pre-driving processing, processing is performed before the driver H starts driving. For example, in the pre-driving processing, the agent 200 provides instructions on precautions regarding the driving environment to the destination and explains rest points. In explaining the pre-driving processing, the agent 200 faces the driver H, moves both hands to explain each item, and also signals completion by nodding its head after each item is completed. Then, when the pre-driving processing is completed, the process proceeds to step S20.

(Step S20)
In step S20, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). Specifically, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), it proceeds to step S300. If the CPU of control unit 110 determines that the shift range is the parking range (in this case, the shift position has not been changed), it proceeds to step S400.

(Step S300)
In step S300, the CPU of the control unit 110 performs a driving process. In the driving process, the CPU performs a control process for the agent 200 while the vehicle is driving. For example, when the shift range input from the shift position sensor 140f is "D" (drive range), the CPU of the control unit 110 performs a control process for the agent 200 while the vehicle is driving forward, such as making the agent 200 face forward (the direction of travel of the vehicle 1) (taking a predetermined posture), and when the shift range is "R" (reverse range), the CPU performs a control process for the agent 200 while the vehicle is driving backward. Details of the driving process will be described later. Note that, while the vehicle is driving, the agent 200 is in a standing position during manual driving and in a seated position during automatic driving, as will be described later. Then, when the driving process is completed, the CPU proceeds to step S30.

(Step S30)
In step S30, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). As described above, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range, it proceeds to step S300, and if it determines that the shift range is the parking range, it proceeds to step S400.

(Step S400)
In step S400, the CPU of the control unit 110 performs end-of-driving processing. In the end-of-driving processing, end processing is performed when a predetermined journey has ended. For example, in the end-of-driving processing, the CPU of the control unit 110 makes the agent 200 face the driver H again and assume the basic posture. In addition, the CPU of the control unit 110 outputs audio and performs actions to congratulate the driver when the drive is over, and if there is an abnormality or notification information in the vehicle, it notifies the abnormality, issues a warning sound, or notifies the information by audio or action. Then, when the end-of-driving processing is completed, the process proceeds to step S40.

(Step S40)
In step S40, the CPU of control unit 110 performs processing to determine whether the shift range is "P" (parking range). As described above, the CPU of control unit 110 determines whether the shift range input from shift position sensor 140f is the parking range. If the CPU of control unit 110 determines that the shift range is not the parking range (in this case, the shift position has been changed), it proceeds to step S300. If the CPU of control unit 110 determines that the shift range is the parking range (in this case, the shift position has not been changed), it proceeds to step S50.

(Step S50)
In step S50, the CPU of the control unit 110 performs a driving resumption determination process. In the driving resumption determination process, a process is performed to determine whether or not the driver H intends to resume driving. For example, the CPU of the control unit 110 determines that the driver H intends to resume driving when the driver H makes a voice input such as "I will resume driving," or when a new destination is input into the navigation system by voice input or manual input. If the CPU of the control unit 110 determines that the driver H intends to resume driving, it proceeds to step S200, and if it determines that the driver H does not intend to resume driving, it proceeds to step S60.

(Step S60)
In step S60, the CPU of control unit 110 determines whether the ignition is turned off. If the ignition is turned off, the CPU of control unit 110 ends the main process of the agent control process. If the ignition is not turned off, the CPU of control unit 110 proceeds to step S40.

Next, the driving process performed by the CPU of the control unit 110 will be explained using Figure 7. Figure 7 shows a subroutine of step S300 (driving process) in Figure 6.

(Step S301)
In the driving process, first, in step S301, the CPU of control unit 110 performs processing to determine whether the vehicle is in automatic driving or not. Specifically, the CPU of control unit 110 detects whether the current driving state is an "automatic driving state" or a "manual driving state," and if it is an "automatic driving state," determines that the vehicle is in automatic driving, and if it is a "manual driving state," determines that the vehicle is not in automatic driving. If the CPU of control unit 110 determines that the vehicle is in automatic driving, it proceeds to step S370, and if it determines that the vehicle is not in automatic driving, it proceeds to step S302.

(Step S302)
In step S302, the CPU of the control unit 110 performs processing to determine whether a switching operation to autonomous driving has been performed or whether a switching condition for autonomous driving has been met. Specifically, the CPU of the control unit 110 determines whether a switching operation to autonomous driving has been performed by the driving state switching operation unit 310, or whether a transition condition for autonomous driving has been met based on information acquired from the surrounding environment recognition unit 120, the occupant state recognition unit 130, the vehicle state recognition unit 140, etc. The transition condition for autonomous driving is a pre-set condition for transitioning to autonomous driving, such as a condition set when the vehicle is traveling in traffic at a speed of 60 km/h or less while traveling on a highway. Note that the CPU of the control unit 110 may not switch to autonomous driving if a predetermined condition is not met, even if a switching operation to autonomous driving has been performed. If the CPU of the control unit 110 determines that an operation to switch to automatic driving has been performed or that the conditions for switching to automatic driving have been met, it proceeds to step S360; if it determines that an operation to switch to automatic driving has not been performed or that the conditions for switching to automatic driving have not been met, it proceeds to step S305.

(Step S360)
In step S360, the CPU of the control unit 110 performs automatic driving switching processing. In the automatic driving switching processing, a warning is issued when switching from manual driving to automatic driving. Details of the automatic driving switching processing will be described later. Then, when the automatic driving switching processing is completed, the processing proceeds to step S370.

(Step S370)
In step S370, the CPU of the control unit 110 performs automatic driving processing. In the automatic driving processing, control processing is performed for the vehicle 1 and the agent 200 during automatic driving. For example, the CPU of the control unit 110 performs operation processing for the agent 200 during automatic driving. Details of the automatic driving processing will be described later. Then, when the automatic driving processing is completed, the process proceeds to step S303.

(Step S303)
In step S303, the CPU of the control unit 110 performs processing to determine whether autonomous driving has ended. Specifically, the CPU of the control unit 110 determines whether a switching operation to means driving has been performed by the driving state switching operation unit 310, or whether an autonomous driving end condition has been met based on information acquired from the surrounding environment recognition unit 120, the occupant state recognition unit 130, the vehicle state recognition unit 140, etc. Here, the autonomous driving end condition is different from the autonomous driving transition condition described above. For example, the fulfillment of the autonomous driving end condition may be defined as the case where the autonomous driving transition condition is not met. If the CPU of the control unit 110 determines that autonomous driving has ended, it proceeds to step S380. If the CPU of the control unit 110 determines that autonomous driving has not ended, it proceeds to step S370.

(Step S380)
In step S380, the CPU of the control unit 110 performs an automatic driving termination process. In the automatic driving termination process, a warning is issued when switching from automatic driving to manual driving, etc. Details of the automatic driving termination process will be described later. Then, when the automatic driving termination process is completed, the process proceeds to step S305.

(Step S305)
In step S305, the CPU of the control unit 110 performs normal driving processing. In the normal driving processing, processing is performed when the vehicle is being driven manually. For example, the CPU of the control unit 110 performs control processing of the agent 200 while the vehicle is being driven. Details of the normal driving processing will not be explained here. After the normal driving processing is performed, the process proceeds to step S306.

(Step S306)
In step S306, the CPU of control unit 110 performs processing to determine whether or not the vehicle speed is "0." Specifically, the CPU of control unit 110 determines whether or not the vehicle 1 has stopped based on the vehicle speed input from vehicle speed sensor 140a. If the CPU of control unit 110 determines that the vehicle speed is not "0," it proceeds to step S302, and if it determines that the vehicle speed is "0," it ends the in-travel processing.

Next, the automatic driving switching process performed by the CPU of the control unit 110 will be explained using Figure 8. Figure 8 shows a subroutine of step S360 (automatic driving switching process) in Figure 7.

(Step S361)
In the automatic driving switching process, first, in step S361, the CPU of the control unit 110 performs processing to notify (warn) the driver of the switch to automatic driving. Specifically, the CPU of the control unit 110 notifies the driver H via the agent speaker 191 that the vehicle will be switched from a manual driving state to an automatic driving state, thereby making the driver H aware that the vehicle will be switched to automatic driving. After notifying the driver of the switch to automatic driving, the process proceeds to step S362.

(Step S362)
In step S362, the CPU of the control unit 110 performs a control process for switching to automatic driving. Specifically, the CPU of the control unit 110 performs a process for switching the driving state from a manual driving state to an automatic driving state. In the automatic driving state, for example, even if the driver H does not operate the steering wheel 4, the vehicle 1 is steered left and right based on information acquired from the surrounding environment recognition unit 120, etc. Then, after the control process for switching to automatic driving is performed, the process proceeds to step S363.

(Step S363)
In step S363, the CPU of the control unit 110 performs processing to change the agent 200 from a standing position to a sitting position. Specifically, as shown in FIG. 9 , the CPU of the control unit 110 causes the agent 200 to bend both legs and assume a sitting position. That is, the CPU of the control unit 110 changes the agent 200 from the standing position shown in FIG. 9( a) to the sitting position shown in FIG. 9( b). Because the driver H perceives the agent 200 in his peripheral vision, he can recognize that the driving mode has switched to autonomous driving from the large change in the agent 200's position, and can intuitively understand that his level of tension has decreased due to the change in position (the small light commonly seen in European cars does not alert the driver to the switch to autonomous driving, and even if he does, the level of intuitive understanding is low). During normal autonomous driving, the agent 200 is in a standby state with both arms down. Then, once the agent 200 has been changed from a standing position to a sitting position, the autonomous driving switching processing ends.

Next, the automatic driving process performed by the CPU of the control unit 110 will be explained using Figure 10. Figure 10 shows a subroutine of step S370 (automatic driving process) in Figure 7.

(Step S371)
In the autonomous driving process, first, in step S371, the CPU of the control unit 110 performs processing to determine whether or not there is an approaching object. Specifically, the CPU of the control unit 110 determines whether or not there is an object (or person, etc.) approaching the vehicle 1 within a predetermined range and at a predetermined speed or faster, based on information about the surrounding situation input from the exterior camera 120a, radar 120b, etc. In other words, the CPU of the control unit 110 determines whether or not it is necessary to avoid an obstacle, based on the acquired surrounding situation. If the CPU of the control unit 110 determines that there is an approaching object, it proceeds to step S372, and if it determines that there is no approaching object, it proceeds to step S375.

(Step S372)
In step S372, the CPU of control unit 110 performs processing to determine whether or not to perform sudden deceleration. That is, even if there is an approaching object, there are cases where it is necessary to perform sudden deceleration and take evasive action immediately for vehicle 1, cases where it is sufficient to perform predetermined evasive action without performing sudden deceleration, or cases where it is not necessary to perform evasive action. If the CPU of control unit 110 determines that sudden deceleration will be performed, it proceeds to step S373, and if it determines that sudden deceleration will not be performed, it proceeds to step S374.

(Step S373)
In step S373, the CPU of the control unit 110 causes the agent 200 to suddenly decelerate. Specifically, as shown in FIG. 11 , the CPU of the control unit 110 causes the agent 200 to turn its palms downward and spread both arms horizontally. During autonomous driving, the CPU of the control unit 110 notifies the driver H of information by having the agent 200 crouch down and mainly move its upper body. This allows the driver H to recognize that the vehicle 1 is decelerating. For example, when the approach of a leading vehicle is detected, the CPU of the control unit 110 causes the agent 200 to extend both arms forward while remaining in a seated position, and then to make both arms horizontal, thereby expressing deceleration. Furthermore, the CPU of the control unit 110 causes the vehicle 1 to decelerate and perform predetermined avoidance control. After the sudden deceleration and avoidance control have been performed, the autonomous driving process ends.

(Step S374)
In step S374, the CPU of the control unit 110 causes the agent 200 to point at the approaching object, etc. Specifically, as shown in FIG. 12 , when an approaching object is detected, the CPU of the control unit 110 causes the agent 200 to, while remaining in a seated position, point one arm toward the approaching object, i.e., raise one arm and point in the direction of the approaching object. Note that, during autonomous driving, if there is any change, the CPU of the control unit 110 causes the agent 200 to tilt its head and point, as shown in FIG. 12 . Furthermore, if no significant change occurs in the driving, the agent 200 merely tilts its head, without making any major changes in its posture. Furthermore, if a predetermined action is required due to the approaching object, the CPU of the control unit 110 causes deceleration control or predetermined avoidance control to be performed. After pointing at the approaching object, etc., the autonomous driving process is terminated. Even if the automatic driving processing shown in Figure 10 is terminated, as described above, if the automatic driving end determination (step S303 in Figure 7) determines that automatic driving is continuing, the automatic driving processing returns to this automatic driving processing and continues.

(Step S375)
In step S375, the CPU of the control unit 110 performs processing to determine whether the traveling path of the vehicle 1 is at a corner. That is, the CPU of the control unit 110 performs processing to determine whether the traveling path of the vehicle 1 is immediately before a corner. For example, the CPU of the control unit 110 determines that the traveling path of the vehicle 1 is at a corner when there is a corner on the traveling path of the vehicle 1 and the distance to the corner is within a predetermined distance or the estimated time to reach the corner is within a predetermined time. Note that the distance, time, etc. used for this determination are desirably changed depending on the speed of the vehicle 1, the conditions of the road, etc. If the CPU of the control unit 110 determines that the traveling path of the vehicle 1 is at a corner, it proceeds to step S376. If the CPU of the control unit 110 determines that the traveling path of the vehicle 1 is not at a corner, it proceeds to step S377.

(Step S376)
In step S376, the CPU of the control unit 110 performs processing to tilt the agent 200 while it is in a seated position. Specifically, as shown in FIG. 13 , when the vehicle approaches a corner, the CPU of the control unit 110 causes the agent 200 to tilt its upper body in the direction of centrifugal force and to extend its opposite arm diagonally. The CPU then changes the degree of tilt depending on the duration of the corner and the size of the turning radius, thereby expressing the amount of change during the turn. The CPU of the control unit 110 also performs similar operations when changing lanes, etc. Once the agent 200 has been caused to tilt, the autonomous driving processing ends.

(Step S377)
In step S377, the CPU of the control unit 110 performs operation processing according to the driving state. Here, processing other than the processing when there is an approaching object during autonomous driving or when cornering is performed. For example, the CPU of the control unit 110 performs deceleration processing when approaching an intersection during autonomous driving, and processing to control the lighting of lights at night. Details of the operation processing according to the driving state will not be explained here. Then, after the operation processing according to the driving state is performed, the processing during autonomous driving ends.

Next, the automatic driving termination process performed by the CPU of the control unit 110 will be explained using Figure 14. Figure 14 shows a subroutine of step S380 (automatic driving termination process) in Figure 7.

(Step S381)
In the automatic driving termination process, first, in step S381, the CPU of the control unit 110 performs a process of notifying (warning) of a switch to manual driving. Specifically, the CPU of the control unit 110 notifies the driver H via the agent speaker 191 that the automatic driving state will be switched to a manual driving state, and makes the driver H aware that automatic driving has ended and that the vehicle will be switched to manual driving by the driver H's own operation. Then, after notifying the driver of the switch to manual driving, the process proceeds to step S382.

(Step S382)
In step S382, the CPU of the control unit 110 performs processing to change the agent 200 from a seated position to a standing position. Specifically, the CPU of the control unit 110 changes the agent 200 from the seated position shown in FIG. 9(b) to the standing position shown in FIG. 9(a). Driver H can recognize from the large change in the position of the agent 200 that automatic driving will switch to manual driving, and can make preparations for manual driving. Then, after changing the agent 200 from a seated position to a standing position, the process proceeds to step S383.

(Step S383)
In step S383, the CPU of the control unit 110 performs a control process for switching to manual driving. Specifically, the CPU of the control unit 110 performs a process for switching the driving state from an automatic driving state to a manual driving state. Then, after performing the control process for switching to manual driving, the automatic driving end process is terminated.

As described above, when switching between manual and automated driving, the agent device 100 of this embodiment has the agent 200 switch between a standing and a sitting position, clearly communicating whether the vehicle is being driven manually or automatically. This allows the driver H to easily distinguish between manual and automated driving, thereby supporting safety while driving. Furthermore, when switching from manual to automated driving, the agent 200 switches from a standing to a sitting position, allowing the driver H to detect the switch to automated driving with his or her peripheral vision, supporting safety while driving. In other words, while driving manually, the driver H can recognize the switch to automated driving even if he or she is driving normally while paying attention to the road ahead, without staring at any particular lamp or the like. This supports safety while driving without causing the driver to lose focus of the road ahead.

Furthermore, when switching from manual driving to automatic driving, the agent 200 is switched from a seated position to a standing position, so that the driver H can easily recognize the switch to automatic driving, thereby supporting safety while driving. Furthermore, before switching to manual driving, the driver H is notified by voice and by a change in the position of the agent 200, so that the driver H can prepare for driving before switching to manual driving, thereby supporting safety.
Furthermore, during automatic driving, the agent 200 operates according to the driving conditions, making it easier for the occupants to take a stance according to the driving conditions, thereby supporting safety.

In the agent control process of this embodiment, a program stored in the ROM, RAM, EEPROM, etc. of the control unit 110 is loaded into the RAM, etc. of the control unit 110 and executed by the CPU, etc. of the control unit 110.
Furthermore, in this embodiment, the shift position sensor 140f constitutes a shift position detection unit of the present invention. Also, in this embodiment, the surrounding environment recognition unit 120 constitutes a surrounding situation acquisition unit of the present invention. Also, in this embodiment, the control unit 110 constitutes a driving state switching unit and an autonomous driving determination unit of the present invention.

Furthermore, in this embodiment, the agent control unit of the present application is configured using both the control unit 110 and the agent control unit 190, but as mentioned above, the agent control unit 190 alone may have the functions of both the control unit 110 and the agent control unit 190 of this embodiment.

1: vehicle, 100: agent device, 110: control unit, 120: surrounding environment recognition unit, 130: occupant state recognition unit, 140: vehicle state recognition unit, 140f: shift position sensor, 150: speaker, 160: microphone, 170: memory unit, 180: transceiver, 190: agent control unit, 191: agent speaker, 200: agent, 310: driving state switching operation unit

Claims (7)

a memory unit that stores a dialogue with a driver;
a surrounding situation acquisition unit for acquiring a surrounding situation;
a driving state switching unit that switches between manual driving and automatic driving based on the surrounding conditions acquired by the surrounding condition acquisition unit or an instruction from the driver;
an automatic driving determination unit that determines whether the vehicle is in an automatic driving state;
an agent control unit that controls a personified agent;
Equipped with
The agent:
It is positioned in the driver's peripheral vision,
The agent control unit
A dialogue tailored to the driver's preferences can be executed based on at least the information stored in the storage unit,
When switching between manual and automatic driving, the agent is switched between a standing position and a sitting position.
1. An agent device comprising: The agent control unit
When switching from manual driving to automatic driving, the agent is switched from a standing position to a sitting position.
2. The agent device according to claim 1. The agent control unit
When an approaching object is detected by the surrounding situation acquisition unit, in the case of autonomous driving, the driver is instructed to point one arm toward the approaching object while remaining in a seated position.
3. The agent device according to claim 2. The agent control unit
When approaching a corner during autonomous driving, the driver remains seated and tilts only the upper body in the direction of centrifugal force.
4. The agent device according to claim 2 or 3. The agent control unit
If the approach of a vehicle ahead is detected during autonomous driving, the driver will remain seated and extend both arms forward, then horizontally, to indicate deceleration.
5. The agent device according to claim 2, wherein the agent device is a device for receiving an information from the agent. The agent control unit
When switching from automatic driving to manual driving, the agent is switched from a seated position to a standing position.
6. The agent device according to claim 1, wherein the agent device is a device for receiving an information from a user. The agent control unit
When the driving state switching unit switches from automatic driving to manual driving, it notifies the driver that the driving state will be switched before switching to manual driving.
7. The agent device according to claim 1, wherein the agent device is a device that performs a plurality of operations.