JP7632097B2 - CONTROL DEVICE, CONTROL SYSTEM, AND CONTROL METHOD - Google Patents

Description

This disclosure relates to a control device, a control system, and a control method.

In recent years, various technologies have been developed to control the sound image of the output sound from an audio output device, such as the technologies described in Patent Documents 1 and 2.

Japanese Patent Application Publication No. 5-183998 Japanese Patent Application Publication No. 5-336599

However, the technologies described in Patent Documents 1 and 2 are technologies for controlling the position of the sound image of the sound output from a single headphone. In other words, each of the technologies described in Patent Documents 1 and 2 merely discloses a method for controlling the position of the sound image in a single audio output device.

One aspect of the present disclosure aims to comprehensively control (i) the position of the sound image of the output sound from a certain audio output device worn by a user, and (ii) the position of the sound image of the output sound from another audio output device.

To solve the above problem, the control device according to aspect 1 of the present invention includes an information acquisition unit that acquires from a detection device position information indicating the head position and orientation information indicating the head orientation for each of a plurality of users staying within a specified space, a sound image position determination unit that determines the position of a sound image of an output sound of a first output device worn by each of the plurality of users, and an output control unit that controls the output of each of the first output devices based on the head position and head orientation of each of the plurality of users so that a sound image of the output sound is formed at the position determined by the sound image position determination unit.

According to the above configuration, it is possible to comprehensively control the positions of the sound images of the output sounds of multiple first output devices worn by multiple users. In other words, according to the above configuration, it is possible to comprehensively control (i) the position of the sound image of the output sound from a first output device worn by a certain user, and (ii) the position of the sound image of the output sound from another first output device worn by a user other than the certain user.

In the control device according to aspect 2 of the present invention, in the above aspect 1, the sound image position determination unit may determine the position of the sound image such that the positions of the sound images of the output sounds of the first output devices are in the same position.

This configuration allows multiple users wearing the first output device to feel as if they are hearing sound coming from the same position.

The control device according to aspect 3 of the present invention is the same as in aspect 2, and includes a voice acquisition unit that acquires the voice of one of the multiple users from a microphone disposed in the specified space or worn by the user, and when the voice acquisition unit acquires the voice of the utterance, the sound image position determination unit determines the position of the sound image based on the head position and head orientation of the user who made the utterance so that the vicinity of the user who made the utterance becomes the position of the sound image of the output sound of each of the first output devices, and the output control unit may output the voice of the utterance acquired by the voice acquisition unit from each of the first output devices.

According to the above configuration, the sound image position determination unit determines the sound image position of the output sound of each first output device to be near the speaker. Then, the output control unit can output the sound of the utterance acquired by the sound acquisition unit from each first output device. This makes it easier to hear the speech of each user within a specified space.

In the control device according to aspect 4 of the present invention, in the above aspect 1, the sound image position determination unit may individually determine the position of the sound image of the output sound of each of the first output devices.

With the above configuration, the sound image position of the output sound of each first output device can be determined individually. This allows each user to feel as if the sound is coming from a position that is appropriate for each individual.

In the control device according to aspect 5 of the present invention, in any one of aspects 1 to 4, the sound image position determination unit may determine the position of the sound image of each of the first output devices when causing the first output devices to output a specific sound, and the output control unit may control the output of each of the first output devices so that the sound image of the specific sound is formed at the position determined by the sound image position determination unit.

With this configuration, when a sound image of a specific sound is output from multiple first output devices, the position of the sound image of the output sound can be controlled.

A control device according to aspect 6 of the present invention is any one of aspects 1 to 5, in which the detection device includes a status sensor for measuring or acquiring status information for identifying the status of each of the multiple users, the information acquisition unit acquires the status information of each user, and the control device may include a sound determination unit that determines the output sound of the first output device based on the status information of each user.

The above configuration allows appropriate audio to be output according to the user's condition.

In order to solve the above problem, the control device according to aspect 7 of the present invention includes an information acquisition unit that acquires from a detection device position information indicating the head position and orientation information indicating the head orientation for each of a plurality of users staying within a specified space, a sound image position determination unit that determines the position of a sound image of an output sound of a first output device worn by at least one of the plurality of users and the position of a sound image of an output sound of a second output device arranged within the specified space, and an output control unit that controls the output of each of the first output device and the second output device based on the head position and head orientation of each of the plurality of users so that the sound image of the output sound is formed at the position determined by the sound image position determination unit.

With this configuration, it is possible to comprehensively control the position of the sound image of the output sound of the first output device worn by the user and the position of the sound image of the output sound of the second output device placed within a specified space.

In the control device according to aspect 8 of the present invention, in the above aspect 7, the sound image position determination unit may determine the positions of the sound images of the output sounds of the first output device and the second output device such that the positions of the sound images of the output sounds of the first output device and the second output device are in the same position.

With this configuration, a user wearing the first output device and a user listening to the output sound from the second output device can feel as if the sound is coming from the same position.

The control device according to aspect 9 of the present invention is the same as in aspect 8, and includes a voice acquisition unit that acquires the voice of one of the multiple users from a microphone disposed in the specified space or worn by the user, and when the voice acquisition unit acquires the voice of the utterance, the sound image position determination unit determines the position of the sound image based on the head position and head orientation of the user who made the utterance so that the vicinity of the user who made the utterance becomes the position of the sound image of the output sound of each of the first output device and the second output device, and the output control unit may output the voice of the utterance acquired by the voice acquisition unit from each of the first output device and the second output device.

According to the above configuration, the sound image position determination unit determines the sound image positions of the output sounds of the first output device and the second output device to be near the speaker. Then, the output control unit can output the sound of the utterance acquired by the sound acquisition unit from the first output device and the second output device. This makes it easier to hear the speech of each user within a specified space.

In the control device according to aspect 10 of the present invention, in the above aspect 7, the sound image position determination unit may determine the position of the sound image of each of the output sounds of the first output device and the second output device individually.

With the above configuration, the sound image positions of the output sounds of the first output device and the second output device can be determined individually. This allows each user to feel as if the sound is coming from a position appropriate to them.

In the control device according to aspect 11 of the present invention, in any one of aspects 7 to 10, when the sound image position determination unit causes the first output device and the second output device to output a specific sound, the sound image position determination unit may determine the position of the sound image of each of the first output device and the second output device, and the output control unit may control the output of each of the first output device and the second output device so that the sound image of the specific sound is formed at the position determined by the sound image position determination unit.

With this configuration, when a sound image of a specific sound is output from the first output device and the second output device, the position of the sound image of the output sound can be controlled.

A control device according to aspect 12 of the present invention is any one of aspects 7 to 11, in which the detection device includes a status sensor for measuring or acquiring status information for identifying the status of each of the multiple users, the information acquisition unit acquires the status information of each user, and the control device includes a sound determination unit that determines the output sounds of the first output device and the second output device based on the status information of each user.

The above configuration allows appropriate audio to be output according to the user's condition.

To solve the above problem, the control system according to aspect 13 of the present invention includes the control device according to any one of aspects 1 to 6, the detection device, and the plurality of first output devices. With the above configuration, the same effect as the control device according to any one of aspects 1 to 6 is achieved.

To solve the above problem, the control system according to aspect 14 of the present invention includes the control device according to any one of aspects 7 to 12, the detection device, the first output device, and the second output device. With the above configuration, the same effect as the control device according to any one of aspects 7 to 12 is achieved.

In order to solve the above problem, the control method according to aspect 15 of the present invention includes an information acquisition step of acquiring, from a detection device, position information indicating the head position and orientation information indicating the head orientation for each of a plurality of users staying in a predetermined space, a sound image position determination step of determining the position of a sound image of an output sound of a first output device worn by each of the plurality of users, and an output control step of controlling the output of each of the first output devices based on the head position and head orientation of each of the plurality of users so that a sound image of the output sound is formed at the position determined in the sound image position determination step. With the above configuration, the same effect as the control device according to aspect 1 is achieved.

In order to solve the above problem, the control method according to aspect 16 of the present invention includes an information acquisition step of acquiring, from a detection device, position information indicating the head position and orientation information indicating the head orientation for each of a plurality of users staying in a predetermined space; a sound image position determination step of determining the position of a sound image of an output sound of a first output device worn by at least one of the plurality of users and the position of a sound image of an output sound of a second output device arranged in the predetermined space; and an output control step of controlling the output of each of the first output device and the second output device based on the head position and head orientation of each of the plurality of users so that the sound image of the output sound is formed at the position determined in the sound image position determination step. With the above configuration, the same effect as the control device according to aspect 7 is achieved.

According to one aspect of the present disclosure, it is possible to comprehensively control (i) the position of the sound image of the output sound from a certain audio output device worn by a user, and (ii) the position of the sound image of the output sound from another audio output device.

FIG. 2 is a diagram showing an overview of the operation of the control system according to the first embodiment. 1 is a block diagram showing an example of a configuration of a main part of a control system according to a first embodiment; 1 is a diagram showing an example of an outline of an interior space of a vehicle according to a first embodiment, and an example of a positional relationship between a detection device, a headphone 31, and a speaker. 1 is a diagram showing the relationship between the position and direction of an occupant's head and the position of a sound image on the in-vehicle spatial coordinate system in the first embodiment. FIG. 4 is a flowchart showing an example of a flow of a process executed by the control device according to the first embodiment. FIG. 11 is a diagram showing an outline of the operation of a control system according to a second embodiment. 13 is a diagram showing the relationship between the position and direction of an occupant's head and the position of a sound image on the in-vehicle spatial coordinate system in the second embodiment. FIG. 11 is a flowchart showing an example of a flow of a process executed by a control device according to a third embodiment. FIG. 11 is a diagram showing an overview of the operation of a control system according to a third embodiment. FIG. 11 is a block diagram showing an example of a configuration of a main part of a control system according to a third embodiment. 13 is a diagram showing the relationship between the position and direction of an occupant's head and the position of a sound image on the in-vehicle spatial coordinate system in the third embodiment. FIG. 11 is a flowchart showing an example of a flow of a process executed by a control device according to a third embodiment.

The control system according to the present disclosure is a system that comprehensively controls multiple audio output devices that exist within a specified space. The control system controls the output of each audio output device, thereby controlling the position of the sound image that a person in the specified space perceives when hearing the output sound.

In this disclosure, a "predetermined space" refers to a space large enough for humans to stay in, such as the interior space of a moving object or a building. Examples of moving objects include automobiles, airplanes, and trains. Examples of buildings include movie theaters, commercial facilities, and homes.

In addition, in this disclosure, "sound image" refers to the sound source as perceived by humans. Therefore, in this disclosure, the "position of the sound image" of a certain sound refers to the position where a human perceives "this is the source of this sound." Note that the position of the sound image does not necessarily coincide with the position of the actual sound source. Therefore, the position of the sound image of a certain sound may be the position of the virtual sound source of that certain sound.

In addition, in this disclosure, the "audio output device" that is the object of control by the control system is classified into a first output device and a second output device. The first output device is, for example, an audio output device worn by the user. The first output device can be realized, for example, by headphones and earphones. In contrast, the second output device is an audio output device that is placed in a specified space. The second output device can be realized, for example, by an in-car speaker or a speaker integrated with a television.

The control system according to the present disclosure controls a plurality of first output devices. Alternatively, the control system according to the present disclosure controls one or more first output devices and one or more second output devices. Various embodiments of the control system according to the present disclosure are specifically described below.

In the following embodiments, the "predetermined space" is the interior space of a vehicle. In other words, the users staying in the predetermined space are the vehicle occupants. In the following embodiments, the first output device is headphones worn by the occupants, and the second output device is a speaker arranged in the interior space of the vehicle. In the following embodiments, the vehicle is assumed to have a total of four occupants, including the driver. The three occupants other than the driver are each assumed to be wearing headphones. However, the configuration of the present disclosure is not limited to these specific examples.

Furthermore, in each of the following embodiments, it is assumed that some kind of audio data is output from the headphones and the speaker, respectively. However, the configuration of the present disclosure is not limited to this. For example, even if the same audio data is output from the headphones and the speaker, the configuration of the embodiment described below is applicable. Also, for example, the control system may be configured to control the position of the sound image as described above at the timing when audio is output from the headphones and the speaker.

[Embodiment 1]
Overview
The control system of this embodiment is characterized in that it controls multiple audio output devices (i.e., multiple headphones, or one or more headphones and one or more speakers) respectively, thereby forming sound images of the output sounds from the multiple audio output devices at the same position within the vehicle.

Figure 1 is a diagram showing an overview of the operation of the control system 100 according to this embodiment. In the example of Figure 1, four occupants A to D, who are users of the control system 100, are sitting in seats provided in the interior space of the vehicle 1. Occupant B is wearing headphones 31b, occupant C is wearing headphones 31c, and occupant D is wearing headphones 31d. In addition, speakers 32a and 32e are located near occupant A.

The control system 100 controls the positions of the sound images of the output sounds of the headphones 31b to 31d and the speakers 32a and 32e, which are positioned as shown in FIG. 1. The control system 100 controls the headphones worn by each passenger and/or the speakers located near each passenger based on the position and orientation of the head of each passenger.

In the example of FIG. 1, the control system 100 controls the headphones 31b to 31d and the speakers 32a and 32e so that the sound images of the output sounds from the headphones 31b to 31d and the speakers 32a and 32e are all formed in the position of the illustrated range R1. This allows the passengers A to D, who are listening to the output sounds from different audio output devices, to feel as if they are all hearing the sounds from the same position.

<Main components>
Next, a specific configuration of the control system 100 will be described. Fig. 2 is a block diagram showing an example of a main configuration of the control system 100 according to this embodiment. In the example of Fig. 2, the control system 100 includes a control device 10, a detection device 20, a storage device 30, a headphone 31, and a speaker 32.

Headphone 31 is a general term for one or more headphones, and speaker 32 is a general term for one or more speakers. More specifically, in this embodiment, headphones 31 are headphones 31b, 31c, and 31d shown in FIG. 1. Furthermore, storage device 30 is not a required component of control system 100. Control device 10 is connected to detection device 20, storage device 30, headphones 31, and speaker 32 via wired or wireless communication.

In the example of FIG. 2, the control device 10, the detection device 20, the storage device 30, and the speaker 32 are arranged in the interior space of the vehicle 1. In addition, a user staying in the interior space of the vehicle 1 is wearing headphones 31. The structure of the vehicle 1 and the relative positions of the detection device 20, the headphones 31, and the speaker 32 will be described in detail later.

(Detection device 20)
The detection device 20 is a group of one or more devices arranged in the interior space of the vehicle 1. Each device included in the detection device 20 detects at least one of information indicating the position of the user's head (position information) and information indicating the orientation of the user's head (orientation information), and transmits it to the control device 10. The detection device 20 includes one or more devices capable of detecting position information and one or more devices capable of detecting orientation information. This allows the detection device 20, as a group of devices, to detect position information and orientation information and transmit it to the control device 10.

The format of the "position information" is not particularly limited as long as it is information that can identify the position of the user's head. Similarly, the format of the "orientation information" is not particularly limited as long as it is information that can identify the orientation of the user's head. Furthermore, the specific configurations of the various devices included in the detection device 20 are not limited to the example in FIG. 2.

In the example of FIG. 2, the detection device 20 includes a camera 21 and a gyro sensor 22. The camera 21 captures an image of the upper body of the user, for example, and transmits the captured image to the control device 10. Note that this captured image may be a still image or a video. As will be described in more detail later, the control device 10 that receives the captured image can identify the position and orientation of the user's head in the internal space of the vehicle 1 based on the captured image and the structure of the internal space. Therefore, the captured image of the camera 21 is position information and orientation information.

The gyro sensor 22 is a sensor built into the headphones 31. When the user wears the gyro sensor 22 together with the headphones 31, the gyro sensor 22 can detect a value indicating the tilt of the user's head (i.e., the direction of the head). Therefore, the measurement information of the gyro sensor 22 is direction information.

(Structure of vehicle 1 and positional relationship of devices)
3 is a diagram showing an example of an outline of the interior space of the vehicle 1 according to this embodiment, and an example of the positional relationship between the detection device 20, the headphones 31, and the speaker 32. In FIG. 3, the traveling direction of the vehicle 1 is defined as "forward". The front-rear, back-rear, left-right directions are as shown in the figure. In addition, the direction toward the ceiling of the vehicle 1 is defined as the upward direction, and the direction toward the floor surface of the vehicle 1 is defined as the downward direction.

In the example of Figure 3, four seats, seats ST1 to ST4, are provided in the interior space R of the vehicle 1. Four passengers, passenger A, passenger B, passenger C, and passenger D, are seated in the seats as shown in Figure 3.

The camera 21 is provided in a position that allows it to capture an image of the upper body (or from the neck up) of an occupant sitting in one of the seats ST1 to ST4. For example, as shown in FIG. 3, the camera 21 is positioned at an angle that captures an image of the rear from the center of the front of the vehicle 1. Note that multiple cameras 21 may be provided on the ceiling or walls of the vehicle 1. The camera 21 may also be a 360° camera that is positioned in a location in the center of the front-to-rear and left-to-right of the vehicle 1.

In the example of FIG. 3, similar to the example of FIG. 2, occupant B wears headphones 31b, occupant C wears headphones 31c, and occupant D wears headphones 31d. Headphones 31b to 31d are each an example of headphones 31 in FIG. 2. As described above, each headphone may be equipped with a gyro sensor 22 that is part of the detection device 20. In the example of FIG. 3, the gyro sensor 22 is located at the position of headphones 31b to 31d.

In the example of FIG. 3, speakers 32a to 32h are arranged in the interior space of the vehicle 1. Each of the speakers 32a to 32h is an example of the speaker 32 in FIG. 2. The speakers 32a to 32h are arranged, for example, as follows. That is, the speaker 32a is arranged in the front right door of the vehicle 1. The speaker 32b is arranged in the front left door of the vehicle 1. The speaker 32c is arranged in the rear right door of the vehicle 1. The speaker 32d is arranged in the rear left door of the vehicle 1. The speaker 32e is arranged in the instrument panel or the like in the front of the vehicle 1. The speaker 32f is arranged in the center part of the ceiling inside the vehicle 1. The speaker 32g is arranged behind the seat ST3. The speaker 32h is arranged behind the seat ST4. The number and arrangement positions of the speakers 32 can be changed as appropriate.

(Headphones 31 and speakers 32)
The description will be given again with reference to Fig. 2. The headphones 31 are headphones having a configuration capable of adjusting the position of the sound image of the output sound. The headphones 31 output sound according to the control of the control device 10. For example, the headphones 31 may be a device equipped with a plurality of speakers (e.g., nine speakers) therein. In this case, the sound image position of the output sound from the headphones 31 can be adjusted by adjusting the volume and output timing of each speaker included in the headphones 31.

The speaker 32 is a speaker having a configuration that allows the position of the sound image of the output sound to be adjusted. The speaker 32 outputs sound according to the control of the control device 10. The speaker 32 forms a sound image either alone or by multiple speakers working together.

(Storage device 30)
The storage device 30 stores various data necessary for the operation of the control device 10. The storage device 30 may also store sound source data to be output to the headphones 31 and/or the speaker 32. More specifically, the sound source data may be data such as music, a navigation voice for providing guidance or information, and a warning sound such as a buzzer. The storage device 30 may be an external device to the control device 10 as shown in FIG. 2, or may be a storage unit built into the control device 10.

(Control device 10)
The control device 10 controls the headphones 31 and the speaker 32 based on the information acquired from the detection device 20. The control device 10 includes an information acquisition unit 11, a coordinate processing unit 12, a sound image position determination unit 13, and an output control unit .

The information acquisition unit 11 receives position information and orientation information from various devices of the detection device 20. The information acquisition unit 11 outputs the received position information and orientation information to the coordinate processing unit 12.

The coordinate processing unit 12 identifies the position and orientation of each occupant's head as a numerical value or vector on three-dimensional coordinates representing the interior space of the vehicle 1. The coordinate processing unit 12 includes a position identification unit 121 and a direction identification unit 122. Hereinafter, the "three-dimensional coordinates representing the interior space of the vehicle 1" will be simply referred to as "in-vehicle space coordinates." Note that, while a "vector" generally has a magnitude and a direction, in this disclosure, the magnitude of the vector indicating the head orientation may be any value as long as the directional component can be identified.

The position identification unit 121 identifies the head position of each occupant in the vehicle interior space coordinates from the position information. The position identification unit 121 plots the identified head position in the vehicle interior space coordinates. Note that the head position may be indicated by the coordinates of a single point or may be indicated by a range of coordinate values.

The direction identification unit 122 identifies the head direction of each occupant in the vehicle interior space coordinates from the orientation information. For example, the direction identification unit 122 plots the head direction of each occupant in the vehicle interior space coordinates as a vector in the vehicle interior space coordinates. The method of identifying the head position in the position identification unit 121 and the method of identifying the head direction in the direction identification unit 122 may be determined appropriately depending on the type and content of the information input to the coordinate processing unit 12.

For example, suppose an image captured by camera 21 is input to coordinate processing unit 12. In this case, position identification unit 121 identifies the position of the occupant's head in the interior space of vehicle 1 from the positions and shapes of windows, doors, and seats captured in the captured image, and the position and shape of the user's head captured in the same image. Similarly, direction identification unit 122 identifies the direction of the occupant's head in the interior space of vehicle 1 from the positions and shapes of windows, doors, and seats captured in the captured image, and the position and shape of the user's head captured in the same image.

For example, assume that measurement information from the gyro sensor 22 is input to the coordinate processing unit 12 in addition to the captured image. In this case, the direction identification unit 122 may identify the direction of the head of each occupant from the measurement information from the gyro sensor 22. Alternatively, the direction identification unit 122 may identify the direction of the head of each occupant by taking into account both the measurement information from the gyro sensor 22 and the captured image from the camera 21.

The head position and head direction of each occupant may be identified in random order or in parallel. The position identification unit 121 and the direction identification unit 122 may refer to each other's processing results or information in the middle of processing in order to identify which occupant the information identified by each unit pertains to.

The coordinate processing unit 12 outputs the head position coordinates and head direction vector of each identified occupant to the output control unit 14. The coordinate processing unit 12 may also output the head position coordinates and head direction vector of each identified occupant to the sound image position determination unit 13. The coordinate processing unit 12 may plot the head position coordinates of each occupant identified by the position identification unit 121 on the vehicle interior space coordinates. The coordinate processing unit 12 may also plot the head direction vector of each occupant identified by the direction identification unit 122 on the vehicle interior space coordinates. The coordinate processing unit 12 may then output information on the vehicle interior space coordinates on which the head positions and head directions of each occupant are plotted to the output control unit 14 (and the sound image position determination unit 13).

The coordinate processing unit 12 may also identify the number of occupants from the number of coordinates (i.e., the position coordinates of the occupants' heads) identified by the position identification unit 121. The coordinate processing unit 12 may also output information indicating the identified number of occupants to the sound image position determination unit 13 and the output control unit 14.

The sound image position determination unit 13 determines the position coordinates on the in-vehicle spatial coordinates of the sound image to be perceived by each occupant. The "sound image to be perceived by each occupant" refers to the sound image that each occupant of the vehicle 1 (in this embodiment, four occupants A to D) feels when hearing sound from their corresponding audio output device (headphones 31 or speakers 32). The sound image position determination unit 13 may determine the position of the sound image according to the position and direction of each occupant's head input from the coordinate processing unit 12. The sound image position determination unit 13 outputs the determined position coordinates of the sound image to the output control unit 14. The position coordinates of each sound image may be set to a single point or may be indicated by a range of coordinate values.

The sound image position determination unit 13 according to this embodiment determines the position coordinates of the sound images of the headphones 31 and the speaker 32 to be one (or one range). In other words, the sound image position determination unit 13 determines the position of the sound image so that the sound image positions of the output sounds of the headphones 31 and the speaker 32 are in the same position.

The output control unit 14 determines the headphones 31 and/or speakers 32 for which the position of the sound image is to be controlled, and controls the output of the determined headphones 31 and/or speakers 32. When the position of the sound image is input from the sound image position determination unit 13, the output control unit 14 first determines the headphones 31 and/or speakers 32 that will form a sound image at that position. In other words, the output control unit 14 determines the headphones 31 and/or speakers 32 to be controlled. Note that the output control unit 14 may determine only some of the headphones 31 and/or speakers 32 to be controlled, or may determine all of the headphones 31 and/or speakers 32 to be controlled.

Next, the output control unit 14 controls each of the headphones 31 and/or speakers 32 so that a sound image of the output sound of the controlled headphones 31 and/or speakers 32 is formed at a specified position. More specifically, the output control unit 14 controls each of the headphones 31 and/or speakers 32 based on the relative positional relationship between the position and orientation of each occupant's head and the position of the sound image. This makes it possible to form sound images of the output sound of the headphones 31 and/or speakers 32 that are present at different positions and in different orientations on the vehicle interior spatial coordinate system at the respective target positions.

The coordinate processing unit 12 may determine which headphones 31 each occupant is wearing and which speaker 32 each occupant can hear from, based on the position information and orientation information of the detection device 20. For example, the coordinate processing unit 12 may determine which headphone 31 or speaker 32 each occupant can hear by comparing the image captured by the camera 21 with the measurement information of the gyro sensor 22.

In this embodiment, occupants B to D are wearing headphones 31b to 31d, respectively. Therefore, the sound images heard by occupants B to D are the output sounds from headphones 31b to 31d, respectively. On the other hand, occupant A is not wearing headphones 31. Therefore, the sound image perceived by occupant A is the output sound from one or more of speakers 32a to 32h.

The output control unit 14 only needs to execute at least the above-mentioned output control for the headphones 31 and/or the speaker 32, and does not need to be involved in determining the audio data itself to be output from the headphones 31 and/or the speaker 32. Specifically, the audio to be output from the headphones 31 and/or the speaker 32 may be determined by the output control unit 14, or may be determined by an external device such as a music player (not shown) connected to the headphones 31 and/or the speaker 32.

<Examples of vehicle interior space coordinates>
Fig. 4 is a diagram showing the relationship between the position and direction of the head of the passenger and the position of the sound image in the vehicle interior space coordinate system in this embodiment. The definitions of front, rear, left, right, top and bottom of the vehicle 1 in Fig. 4 are the same as those in Fig. 3. In addition, in the vehicle interior space coordinate system in Fig. 4, as shown in the figure, the left front point of the vehicle 1 is set as the origin, the axis extending from the left direction to the right direction is the x axis, and the axis perpendicular to the x axis and extending from the front to the rear is the y axis. In addition, the axis extending from the bottom (floor side) to the top (ceiling side) of the vehicle 1 in Fig. 4 and perpendicular to the x axis and y axis is the z axis.

In the example of Figure 4, the head positions of occupants A to D, the head orientations of occupants A to D, and the positions of sound images are shown diagrammatically in the vehicle interior space coordinate system.

The output control unit 14 controls the output of headphones 31b in accordance with the relative positional relationship between the position of occupant B's head (i.e., the position of headphones 31b) and range R1, and the orientation of occupant B's head (i.e., the orientation of headphones 31b), so that the sound image of the output sound of headphones 31b (i.e., the sound heard by occupant B) is formed in range R1. Similarly, the output control unit 14 controls the output of headphones 31c in accordance with the relative positional relationship between the position of occupant C's head and range R1, and the orientation of occupant C's head, so that the sound image of the output sound of headphones 31b is formed in range R1. The same applies to headphones 31d.

The output control unit 14 also controls the output of each speaker 32 according to the relative positional relationship between the position of each speaker 32 and the position of range R1, and the orientation of each speaker 32. Note that the output control unit 14 may cause multiple speakers 32 to cooperate with each other to form a sound image at the position of range R1.

<Processing flow>
Fig. 5 is a flowchart showing an example of a flow of processes executed by the control device 10 according to the present embodiment. The series of processes shown in Fig. 5 may be started at any timing. For example, the series of processes shown in Fig. 5 may be started at a stage when the detection device 20 detects the position information and the orientation information of any occupant.

The information acquisition unit 11 of the control device 10 acquires position information and orientation information from the detection device 20 (S1, information acquisition step). In this embodiment, the information acquisition unit 11 acquires an image captured by the camera 21 (position information and orientation information) and measurement information from the gyro sensor 22 (orientation information). The information acquisition unit 11 outputs the acquired position information and orientation information to the coordinate processing unit 12.

The coordinate processing unit 12 identifies the head position and head direction of each occupant from the position information and orientation information (S2). The coordinate processing unit 12 outputs the identified head position and head direction to the output control unit 14 (and the sound image position determination unit 13). The coordinate processing unit 12 may plot the identified coordinates of the head position and the vector of the head direction on the in-vehicle spatial coordinate system in a format such as that shown in FIG. 4 (S3).

The sound image position determination unit 13 determines the position of the sound image of the output sound of the headphones 31 and the speaker 32 to a certain coordinate or a certain coordinate range on the vehicle interior spatial coordinate system. That is, the sound image position determination unit 13 determines the position of the sound image so that the sound images of the output sounds of the headphones 31 and the speaker 32 are in the same position (S4, sound image position determination step). The sound image position determination unit 13 outputs the determined position coordinate or coordinate range of the sound image to the output control unit 14.

The output control unit 14 determines the audio output devices to be controlled (S5). In this embodiment, the output control unit 14 determines, for example, three headphones 31, 31b to 31d, and two speakers 32, 32a and 32e, as the devices to be controlled.

The output control unit 14 controls the output of the headphones 31 and speakers 32 determined to be controlled based on the position and orientation of each occupant's head. As a result, the output control unit 14 causes the headphones 31 and speakers 32 to form a sound image at the position determined by the sound image position determination unit 13 (S6, output control step).

[Embodiment 2]
Other embodiments of the present disclosure will be described below. For the sake of convenience, the same reference numerals will be used to designate components having the same functions as those described in the previous embodiment, and the description thereof will not be repeated. This also applies to the following embodiments.

Overview
The control system 200 of this embodiment is characterized in that it controls multiple audio output devices (i.e., multiple headphones, or one or more headphones and one or more speakers) respectively, thereby individually forming sound images of the output sounds from the multiple audio output devices.

Figure 6 is a diagram showing an overview of the operation of the control system 200 according to this embodiment. Figure 6 corresponds to Figure 1 in embodiment 1. Therefore, the same points as Figure 1 will not be described repeatedly.

In the example of FIG. 6, the control system 200 controls each of these audio output devices so that the sound image of the output sound from headphone 31b is formed in range A, the sound image of the output sound from headphone 31c is formed in range B, the sound image of the output sound from headphone 31d is formed in range C, and the sound images of the output sounds from speakers 32a and 32e are formed in range D. In other words, each audio output device is controlled so that the sound image of the output sound from each headphone 31 and the sound image formed by one or more speakers 32 are formed individually. This allows occupants A to D to feel as if they are hearing sounds from a position appropriate to each individual.

<Examples of vehicle interior space coordinates>
Fig. 7 is a diagram showing the relationship between the position and direction of the head of an occupant and the position of a sound image in the present embodiment on the vehicle interior space coordinate system. Fig. 7 corresponds to Fig. 4 in the first embodiment. Therefore, the description of the same points as Fig. 4 will not be repeated. In the control system 200 according to the present embodiment, the sound image position determination unit 13 determines the position of the sound image formed by the output sound of each headphone 31 and speaker 32 individually for each occupant (i.e., for each headphone 31 and for each group of speakers 32) as shown in ranges A to D in Fig. 7.

<Processing flow>
Fig. 8 is a flowchart showing an example of the flow of processing executed by the control device 10 according to this embodiment. In the flowchart of Fig. 8, the processing of S11 to S13 is similar to the processing of S1 to S3 in Fig. 5, respectively. However, the coordinate processing unit 12 according to this embodiment also outputs the head position and head orientation of each occupant to the sound image position determination unit 13.

The sound image position determination unit 13 determines the position of the sound image to be perceived by each occupant individually according to the head position and head orientation of each occupant (S14, sound image position determination step). The sound image position determination unit 13 outputs information indicating the determined position of the sound image (range A to range D in the example of Figure 7) to the output control unit 14.

The output control unit 14 determines the audio output device to be controlled (S15), similar to S5 in FIG. 5. Then, the output control unit 14 controls the output of the headphones 31 and speakers 32 determined to be the control targets, based on the position and orientation of each occupant's head. As a result, the output control unit 14 causes the headphones 31 and speakers 32 to form a sound image at the position determined by the sound image position determination unit 13 (S16, output control step).

[Embodiment 3]
Overview
The control system 300 according to this embodiment outputs the speech of the occupant from a plurality of audio output devices (i.e., a plurality of headphones, or one or more headphones 31 and one or more speakers 32). The control system 300 is also characterized in that it forms a sound image of the output sound (i.e., the speech) from the plurality of audio output devices in the vicinity of the speaker.

Figure 9 is a diagram showing an overview of the operation of the control system 300 according to this embodiment. Figure 9 corresponds to Figure 1 in embodiment 1. Therefore, the same points as Figure 1 will not be described repeatedly.

In the example of FIG. 9, the control system 300 includes microphones 23a to 23d as the detection device 20. These microphones are arranged on the ceiling of each seat of the vehicle 1, for example as shown in FIG. 9, and capture the speech of the occupant sitting in each seat. In the example of FIG. 9, the speaker is occupant B, as shown in the figure. Microphone 23b captures the speech of occupant B.

The control system 300 reproduces the speech captured by the microphone 23b through the headphones 31b-d and the speakers 32a and 32e. At this time, the control system 300 controls the headphones 31b-31d and the speakers 32a and 32e so that the sound images of the output sounds of the headphones 31b-31d and the speakers 32a and 32e are all formed in a position in the range R2 near the speaker, the occupant B. This makes it possible to make the speech of occupant B easier to hear for occupants A-D, including the speaker, occupant B. In particular, the speech of occupant B can be heard as the output sound of the headphones 31 for occupants B-D who are wearing the headphones 31, making it easier for occupants B-D to hear the speech.

<Main components>
10 is a diagram showing the configuration of a main part of a control system 300 according to this embodiment. The control system 300 differs from the control systems 100 and 200 in that the control system 300 includes a microphone 23.

The microphone 23 acquires the speech of each occupant in real time. The microphone 23 transmits the acquired speech to the information acquisition unit 11 at any time. When the information acquisition unit (speech acquisition unit) 11 acquires speech from the microphone 23, it outputs identification information of the microphone from which the speech was acquired to the sound image position determination unit 13. In addition, the information acquisition unit 11 outputs the speech acquired from the microphone 23 to the output control unit 14.

The sound image position determination unit 13 determines the sound image position based on the identification information of the microphone 23 and the position of the occupant's head input from the coordinate processing unit 12. Specifically, the sound image position determination unit 13 determines the position of the sound image to be a position near the occupant who emitted the speech voice captured by the microphone 23 (i.e., the speaker).

The output control unit 14 causes the audio output device determined as the controlled object to output the spoken voice. At this time, the output control unit 14 causes the controlled object audio output device to form a sound image at the position determined by the sound image position determination unit 13 (i.e., a position near the speaker).

<Examples of vehicle interior space coordinates>
Fig. 11 is a diagram showing the relationship between the position of the head of an occupant and the position of a sound image in the present embodiment on the vehicle interior space coordinate system. Fig. 11 corresponds to Fig. 4 in the first embodiment. Therefore, the description of the same points as Fig. 4 will not be repeated. The output control unit 14 controls each of the headphones 31 so that the position of the sound image of the speech sound output from the headphones 31b to 31d is within a position in range R2. The output control unit 14 also controls each of the speakers 32 so that the position of the sound image of the speech sound output from each of the speakers 32 is within a position in range R2.

<Processing flow>
12 is a flowchart showing an example of the flow of processing executed by the control device 10 according to the present embodiment. In the flowchart of FIG. 12, the processing in S21 to S23 is similar to the processing in S1 to S3 in FIG.

The sound image position determination unit 13 determines whether the occupant has spoken (S24). For example, when the sound image position determination unit 13 acquires identification information of the microphone 23 from the information acquisition unit 11, it may determine that the occupant has spoken.

If the sound image position determination unit 13 determines that the occupant is not speaking (NO in S24), the sound image position determination unit 13 does not perform any further processing, and the output control unit 14 does not need to execute any particular output control. Alternatively, the sound image position determination unit 13 may determine the sound image positions of the headphones 31 and speakers 32 as shown in the first or second embodiment. In this case, the output control unit 14 controls the headphones 31 and speakers 32, respectively, so that sound images are formed at the determined positions.

On the other hand, if the sound image position determination unit 13 determines that the occupant is speaking (YES in S24), the sound image position determination unit 13 determines a position near the speaker as the position of the sound image (S25, sound image position determination step). The sound image position determination unit 13 outputs the determined position of the sound image to the output control unit 14.

The output control unit 14 determines the audio output device to be controlled (S26). The output control unit 14 controls the output settings of the headphones 31 and the speaker 32 so that a sound image of the speech voice is formed at the position determined by the sound image position determination unit 13 (S27, output control step). The output control unit 14 then transmits the speech voice input from the information acquisition unit 11 to the headphones 31 and the speaker 32, thereby causing the headphones 31 and the speaker 32 to output the speech voice (S28). In this way, the output control unit 14 can cause the headphones 31 and the speaker 32 to form a sound image of the speech voice at the position determined by the sound image position determination unit 13.

[Modifications]
(Variation 1: Controlling the sound image when playing a specific sound)
In the control device 10 according to each embodiment, the sound image position determination unit 13 may determine the position of the sound image of each headphone 31 (and speaker 32) when a specific sound is output to multiple headphones 31, or to one or more headphones 31 and one or more speakers 32. Then, the output control unit 14 may control the output of the headphones 31 (and speaker 32) so that the sound image of the specific sound is formed at the position determined by the sound image position determination unit 13.

The specific content of the "specific sound" is not particularly limited as long as it is a sound that is reproduced by the output control unit 17 of the control device 10 instructing the headphones 31 and the speaker 32. For example, the specific sound may be background music reproduced by the headphones 31 and the speaker 32, or a warning sound notifying an abnormality in the vehicle 1, a voice providing road information, or other sound that is output together with the background music or by interrupting the output of the background music. In this modified example, the storage device 30 may store audio data of the specific sound to be reproduced by the headphones 31 (and the speaker 32). The output control unit 17 may then read out the audio data from the storage device 30 and transmit it to the headphones 31 and the speaker 32. According to this modified example, when the sound image of the specific sound is output from the headphones 31 (and the speaker 32), the position of the sound image of the output sound can be controlled. For example, it is assumed that background music is output from the headphones 31 and the speaker 32. In this state, if a specific sound, a warning sound, is output to the headphones 31 and the speaker 32, the position of the sound image of this warning sound can be controlled.

(Variation 2: Output of voice data according to the state of the occupant)
In the control systems 100 to 300 according to each embodiment, the detection device 20 may include a status sensor that measures or acquires status information for identifying the status of each of the occupants A to D. The status sensor is, for example, a biosensor, such as a heart rate monitor, for measuring vital data of the occupants. Note that the status of the occupants A to D can also be determined from the image captured by the camera 21. Therefore, the camera 21 may also function as the status sensor.

The information acquisition unit 11 of this modified example acquires status information of each occupant from a status sensor. The output control unit (audio determination unit) 14 of the control device 10 determines the audio to be output from each headphone 31 and speaker 32 based on the status information, in addition to controlling the sound image of the headphones 31 (and the speaker 32). The output control unit 14 then outputs the determined audio from the headphones 31 and the speaker 32. At this time, the output control unit 14 controls the position of the sound image of the output sound, as described in each embodiment. This makes it possible to output content according to the status of each occupant to the headphones 31 (and the speaker 32). The audio data that the output control unit 14 outputs to the headphones 31 (and the speaker 32) may be stored in the storage device 30.

The control device 10 may analyze the state information acquired by the information acquisition unit 11 to determine the level of drowsiness, the level of discomfort, etc. of the occupant. The output control unit 14 may then output content corresponding to the determined state from the headphones 31 (and the speakers 32). For example, if it is analyzed that a certain occupant is very drowsy, a warning sound may be output from the headphones 31 corresponding to that occupant.

(Modification 3: Method of Acquiring Audio Data)
The control device 10 according to each embodiment may include a communication unit for performing Internet communication and/or a receiving antenna for receiving broadcast waves of a television, a radio, etc. The output control unit 14 may output, to the headphones 31 and/or the speaker 32, audio data downloaded via the Internet and/or audio data included in the broadcast waves received by the receiving antenna.

(Modification 4: When a timer is provided)
The control systems 100 to 300 according to the respective embodiments may include a boarding/alighting sensor that detects whether or not an occupant has boarded/alighted the vehicle 1, and a timing unit that measures time. The control device 10 of the control systems 100 to 300 may determine the audio data to be output from the headphones 31 and/or the speaker 32 according to the riding time of each occupant measured by the timing unit.

In this modified example, the boarding/alighting sensor and the timing unit are connected to the control device 10 by wire or wirelessly. The timing unit may be built into the control device 10. When the boarding/alighting sensor detects that an occupant has boarded the vehicle 1, it notifies the control device 10 that the occupant has boarded. When the control device 10 receives this notification, it starts measuring time using the timing unit.

The storage device 30 may store audio data to be output by the headphones 31 and/or the speaker 32. The output control unit 14 of the control device 10 causes the headphones 31 and/or the speaker 32 to output audio corresponding to the time measured by the timer unit (i.e., the passenger's riding time) at any timing.

For example, when the ride time exceeds a predetermined time (e.g., one hour), the output control unit 14 reads data of a guidance voice encouraging the passenger to take a break from the storage device 30 and outputs this data from the headphones 31 and the speaker 32. Note that, if the sound image position determination unit 13 has determined the position of the sound image, the output control unit 14 may also control the position of the sound image when outputting the guidance voice.

When the boarding/alighting sensor detects that an occupant has disembarked, it notifies the control device 10. When the control device 10 receives this notification, it stops the measurement of the timing unit.

[Software implementation example]
The functions of the control device 10 (hereinafter referred to as the "device") can be realized by a program for causing a computer to function as each control block of the device.

In this case, the device is a computer having, as hardware for executing the program, at least one memory for expanding the program and at least one processor for executing the arithmetic processing described in the program. The program is executed by the memory and the processor in this manner, thereby realizing each of the functions described in each of the above embodiments.

The processor may be, for example, a CPU (Central Processing Unit). The program may be recorded on one or more computer-readable recording media, not on a temporary basis. The recording media may or may not be included in the device. In the former case, the recording media may be, for example, a ROM (Read Only Memory), tape, disk, card, or semiconductor memory. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

In addition, some or all of the functions of each of the control blocks can be realized by a logic circuit. For example, the scope of this disclosure also includes an integrated circuit in which a logic circuit that functions as each of the control blocks is formed. In addition, it is also possible to realize the functions of each of the control blocks by, for example, a quantum computer.

This disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of this disclosure.

Reference Signs List 1 Vehicle 100, 200, 300 Control system 10 Control device 11 Information acquisition unit 12 Coordinate processing unit 121 Position identification unit 122 Direction identification unit 13 Sound image position determination unit 14 Output control unit 20 Detection device 21 Camera 22 Gyro sensor 23 Microphone 30 Storage device 31 Headphones 32 Speaker

Claims (5)

an information acquisition unit that acquires, from a detection device, position information indicating a head position and orientation information indicating a head orientation for each of a plurality of users staying in a predetermined space;
a sound image position determination unit that determines a position of a sound image of an output sound of a first output device worn by each of the plurality of users;
an output control unit that controls an output of each of the first output devices based on a head position and a head orientation of each of the plurality of users so that a sound image of the output sound is formed at a position determined by the sound image position determination unit;
a voice acquisition unit that acquires a voice of any one of the users from a microphone that is arranged in the predetermined space or that is worn by the user.
the sound image position determination unit determines the positions of the sound images of the output sounds of the first output devices so that the positions of the sound images are the same;
When the voice acquisition unit acquires the voice of the utterance,
the sound image position determination unit determines a position of the sound image based on a head position and a head orientation of the user who has spoken such that a vicinity of the user who has spoken becomes a position of a sound image of the output sound of each of the first output devices;
The control device , wherein the output control unit causes each of the first output devices to output the speech acquired by the speech acquisition unit . the sound image position determination unit determines a position of a sound image of each of the first output devices when causing the first output devices to output a specific sound;
The control device according to claim 1 , wherein the output control unit controls the output of each of the first output devices so that a sound image of the specific sound is formed at a position determined by the sound image position determination unit. the detection device includes a status sensor for measuring or acquiring status information for identifying a status of each of the plurality of users;
The information acquisition unit acquires the status information of each user,
The control device according to claim 1 , further comprising a sound determination unit that determines an output sound of the first output device based on the state information of each of the users. A control device according to any one of claims 1 to 3 ;
The detection device;
a plurality of said first output devices. an information acquisition step of acquiring, from a detection device, position information indicating a head position and orientation information indicating a head orientation for each of a plurality of users staying in a predetermined space;
a sound image position determining step of determining positions of sound images of output sounds of the first output devices respectively worn by the plurality of users;
an output control step of controlling an output of each of the first output devices based on a head position and a head orientation of each of the plurality of users so that a sound image of the output sound is formed at the position determined in the sound image position determination step;
a voice acquisition step of acquiring a voice of any one of the plurality of users from a microphone disposed in the predetermined space or worn by the user ,
In the sound image position determination step, a position of the sound image is determined so that a position of the sound image of each of the first output devices is the same;
When the voice of the utterance is acquired in the voice acquisition step,
in the sound image position determination step, a position of the sound image is determined based on a head position and a head orientation of the user who has spoken so that a vicinity of the user who has spoken becomes a position of a sound image of each of the first output devices;
The control method , wherein the output control step causes each of the first output devices to output the voice of the utterance acquired in the voice acquisition step .

Images