JP7611682B2 - Information processing device, information processing method, and program - Google Patents

Description

The disclosed embodiments relate to an information processing device, an information processing method , and a program.

Conventionally, there is known technology that provides users with digital content that includes virtual space experiences, such as VR (Virtual Reality) and MR (Mixed Reality), using a head mounted display (HMD) or the like.

Furthermore, in this technology, a VR system has been proposed that can be mounted on a moving object such as a vehicle and use the moving object as a motion platform (see, for example, Patent Document 1).

JP 2017-102401 A

However, conventional technology has room for further improvement in preventing users from becoming motion sickness-prone due to digital content, including virtual world experiences.

For example, it is known that users who receive VR content can experience "VR sickness," which is similar to motion sickness. VR sickness is a type of motion sickness that can occur due to factors such as a missynchronization of audio and video, or large fluctuations in audio and video that the brain cannot keep up with. In particular, with VR systems installed in vehicles, users' sickness tends to become even worse because it also involves regular motion sickness.

One aspect of the embodiment has been made in consideration of the above, and aims to provide an information processing device, information processing method, program, and data structure that can prevent users from becoming motion sickness caused by digital content that includes virtual space experiences.

An information processing device according to an aspect of an embodiment includes a controller . The controller acquires indoor and outdoor situations related to a user of digital content including a virtual space experience. The controller estimates a sickness level of the user based on the acquired situations. The controller executes a sickness suppression process related to audio of the digital content in accordance with the estimated sickness level of the user. The controller executes a process of reducing a volume change rate of the audio of the digital content in the suppression process.

According to one aspect of the embodiment, it is possible to prevent users from becoming motion sickness caused by digital content that includes a virtual space experience.

FIG. 1 is a diagram showing a schematic configuration of an information processing system according to an embodiment. FIG. 2 is an explanatory diagram of VR sickness. FIG. 3 is a diagram illustrating an outline of an information processing method according to an embodiment. FIG. 4 is a block diagram illustrating an example of the configuration of an information processing system according to the embodiment. FIG. 5 is a diagram showing the process contents of the suppression process. FIG. 6 is a diagram illustrating an example of the suppression process information. FIG. 7 is a flowchart showing a processing procedure executed by the information processing apparatus according to the embodiment.

Below, embodiments of the information processing device, information processing method, program, and data structure disclosed in the present application will be described in detail with reference to the attached drawings. Note that the present invention is not limited to the embodiments described below.

In the following, the information processing system 1 according to the embodiment will be described as an example in which the information processing system 1 is an in-vehicle system mounted in a vehicle. In the following, the information processing system 1 according to the embodiment will be described as a VR system that provides VR content to a user as digital content that includes a virtual space experience.

First, an overview of the information processing method according to the embodiment will be described with reference to Figs. 1 to 3. Fig. 1 is a diagram showing a schematic configuration of an information processing system 1 according to the embodiment. Fig. 2 is an explanatory diagram of VR sickness. Fig. 3 is an explanatory diagram showing the overview of the information processing method according to the embodiment.

As shown in FIG. 1, the information processing system 1 according to the embodiment includes an HMD 3 and an information processing device 10.

The HMD 3 is an information processing terminal that presents VR content provided by the information processing device 10 to the user U, allowing the user to enjoy a VR experience. The HMD 3 is a wearable computer that is worn on the head of the user U, and is goggle-type in the example of FIG. 1. The HMD 3 may be glasses-type or hat-type.

The HMD 3 includes a display unit 31, a speaker 32, and a sensor unit 33. The display unit 31 is arranged to be placed in front of the eyes of the user U, and displays images included in the VR content provided by the information processing device 10.

In the example of FIG. 1, the display unit 31 is provided in front of each of the left and right eyes of the user U, but there may be only one. The display unit 31 may be a non-transparent type that completely covers the field of view, or may be a video-transparent type or an optical-transparent type. In this embodiment, the display unit 31 is a non-transparent type.

The speaker 32 is provided, for example, in a headphone type as shown in FIG. 1, and is worn on the ear of the user U. The speaker 32 outputs the sound contained in the VR content provided by the information processing device 10.

The sensor unit 33 is a device that detects changes in the inside and outside conditions of the user U, and includes, for example, a camera and a motion sensor.

The information processing device 10 is, for example, a computer, an in-vehicle device mounted in a vehicle, connected to the HMD 3 by wire or wirelessly, and provides VR content to the HMD 3. The information processing device 10 also acquires changes in the situation detected by the sensor unit 33 at any time, and reflects such changes in the situation in the VR content.

For example, the information processing device 10 can change the direction of the field of view in the virtual space of the VR content in response to changes in the head or line of sight of the user U detected by the sensor unit 33.

However, it is known that when VR content is provided using such an HMD 3, the user U may experience "VR sickness," which is similar to motion sickness.

As shown in Figure 2, VR sickness can occur due to factors such as a mismatch between audio and video synchronization, or large fluctuations in audio and video that the brain cannot keep up with. VR sickness can also occur due to a mismatch between the sensations experienced through the VR experience using the HMD 3 and the user U's own physical sensations due to changes in the surrounding environment.

Therefore, in the information processing method according to the embodiment, the indoor and outdoor conditions related to the user U are acquired, the VR sickness state of the user U is estimated based on the acquired conditions, and processing for suppressing VR sickness related to audio is performed according to the estimated VR sickness state.

Specifically, as shown in FIG. 3, in the information processing method according to the embodiment, the information processing device 10 acquires indoor and outdoor conditions related to the user U at any time and estimates the VR sickness state of the user U (step S1). The information processing device 10 estimates the VR sickness state by, for example, detecting a change in the physical condition of the user U.

In addition, the information processing device 10 estimates the VR sickness state based on the usage status of the VR content, such as the type of VR content being provided, the video status, and the audio status.

The information processing device 10 also estimates the VR sickness state based on the vehicle's driving conditions, such as road conditions, vehicle conditions, and operation conditions. The information processing device 10 also estimates the VR sickness state based on user information including various parameters indicating each user's susceptibility to sickness, for example.

In addition, in the process of estimating such a VR sickness state, the information processing device 10 can use an estimation model generated using, for example, a machine learning algorithm. Such an estimation model is appropriately reinforced learning based on the estimation result of the actual VR sickness state. As a result of the reinforcement learning, for example, a judgment threshold for estimating the VR sickness state, etc., is appropriately updated.

Then, the information processing device 10 executes a process for suppressing VR sickness related to the sound of the VR content according to the estimation result in step S1 (step S2). The process for suppressing VR sickness related to the sound is generally a process for weakening the stimuli related to the sound of the VR content, for example, a process for reducing the sense of sound image localization.

The process for suppressing VR sickness related to audio is, for example, a process for decreasing the rate of change in volume. The process for suppressing VR sickness related to audio is, for example, a process for reducing the range of change in volume. Details of these processes for suppressing VR sickness related to audio will be described later using FIG. 5 etc.

In this way, by weakening the stimuli related to the sound of the VR content, it is possible to mitigate at least the tendency for sickness to be caused by the sound of the VR content. In other words, this can help prevent user U from getting VR sickness due to the VR content.

Note that in the explanation of FIG. 3, processing to suppress VR sickness related to audio is performed, but processing to suppress VR sickness related to video may also be used in conjunction with this. Processing to suppress VR sickness related to video generally reduces the stimulation related to the video of the VR content, for example by reducing contrast, color, etc.

As described above, the information processing method according to the embodiment acquires indoor and outdoor conditions related to the user U, estimates the user U's VR sickness state based on the acquired conditions, and performs processing to suppress VR sickness related to audio according to the estimated VR sickness state.

Therefore, according to the information processing method of the embodiment, it is possible to prevent the user U from getting VR sickness due to the VR content. Below, a more specific description will be given of an example configuration of the information processing system 1 to which the information processing method of the embodiment is applied.

Figure 4 is a block diagram showing an example of the configuration of an information processing system 1 according to an embodiment. Note that Figure 4 shows only the components necessary to explain the features of the embodiment, and omits a description of general components.

In other words, each component shown in FIG. 4 is a functional concept, and does not necessarily have to be physically configured as shown. For example, the specific form of distribution and integration of each block is not limited to that shown, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

In addition, in the explanation using Figure 4, the explanation of components that have already been explained may be simplified or omitted.

As shown in FIG. 4, the information processing system 1 according to the embodiment includes an HMD 3 and an information processing device 10.

The HMD 3 has already been described using FIG. 1, so a description thereof will be omitted here. The information processing device 10 includes a storage unit 11 and a control unit 12. In addition, various sensors 5 are connected to the information processing device 10.

The various sensors 5 are a group of sensors that sense the conditions inside and outside the vehicle, and include, for example, a camera 5a, a vital sign sensor 5b, an acceleration sensor 5c, a steering angle sensor 5d, etc.

The camera 5a is a front camera, rear camera, side camera, interior camera, etc., mounted on the vehicle, and captures images of the inside and outside of the vehicle. The interior camera, for example, captures images of the user U's condition.

The vital sensor 5b is a sensor that detects the physical condition of the user U, and is, for example, worn by the user U to measure vital data such as the user U's heart rate, brain waves, blood oxygen concentration, and sweating.

The acceleration sensor 5c measures the acceleration and vehicle speed applied to the vehicle. The steering angle sensor 5d measures the steering angle of the vehicle. Of course, the various sensors 5 may include sensors other than the sensors 5a to 5d shown in FIG. 4.

The storage unit 11 is realized, for example, by semiconductor memory elements such as RAM (Random Access Memory) and flash memory, and in the example of FIG. 4, it stores a VR content DB (database) 11a, user information 11b, an estimation model 11c, and suppression processing information 11d.

The VR content DB 11a is a database that stores a group of VR contents to be provided to the HMD 3. The user information 11b is information about the user who uses the HMD 3, and includes various parameters that indicate the susceptibility of each user to sickness, etc., as described above. The user information 11b is appropriately updated based on the estimation results of the user U's past VR sickness state.

The estimation model 11c is an estimation model generated using the above-mentioned machine learning algorithm. For example, the estimation model 11c outputs a value indicating the VR sickness state of the user U (for example, a level value indicating the degree of VR sickness) by inputting data indicating various internal and external situations of the user U acquired by the acquisition unit 12b described below.

The suppression processing information 11d is information that defines the VR sickness suppression processing to be performed depending on the level of VR sickness of the user U. A specific example of the suppression processing information 11d will be described later using FIG. 6.

The control unit 12 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (not shown) stored in the memory unit 11 using a RAM as a working area. The control unit 12 can also be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The control unit 12 has a providing unit 12a, an acquiring unit 12b, an estimating unit 12c, and a suppression processing unit 12d, and realizes or executes the functions and actions of information processing described below.

The providing unit 12a provides the VR content stored in the VR content DB 11a to the HMD 3. The providing unit 12a also acquires changes in the situation detected by the sensor unit 33 of the HMD 3 at any time and reflects such changes in the situation in the VR content.

The acquisition unit 12b acquires sensing data from the various sensors 5 at any time. The acquisition unit 12b also acquires from the provision unit 12a at any time information about the use of the VR content, such as the type of VR content being provided, the video status, and the audio status. The acquisition unit 12b also outputs the various acquired data to the estimation unit 12c.

The estimation unit 12c estimates the VR sickness state of the user U using the estimation model 11c based on the various data acquired by the acquisition unit 12b. The estimation unit 12c also outputs the estimated result to the suppression processing unit 12d.

The suppression processing unit 12d performs a process to suppress VR sickness related to the audio of the VR content according to the estimation result of the estimation unit 12c.

The suppression process executed by the suppression processing unit 12d will now be described in more detail with reference to Figs. 5 and 6. Fig. 5 is a diagram showing the process contents of the suppression process. Fig. 6 is a diagram showing an example of suppression process information 11d.

As shown in FIG. 5, the process of suppressing VR sickness related to the sound of VR content is, for example, a "process of reducing the sense of sound localization." In the process of reducing the sense of sound localization, the suppression processing unit 12d, for example, mixes in a non-localized sound (monaural sound). When mixing in such a non-localized sound, the suppression processing unit 12d adjusts the mixing ratio of the localized sound and the non-localized sound according to the VR sickness state.

For example, the suppression processing unit 12d increases the mixing ratio of non-localized sound as the degree of VR sickness increases. It is effective to gradually change the mixing ratio of localized sound and non-localized sound. In addition, the suppression processing unit 12d may mix random noise as non-localized sound.

In addition, in the process of reducing the sense of sound image localization, the suppression processing unit 12d mixes, for example, sounds whose localization is uncorrelated with the video. Such sounds are, for example, healing sounds or environmental sounds (such as the sound of the wind or the sound of a bonfire). When mixing such uncorrelated sounds, the suppression processing unit 12d adjusts the mixing ratio of the localized sounds and the uncorrelated sounds according to the VR sickness state.

For example, the suppression processing unit 12d increases the mixing ratio of uncorrelated sounds as the degree of VR sickness increases.

In addition, in the process of decreasing the sense of sound image localization, the suppression processing unit 12d performs, for example, localization sound processing, but prohibits the movement process. At this time, the suppression processing unit 12d allows instantaneous movement between two points, but prohibits gradual movement.

In addition, in the process of decreasing the sense of sound image localization, the suppression processing unit 12d, for example, shortens the localization movement distance. Note that at this time, the suppression processing unit 12d does not change the localization movement time.

In addition, in the process of reducing the sense of sound image localization, the suppression processing unit 12d outputs left and right sounds in opposite phases, for example. This makes the stereo effect closer to monaural, for example.

In addition, in the process of reducing the sense of sound localization, the suppression processing unit 12d, for example, emphasizes and synthesizes deep bass sounds. In addition, in the process of reducing the sense of sound localization, the suppression processing unit 12d, for example, outputs healing sounds or environmental sounds (such as the sound of the wind or the sound of a bonfire). This is the case when these sounds are not mixed with the sound of the VR content.

In addition, in the process of decreasing the sense of sound localization, the suppression processing unit 12d, for example, synthesizes the sound of BGM (i.e., constantly mixed sound) to be louder. When synthesizing the sound of such BGM to be louder, the suppression processing unit 12d adjusts the synthesis ratio according to the VR sickness state.

In addition, in the process of decreasing the sense of sound image localization, the suppression processing unit 12d switches the sound source, for example. In such a case, the suppression processing unit 12d switches the sound source, for example, from the speaker 32 of the HMD 3 to a bone conduction speaker or body sonic (not shown), etc.

The suppression processing unit 12d may also appropriately combine the sound image localization reduction processes described above.

As shown in the figure, the process for suppressing VR sickness related to the audio of VR content is, for example, a process for reducing the volume change rate. As shown in the figure, the process for suppressing VR sickness is, for example, a process for reducing the volume change range.

The sense of sound image localization can be generated by either giving the sound source itself a sense of localization or by processing the sound from the sound source. In the former case, the sense of localization can be strengthened or weakened by performing various mixing operations such as level adjustment, time and phase adjustment, etc. on each multi-track audio signal. In the latter case, the sense of localization can be strengthened or weakened by adjusting each audio processing parameter. In addition, the sound source of a single VR content itself may be recorded in advance with a data structure consisting of multiple sound source patterns with different sound image localization sense, localization movement distance, volume change range, etc., and the level of the VR sickness suppression effect may be changed by switching between these sound source patterns depending on the VR sickness state.

As shown in FIG. 6, the suppression processing information 11d defines the VR sickness suppression processing that the suppression processing unit 12d should execute depending on the degree of VR sickness. As shown in the figure, the suppression processing information 11d may further be associated with a content type.

In the example shown in the figure, the degree of motion sickness gradually increases from L1 to L3. Also, the content type indicates that it becomes increasingly easy to feel motion sickness from A to C. Examples of types that are likely to cause motion sickness include action content and horror content.

In addition, in the example of the figure, processes a, b, and c, which are suppression processes related to audio, show that the strength of the suppression process gradually increases from process a to process c. In the example of Figure 6, if the level of drowsiness is L1, the suppression processing unit 12d will execute process a only when the content type is C.

In addition, if the level of sickness is L2, the suppression processing unit 12d will execute process a if the content type is B, and process b, which is stronger than process a, if the content type is C.

In addition, if the level of sickness is L3, the suppression processing unit 12d will execute process a if the content type is A, process b if the content type is B, and process c, which is stronger than processes a and b, if the content type is C.

In addition, Figure 6 shows an example in which a process for suppressing VR sickness related to video is also defined, which shows that this can be used in conjunction with a process for suppressing VR sickness related to audio. Incidentally, processes l, m, n, and o indicate that the strength of the suppression process gradually increases from process l to process o, similar to processes a, b, and c described above.

Returning to the explanation of FIG. 4, the suppression processing unit 12d executes the suppression processing explained using FIG. 5 and FIG. 6 based on the estimation result of the estimation unit 12c and the suppression processing information 11d, and reflects the execution result in the VR content provided by the provision unit 12a to the HMD 3.

Next, the processing procedure executed by the information processing device 10 according to the embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing the processing procedure executed by the information processing device 10 according to the embodiment. Note that the processing procedure shown in FIG. 7 is repeated as needed while the providing unit 12a is providing VR content to the HMD 3.

As shown in FIG. 7, first, the acquisition unit 12b acquires indoor and outdoor conditions related to the user U (step S101). Then, the estimation unit 12c estimates the VR sickness condition of the user U based on the acquired conditions (step S102).

Then, the suppression processing unit 12d determines a suppression process for VR sickness related to audio based on the estimated result (step S103). The suppression processing unit 12d then executes the determined suppression process (step S104) and ends the process.

As described above, the information processing device 10 according to the embodiment includes an acquisition unit 12b, an estimation unit 12c, and a suppression processing unit 12d. The acquisition unit 12b acquires the indoor and outdoor conditions of a user U of the VR content (corresponding to an example of "digital content including a virtual space experience"). The estimation unit 12c estimates the VR sickness (corresponding to an example of "sickness") condition of the user U based on the conditions acquired by the acquisition unit 12b. The suppression processing unit 12d executes a process of suppressing VR sickness related to the sound of the VR content according to the VR sickness condition of the user U estimated by the estimation unit 12c.

Therefore, the information processing device 10 according to the embodiment can prevent the user U from experiencing VR sickness due to VR content. In particular, it can prevent VR sickness that can occur due to factors such as a missynchronization between audio and video, or large fluctuations in audio and video that the brain cannot keep up with.

In addition, as part of the suppression process, the suppression processing unit 12d performs a process to reduce the sense of sound image localization in the audio of the VR content.

Therefore, according to the information processing device 10 according to the embodiment, it is possible to prevent the user U from getting VR sickness caused by the VR content by performing a process of reducing the sense of sound localization without increasing the processing load, without involving video processing.

In addition, as part of the suppression process, the suppression processing unit 12d executes a process of reducing the rate of change in the volume of the audio of the VR content.

Therefore, according to the information processing device 10 according to the embodiment, the process of reducing the volume change rate without involving video processing can prevent the user U from experiencing VR sickness due to VR content without increasing the processing load.

In addition, as part of the suppression process, the suppression processing unit 12d performs a process of reducing the range of volume changes in the audio of the VR content.

Therefore, according to the information processing device 10 of the embodiment, the volume change range reduction process that does not involve video processing can prevent the user U from experiencing VR sickness due to VR content without increasing the processing load.

In addition, the sound source data of the VR content has multiple sound source patterns that differ at least in the sense of sound image localization, the localization movement distance, or the range of volume change, and the suppression processing unit 12d executes a switching process for switching the sound source patterns as the suppression process.

Therefore, according to the information processing device 10 of the embodiment, it is possible to switch, for example, the level of the VR sickness suppression effect by switching the pre-recorded sound source pattern.

In the above-described embodiment, an example was given in which the HMD 3 and the information processing device 10 are configured separately, but this is not limited thereto, and the HMD 3 and the information processing device 10 may be configured as an integrated unit.

In addition, in the above-described embodiment, the HMD 3 is given as an example of a presentation device that presents the VR content provided by the information processing device 10 to the user U, but the presentation device is not limited to this and may include the bone conduction speaker described above, or may include a vibration presentation device that presents vibrations such as BodySonic.

Furthermore, the presentation device is not limited to a wearable computer. For example, if it is a vehicle, the front window or side windows may be configured as displays, and video output may be performed to such displays. Audio output may also be performed to in-vehicle speakers. In-vehicle speakers are typically suitable for 3D playback, as multiple speakers can be appropriately positioned in multiple directions, including front, back, left and right. If the device is not a vehicle, the walls of the space in which the VR content is provided may be configured as displays, and multiple speakers may be positioned in the space in the same way as in-vehicle speakers.

In addition, in the above-described embodiment, an example was given in which the information processing device 10 provides VR content, but any digital content that includes a virtual space experience may be used, and may be AR (Augmented Reality) content or MR content.

In addition, in the above-described embodiment, an example was given in which the information processing device 10 is an in-vehicle device mounted in a vehicle, but this is not limited thereto, and the information processing device 10 may be a computer such as a game console that provides digital content including virtual space experiences.

Further advantages and modifications may readily occur to those skilled in the art. Therefore, the invention in its broader aspects is not limited to the specific details and representative embodiments shown and described above. Accordingly, various modifications may be made without departing from the spirit or scope of the general inventive concept as defined by the appended claims and equivalents thereof.

1 Information processing system 3 HMD
5 Various sensors 10 Information processing device 11 Storage unit 11a VR content DB
11b User information 11c Estimation model 11d Suppression processing information 12 Control unit 12a Providing unit 12b Acquiring unit 12c Estimating unit 12d Suppression processing unit 31 Display unit 32 Speaker 33 Sensor unit

Claims (6)

Acquire internal and external situations regarding users of digital content, including virtual space experiences;
Estimating the user's drunkenness state based on the acquired state;
execute a sickness suppression process related to the audio of the digital content in accordance with the estimated sickness state of the user ;
In the suppression process, a process of reducing a volume change rate of the audio of the digital content is executed.
An information processing device including a controller . The controller :
In the suppression process, a process of reducing a sense of sound image localization in the sound of the digital content is executed .
The information processing device according to claim 1 . The controller :
In the suppression process, a process of reducing a volume change range in the audio of the digital content is performed .
The information processing device according to claim 1 . The sound source data of the digital content includes a plurality of sound source patterns each having a different sound image localization feeling, a localization movement distance, or a volume change range,
The controller :
In the suppression process, a switching process is executed to switch the sound source pattern .
3. The information processing device according to claim 1 or 2 . Acquire internal and external situations regarding users of digital content, including virtual space experiences;
Estimating the user's drunkenness state based on the acquired state;
execute a sickness suppression process related to the audio of the digital content in accordance with the estimated sickness state of the user ;
In the suppression process, a process of reducing a volume change rate of the audio of the digital content is executed.
The information processing method carried out by the controller . Acquiring internal and external conditions regarding a user of digital content including a virtual space experience;
Estimating the user's drunkenness state based on the acquired state; and
Executing a sickness suppression process related to the audio of the digital content according to the estimated sickness state of the user ;
In the suppression process, a process of reducing a volume change rate of the audio of the digital content is executed;
A program that causes a computer to execute the following.

Images