JP7549804B2 - Emotion estimation device - Google Patents

Description

The present invention relates to an emotion estimation device, and in particular to an improvement that improves the estimation accuracy of the subject's emotional state in response to an auditory stimulus.

In the field of vehicle driving assistance control, an emotion estimation device is known that estimates the emotional state of a subject (e.g., a vehicle occupant) who has received various sensory stimuli (visual stimuli, auditory stimuli, etc.). In addition, as a driving assistance control using such an emotion estimation device, a method is known in which auditory and visual stimuli are given to a vehicle occupant so that the emotional state of the vehicle occupant becomes more favorable for driving based on the emotional state estimated by the emotion estimation device. As an example of a technology related to such emotion estimation and driving assistance control, Patent Document 1 (Patent No. 6428748) discloses a technology for estimating the emotions of a vehicle occupant using a model that simulates the hierarchical sensory information processing in the human brain and performing driving assistance control.

Patent No. 6428748

The effect of an auditory stimulus on human sensibility varies depending on the texture of the sound. In other words, how a person feels when receiving an auditory stimulus varies depending on the type of auditory stimulus (type of sound source, such as mechanical sound, natural sound, electronic sound, etc.). However, in conventional emotion estimation devices (for example, the technology disclosed in the above-mentioned Patent Document 1), emotion estimation is performed by processing the entire acquired sound data directly using a common emotion estimation formula, so the effect that differences in the type of sound source have on human sensibility is not properly reflected in the emotion estimation, and as a result, there are cases where the accuracy of the emotion estimation is not sufficient.

The present invention has been made in consideration of the above circumstances, and aims to provide an emotion estimation device that can improve the accuracy of emotion estimation in an emotion estimation device that estimates the emotion of a subject placed in a sound environment.

In order to achieve the above object, the present invention adopts the following solution. That is, as described in claim 1, an emotion estimation device for estimating an emotional state of a subject placed in a sound environment includes a sound data acquisition means for acquiring sound data in the sound environment, an estimation means for estimating an emotional state of the subject when receiving an auditory stimulus specified by the sound data based on a predetermined emotion estimation formula, a classification means for classifying the sound data into a category preset for each type of sound source based on a texture feature of the sound data, and a correction means for correcting the emotion estimation formula according to which category the sound data is classified by the classification means , and the texture feature is calculated based on a frequency component and a modulation frequency component of the sound data .

According to the above-mentioned solution, the sound data is classified into categories according to the type of sound source, and the emotion estimation formula is corrected according to the classified category, so that the effect of the type of sound source on the emotion of the subject is appropriately reflected in the emotion estimation, and the accuracy of the emotion estimation is improved. In addition, the classification into each category and the emotion estimation based thereon can accurately reflect the human hearing characteristics (hierarchical sensory information processing in the human brain).

A preferred embodiment based on the above-mentioned solution method is as described in claim 2 and the following claims. That is, when the classification means classifies the sound data into the category of a plosive sound or a transport equipment sound, the correction means corrects the emotion estimation formula in a direction in which the emotional state is estimated to be more unpleasant and more active (corresponding to claim 2 ). In this case, the tendency of human emotions to change in the direction of unpleasantness and activity when hearing a plosive sound or a transport equipment sound is appropriately reflected in the emotion estimation formula, thereby improving the accuracy of emotion estimation.

When the classification means classifies the sound data into the category of artificial sounds, the correction means corrects the emotion estimation formula in a direction in which the emotional state is estimated to be more unpleasant. In this case, the tendency of human emotions to change in the unpleasant direction when hearing artificial sounds, such as alarms, which often indicate abnormal conditions, is appropriately reflected in the emotion estimation formula, thereby improving the accuracy of emotion estimation.

According to the present invention, the components of the sound environment are classified into types of sound sources based on texture features that are close to the characteristics of human hearing, and the emotion estimation results (emotion estimation formula) are corrected using a predetermined correction method based on the characteristics of how the human brain senses each classification, thereby improving the accuracy of emotion estimation.

An emotion map showing the subject's emotional state. This is a diagram showing a simulation of the hierarchical auditory information processing in the human brain, showing the calculation of feature data at each layer and the autocorrelation and cross-correlation of each feature data. FIG. 13 is a diagram for explaining a method for determining an evaluation model. FIG. 4 is a block diagram showing an example of a control system for setting an evaluation model. FIG. 1 is a diagram showing an example of classification of sounds into categories. FIG. 13 is a diagram showing the result of emotion estimation without correction based on category classification. FIG. 13 is a diagram showing emotion estimation results when correction based on category classification is performed.

Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings. FIG. 1 shows an emotion map for indicating the emotional state of a person being measured in an emotion estimation device. As shown in the figure, the emotion map has two axes, pleasant/unpleasant and active/inactive, as parameters, and the emotional state of the person being measured is represented by coordinate values on the emotion map.

In Figure 1, the emotion map is divided horizontally and vertically into nine regions by dashed lines (the boundaries of each region are indicated by dashed lines). To explain in more detail, the activation/inactivation parameters have A2 as the intermediate value, A1 is set in the inactivation direction, and A3 is set in the activation direction. In other words, values below A1 are inactive, values above A3 are active, and values between A1 and A3 are intermediate states.

The comfort/discomfort parameter is set to B1 on the uncomfortable side, with B2 as the intermediate value, and B3 on the comfortable side. In other words, anything below B1 is an uncomfortable state, anything above B3 is a comfortable state, and anything between B1 and B3 is an intermediate state.

When the emotion estimation device of this embodiment is used for vehicle driving assistance, for example, of the nine divided regions, the region where the pleasant/unpleasant level is greater than B1 and the active/inactive level is between A1 and A3 (region 2-2 and region 3-2) is determined to be the desired emotional state (target emotional state), and driving assistance is provided to bring the driver's emotional state closer to this target emotional state.

Next, with reference to Figure 2, we will explain the calculation of feature data at each layer based on a process that simulates the auditory information processing in the human brain, and the moment and cross-correlation for each feature data. That is, in auditory information processing, sound data starts with being input to the cochlea (basilar membrane), and is finally processed in the brainstem thalamus, and then the first layer processing, second layer processing, and third layer processing are performed in sequence, and Figure 2 shows an imitation of such processing. Note that in the following explanation, numbers in Figure 2 that are circled will be represented as 01, 02, etc. (for example, 01 means the number 1 circled).

In Figure 2, sound data is input to input unit 1, which is made up of a microphone. The input sound data is processed in Fourier transform unit 2 and divided by frequency (first-level processing). For simplicity, Figure 2 shows a case where the data is divided into two groups, high and low frequencies, but in reality, the data is divided into about 20 to 30 groups. This processing at the first level corresponds to processing at the cochlear basilar membrane.

In the second layer of processing, the feature data (divided frequencies) obtained in the first layer is subjected to a nonlinearly compressed envelope extraction process (extraction of feature data shown in O1 and O2), followed by a nonlinear compression process (corresponding to a process in which loud sounds are weakened and soft sounds are strengthened) to obtain the feature data shown in O3 and O4. This second layer of processing corresponds to processing based on the amplitude characteristics of the cochlear basilar membrane and the brainstem thalamus.

In the processing of the third layer, the Fourier transform units 3 and 4 further classify the feature data O3 and O4 in the second layer by frequency (corresponding to the modulation frequency of the feature data in the second layer). The feature data calculated in the third layer are shown as O5 to O8. Processing in the third layer corresponds to processing in the brainstem thalamus.

Moment and cross-correlation are calculated for each of the above-mentioned feature data 01 to 08. Moment is defined as one of the basic statistical values: mean, variance, kurtosis, or skewness, and when the squared value of Moment is used depending on the hierarchy, the term Power is used separately. In Figure 2, M2 and M3 indicate Moment and Power, while C0, C1, and C2 indicate cross-correlation. More specifically, M2 is Moment for 03 and 04, which are feature data in the second hierarchy, and M3 is Power for 05 to 08, which are feature data in the third hierarchy.

Furthermore, C0 is the cross-correlation between O3 and O4, which are feature data in the second layer. C1 shows the cross-correlation between O5 and O7 and the cross-correlation between O6 and O8, which are cross-correlation between different frequencies separated by the processing in the first layer. C2 shows the cross-correlation between O5 and O6 and the cross-correlation between O7 and O8, which are cross-correlation between the same frequency when separated by the processing in the first layer.

Next, an example of setting an evaluation model will be described with reference to FIG. 3. In the following description, the evaluation sensitivity will be appropriately set to "unpleasant" as an example. That is, an example will be described in which the level of discomfort (comfort level) that engine noise causes to the listener is evaluated in multiple stages.

First, in Figure 3, G1, G2, ... Gn indicate a large number of different sound data (e.g., 500 to 1000) that are the subject of an auditory sensitivity evaluation. The auditory sensitivity evaluation values (e.g., evaluation values at multiple stages for discomfort level) for these large number of sound data are linked. The discomfort level evaluation value is the average value obtained by averaging the evaluation values by multiple experts who perform the evaluation. In Figure 3, the thick vertical lines are lines that separate the individual moments and cross-correlations shown in Figure 2. And the thin vertical lines within the range surrounded by the thick vertical lines separate the individual data (correlation coefficients) that indicate the moments and correlation levels.

In Figure 3, "2nd" means feature data in the second layer, and "3rd" means feature data in the third layer. "Across" means cross-correlation between different frequencies among the frequencies divided in the first layer, and "Within" means cross-correlation between the same frequencies among the frequencies divided in the first layer.

In Figure 3, six types of correlation are set. From left to right, they are "2ndMoment", "2ndAcross", "3rdPower", "3rdAcross", "3rdWithin(Real)", and "3rdWithin(Image)". Note that "3rdWithin" is transformed using the Hilbert transform, with its real part being "3rdWithin(Real)" and its imaginary part being "3rdWithin(Image)".

For each sound data, the moment and cross-correlation are calculated. For one sound data, there are many values for the moment and cross-correlation. For each moment and each cross-correlation, a principal component analysis (e.g., five dimensions) is performed, and data extraction (narrowing of data) is performed as shown in column A in FIG. 3. In addition, as shown in column B in FIG. 3, from column A, those that have a certain level of correlation with the subjective evaluation value (for example, parameters that have a large impact on the evaluation of the discomfort level) are selected.

Based on the parameters shown in column A obtained by principal component analysis, the evaluation model (1) is determined as shown in equation 1.

（１）式中、Ｙ、各音データについての評価値であり、ａ０～ａｍ、ｂ１～ｂｍ、ｃ（ｃについてのサフィックスはここでは省略）は係数（定数）であり、Ｘ１～Ｘｍは、図３で示すＡ列でのＭｏｍｅｎｔ（あるいはＰｏｗｅｒ）および相互相関値である。当初は、上記各係数は不知である。（１）式に対して、各音データＧ１～Ｇｎについての主観的な評価値ＹとＭｏｍｅｎｔや相互相関値Ｘ１～Ｘｍをあてはめて、回帰手法によって各係数が決定される。回帰モデルとなる（１）式において、個々の係数は、多数のＭｏｍｅｎｔや相互相関値のうち特定の１つが対応づけられている。

In formula (1), Y is an evaluation value for each sound data, a0 to am, b1 to bm, and c (the suffix for c is omitted here) are coefficients (constants), and X1 to Xm are Moment (or Power) and cross-correlation values in column A shown in FIG. 3. Initially, the above coefficients are unknown. Each coefficient is determined by applying the subjective evaluation value Y and Moment and cross-correlation values X1 to Xm for each sound data G1 to Gn to formula (1) using a regression method. In formula (1) which is a regression model, each coefficient is associated with a specific one of many Moments and cross-correlation values.

Formula (1) is determined so as to include the extracted moments and cross-correlation values shown in column A. From this formula (1), for example, only parameters that have a large effect on the evaluation of the discomfort level, as shown in column B of Figure 3, are left (parameters that have no effect on the evaluation are deleted), and the evaluation model formula for evaluating the discomfort level is determined as shown in formula (2) below. That is, formula (2) has a suffix n that is smaller than the suffix m in formula (1).

ここで、上記（１）式では、全ての１次項に対して、２次の項および交互作用の項（初期値ａ０を除く右辺第３項）を含んでいる。実際には、評価モデルを極力シンプルにするという観点から、主観評価に所定以上の影響を有するある特定の１次の項に対してのみ、２次の項および交互作用の項を有するものとするのが好ましく、このような処理が行われた後の状態が、（２）式である。なお、（２）式は、（１）式から不要なパラメータについての係数分を削除した後に、各係数のサフィックスが連続するように書き直したものとなっている（（１）式と（２）式とでは、同じサフィックスの係数が同じ内容を示すものとは限らない）。

Here, in the above formula (1), a second-order term and an interaction term (the third term on the right side excluding the initial value a0) are included for all first-order terms. In reality, from the viewpoint of simplifying the evaluation model as much as possible, it is preferable to have a second-order term and an interaction term only for a certain first-order term that has a predetermined or greater influence on the subjective evaluation, and the state after such processing is the formula (2). Note that the formula (2) is rewritten so that the suffixes of each coefficient are consecutive after the coefficients for unnecessary parameters are deleted from the formula (1) (coefficients with the same suffix do not necessarily indicate the same content in the formulas (1) and (2)).

When evaluating the discomfort level of certain sound data, the moment (power) and cross-correlation value are calculated for the certain sound data as shown in Figure 3, and the calculation result is applied to the above formula (2), whereby an evaluation value for the certain sound data is calculated numerically (and can be visualized through quantification). Note that the validity of the evaluation model in formula (2) can also be verified using multiple sound data other than the sound data G1 to Gn shown in Figure 3 (moments and cross-correlation values calculated from them) and subjective evaluation values for them.

In the above explanation, the focus has been on auditory sensitivity, but an evaluation model can also be set for visual sensitivity in a similar manner. That is, when simulating human visual information processing, the first layer is divided by frequency (i.e., resolution), the second layer is azimuth selection processing, and the third layer is azimuth signal strength processing. Using the autocorrelation (correlation value indicating, values based on moments can also be used) and cross-correlation (correlation value indicating) for each piece of feature data from the first layer to the third layer, an evaluation model for visual sensitivity is set using a method similar to that described in Figure 3.

Next, with reference to FIG. 4, an example of a control system of an emotion estimation device in an embodiment of the present invention (an example of an overall control system for setting an evaluation model for performing a sensibility evaluation) will be described. In FIG. 4, 11 is an input unit for sound data to be evaluated for auditory sensibility, and is constituted by, for example, a microphone. The sound data input to the input unit 11 is processed by the method described with reference to FIG. 2 and FIG. 3, and autocorrelation and cross-correlation are calculated as feature statistics of the sound data. That is, feature data is calculated for the input sound data by the sound feature calculation unit 12, and autocorrelation and cross-correlation are calculated by the statistics calculation unit 13 based on this feature data.

Meanwhile, personal information (e.g., gender, age, occupation, etc.) is input in the personal information input unit 21, and the input personal information is preprocessed (e.g., age is classified into multiple stages) in the personal information preprocessing unit 22, and then recorded (stored) in the personal information recording unit 23.

The bioinformation inputted by the biosensor input unit 24 and the subjective sensory evaluation value inputted by the subjective evaluation unit 25 are inputted to the sensory actual measurement value estimation unit 26 to estimate the sensory actual measurement value, and this estimated sensory actual measurement value is recorded (stored) in the sensory actual measurement value recording unit 27.

The evaluation target is identified in the evaluation target identification unit 29 based on the evaluation target data (e.g., sound data) input in the evaluation target information input unit 28. The evaluation target identified in the evaluation target identification unit is then recorded (stored) in the evaluation target recording unit 30. In addition, the calculation results in the hearing statistics calculation unit 13 are recorded (stored) in the hearing statistics recording unit 31.

The above-mentioned recording units 23, 27, 30, and 31 constitute the database unit D. An evaluation model is set based on the data in this database unit D. Specifically, the data in the database unit D is input to the evaluation parameter selection unit P, appropriate parameters (evaluation axes) are selected according to the type of sensitivity to be evaluated (corresponding to the parameter selection in column B of FIG. 3), and an evaluation model is set according to the selected parameters (the evaluation model shown in formula (2) is set).

The contents recorded in the evaluation object recording unit 30 are used as a large number of sound data G1 to Gn as shown in FIG. 3, and the contents recorded in the sensitivity measurement value recording unit 27 are used as subjective evaluation values for each sound data G1 to Gn. The personal information input unit 21 is used to correct the evaluation value according to personal information (e.g., gender and age group) and is not necessary. The statistics recording unit 31 is used to learn and correct the evaluation model when a new sensitivity evaluation is performed.

The evaluation parameter selection unit P selects evaluation parameters (evaluation axes) according to the content (type) of the sensitivity to be evaluated. In FIG. 4, evaluation parameters are exemplified by evaluation parameter A related to visual stimuli and evaluation parameter B related to auditory stimuli (sound data). In the evaluation model construction unit C, an evaluation model is generated based on the evaluation parameters selected in the evaluation parameter selection unit P, and the emotional state of the subject is estimated based on this evaluation model.

The evaluation parameter selection unit P is provided with a category classification filter 32. When the sensory data to be evaluated is sound data (auditory stimulation), the category classification filter 32 is a filter that classifies the acquired sound data into categories that are set in advance for each type of sound source based on features (texture features) that characterize the sound source. That is, the category classification filter 32 performs texture analysis on the sound data to select a category that is likely to include the sound data, and classifies the sound data into the selected category. The category classification filter 32 is constructed, for example, by machine learning.

When the sound data is classified by the category classification filter 32, the parameter set for forming the evaluation model is corrected for each category (correction suited to the category to which the sound data belongs) according to which category the sound data is classified. For example, in FIG. 4, evaluation parameter A related to visual stimuli is not corrected because it does not relate to sound data, but parameter set B related to auditory stimuli is corrected according to the category to which the sound data belongs.

Specifically, the coefficients in the emotion estimation formula (2) above are corrected to reflect the characteristics of each classified category (how the human brain feels each category of sound) (the coefficients are changed to preset coefficients for each category).

In this way, in the emotion estimation device of the present embodiment, sound data is classified into categories, and the emotion estimation formula is corrected for each category, thereby improving the accuracy of emotion estimation. To explain in detail, the effect of auditory stimuli on human emotions (sensitivity) varies greatly depending on the category of sound data (for example, the type of sound source such as natural sound, electronic sound, etc.) (the contribution of sound features to emotions varies), but this difference is not sufficiently reflected in emotion estimation by simply analyzing the sound data as a whole. For this reason, in the present embodiment, the parameter set related to the auditory stimuli is corrected according to the category to which the sound data belongs, so that the difference due to different categories can be appropriately reflected in emotion estimation (sensitivity evaluation), enabling more accurate and detailed emotion estimation.

The classification of sound data in the category classification filter 32 is performed based on the texture features of the sound data. In this embodiment, the autocorrelation and cross-correlation calculated by the feature calculation unit 12 and the metric calculation unit 13 are used as the texture features of the sound data.

Specifically, autocorrelation and cross-correlation are calculated using the procedure described in conjunction with Figures 2 and 3. That is, the acquired sound data is processed in the order of frequency decomposition → envelope extraction → modulation frequency decomposition, the mean, variance, skewness, and kurtosis are calculated for the amplitude of the frequency components, and autocorrelation and cross-correlation are calculated (to capture the modulation on the time axis), and autocorrelation and cross-correlation are calculated for the modulation frequency components (frequency components of the envelope of the sound wave) (to capture the time/frequency modulation).

In this way, texture features are calculated based on the frequency components and modulation frequency components of the sound, so classification into categories and emotion estimation based on this can accurately reflect the characteristics of human hearing (hierarchical sensory information processing in the human brain).

Figure 5 shows an example of categorizing sound data. In this example, the sound sources of the sound data are classified into 10 categories: "animals," "plosives," "everyday sounds," "electronic sounds," "music," "natural sounds," "conversations," "background sounds," "(electrical) sound effects," and "transportation equipment."

As shown in FIG. 5, the "animal" category includes animal sounds such as dogs, cats, cows, horses, and birds. The "plosive sounds" category includes gunshots, explosions, and collisions. The "everyday sounds" category includes bells, alarms, door opening and closing, and telephone sounds. The "electronic sounds" category includes video game sounds, casino sounds, and smartphone sounds. The "music" category includes music of various genres such as country, rock, and orchestra. The "natural sounds" category includes the sounds of the sea, valleys, fire, thunder, rain, and wind. The "conversation" category ("human" category) includes human voices such as conversation, crying, laughing, and screaming. The "background sounds" category includes sounds of towns, offices, and the like. The "sound effects" category includes sounds used in scenes of horror, suspense, and other video works. The "transportation equipment" category includes sounds of cars, trains, airplanes, motorcycles, and the like.

The emotion estimation formula is corrected for each category according to the characteristics of the emotion value of the sound in each category (the tendency of the actual emotion score compared to the estimation by the emotion estimation formula without correction). For example, the emotion value characteristics of "plosive sounds" and "transportation equipment" are unpleasant and active, so when sound data is classified as "plosive sounds" or "transportation equipment", the emotion estimation formula is corrected in the direction of estimating more unpleasant for pleasant/unpleasant, and in the direction of estimating more active for active/inactive. Similarly, in the case of "electronic sounds" and "background sounds", correction is made in the direction of estimating more active, in the case of "music", correction is made in the direction of estimating more pleasant, and in the case of "conversation", correction is made in the direction of estimating more unpleasant.

Another way of classifying sound data is to classify it into categories of artificial sounds (e.g., electronic sounds, warning sounds, etc.) and non-artificial sounds (natural sounds, human voices, etc.). In this case, artificial sounds are often alarms and other types of sounds that indicate abnormal conditions, and therefore tend to affect human emotions in a more unpleasant direction. Therefore, for sounds classified as artificial sounds, the emotion estimation formula is corrected to estimate them as more unpleasant.

In the above explanation, we have described the case where sound data is classified into one category (where there is a single sound source), but the present invention is also applicable to cases where the sound data has multiple sound sources. In this case, for example, the coefficients in the above formula (2) can be set to reflect the influence of each category to which the sound source belongs, taking into account the proportion of each sound source contained in the sound data.

Figures 6 and 7 show the improvement in the accuracy of emotion estimation by using the emotion estimation device of the present invention (the corrected emotion estimation formula) in the form of emotion maps for each sound category. As shown in Figure 6, in emotion estimation using the emotion estimation formula without the correction of the present invention, there are categories where the emotion estimation values shown as black diamonds in the figure do not overlap with the emotion scores shown as white circles in the figure (emotional states directly detected by a biosensor, etc.), resulting in a significantly off prediction.

In contrast, as shown in Figure 7, emotion estimation using the emotion estimation formula corrected by the present invention improves the degree of agreement (overlap) between the emotion estimation value and the emotion score, and prediction accuracy is significantly improved. For example, in the case of the "destruction sound" category, emotion estimation without category correction estimates the degree of pleasantness and inactivity more than the actual state, with pleasantness/unpleasantness shown on the vertical axis and activity/inactivity shown on the horizontal axis, whereas emotion estimation according to the present invention accurately estimates the degree of unpleasantness and activity, improving the degree of agreement between the emotion score and the emotion estimation value.

Next, we will explain driving assistance control for a vehicle using the emotion estimation device of the present invention. In driving assistance, when estimating the emotion of an occupant using a common emotion estimation formula that is not dependent on the sound category, as in the conventional technology, and controlling the emotional state of the occupant based on this estimation result (for example, by emitting a control sound to turn an unpleasant state into a pleasant state), erroneous estimation is likely to occur because the contribution of sound features to emotions differs depending on the sound category (type of sound source).

For example, if music, which is an auditory stimulus, is judged to be unpleasant, a control sound may be output to cancel out the music even when the occupant is in a comfortable state, changing the sound environment and making the occupant uncomfortable. Also, if driving noise is judged to be pleasant, appropriate control may not be implemented even when the occupant is in an uncomfortable state.

In contrast, driving assistance control using the emotion estimation device of the present invention classifies sounds into categories and estimates emotions using emotion estimation formulas for each category, making it possible to accurately estimate the emotions of occupants and perform appropriate control (e.g., outputting control sounds from speakers, controlling vehicle behavior (co-pilot), etc.).

The following types of driving assistance control are possible for each of the 10 categories. According to the present invention, these controls can perform accurate emotion estimation that reflects the characteristics of each category, making it possible to implement more effective driving assistance control (correction to the emotion estimation formula works particularly effectively).

When the sound category is "animal," control can be performed such that, while driving in a location where animals are likely to appear (for example, detected from map information), if a sound classified as "animal" is detected only by the inner microphone among the microphones inside and outside the vehicle (for example, when an animal's cry is heard over the radio), a control sound with an emphasis on inactivity is emitted to calm the occupants and announce that there is a low possibility of an animal jumping out.

In addition, if an animal is riding in the vehicle and the animal's cries inside the vehicle (e.g., in the back seat) are loud and unpleasant, a sound that emphasizes the loudness and unpleasantness can be output to the occupants to make them aware of something abnormal that has happened to the animal (to inform the occupants of the animal's deteriorating health).

In addition, if there is a possibility that an animal may be lurking around the vehicle, and the sound from the microphone outside the cabin is in the "animal" category (i.e., an animal outside the vehicle is detected), vehicle behavior control can be performed to be on alert for an animal jumping out and reduce the vehicle speed.

When the sound category is "destructive sound," for example, it may be the case that the vehicle breaks down or an accident occurs near the vehicle, and since it is assumed that the occupants are in a confused state, control may be performed to output a control sound with inactive emphasis to calm the occupants. Also, control may be performed to output a control sound that alerts the occupants to check the situation around them. Furthermore, when the sound is due to a breakdown or accident of the vehicle, vehicle behavior control may be performed to make an emergency stop on the shoulder of the road.

When the sound category is "everyday sounds" (bells, alarms, door opening and closing sounds, telephone ringtones, etc.), control can be performed by outputting a control sound that emphasizes the emotional state as active or unpleasant when a warning sound is sounding, and by making the driver follow the warning sound (by making the driver confirm or stop the vehicle). Also, when a siren is sounding around the vehicle, vehicle behavior control can be performed to pull the vehicle over to the shoulder of the road.

When the sound category is "electronic sound," when the sound of a smartphone is detected in the driver's seat (when the driver is thought to be using the smartphone while driving), a control sound that emphasizes the unpleasantness can be output, causing the driver to stop using the smartphone while driving (or to issue a warning) (preventing dangerous driving). Also, when an electronic sound is detected in the back seat (for example, when a passenger is playing a game in the back seat), a masking sound can be output to the driver to prevent the game sound from reaching them, allowing them to concentrate on driving.

When the sound category is "music" (for example, when music is detected from the car audio), the control can be such that the user registers their preferred genre (country, rock, orchestra, etc.) in advance, and sounds with characteristics suited to each genre are output and arranged to fit the preferred music genre.

When the sound category is "natural sounds," control can be performed such that when there is a weather change such as thunder, a control sound that emphasizes deactivation is output to calm occupants who become startled and active by the thunder. Also, when a microphone outside the vehicle detects that the vehicle is approaching the sea or mountains, a control sound that emphasizes the characteristics of natural sounds (e.g., the sound of waves, the sound of swaying trees, birds chirping, etc.) can be output, and by emphasizing and playing the natural sounds, control can be performed to make the occupants feel closer to nature.

When the sound category is "conversation," when a conversation between occupants or a voice from the radio is detected, sounds other than human voices can be inactively emphasized (and their levels reduced) to make the human voice (conversation) easier to hear. Also, if one of the occupants is in a desperate voice due to poor health, a control sound that emphasizes activity and discomfort can be output to the other occupants, making them aware of the poor health of one of the occupants.

When the sound category is "background sound," control can be performed by outputting a control sound that emphasizes activity when a crowd is detected by a microphone outside the vehicle cabin, to draw attention to people's movements. In addition, when a crowd is detected, vehicle behavior control can be performed to reduce the vehicle's speed.

When the sound category is "sound effects," control can be performed such that, when passengers other than the driver are watching video (for example, horror, suspense, etc.), a pleasantly emphasized control sound is output to the driver, making them less susceptible to the video. Also, when sound from an audio/radio is detected, control can be performed to make the sound from the audio/radio easier to hear by deactivating and emphasizing sounds other than the sound from the audio/radio (and further reducing the level).

When the sound category is "transportation equipment," if the interior sound while the vehicle is moving is unpleasant, a control sound that emphasizes comfort may be output to make the emotional state of the occupants more comfortable.

In addition, when an approaching vehicle is detected by a microphone outside the vehicle cabin, a control sound with enhanced activity may be output to make the other vehicle sound closer than it actually is, to notify the driver of its approach. Vehicle behavior control may also be performed to move the vehicle to keep a safe distance from other vehicles.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and appropriate modifications are possible within the scope of the claims. For example, the sound categories disclosed in the above embodiments are merely examples, and categories different from those in the above embodiments can also be set.

The present invention can be used to accurately estimate the emotional state of occupants (drivers and passengers) in vehicle driving assistance.

D Database section P Evaluation parameter selection section C Evaluation model construction section 11 Evaluation target acquisition section 12 Feature amount calculation section 13 Statistics calculation section 21 Personal information input section 22 Personal information preprocessing section 23 Personal information recording section 24 Biosensor input section 25 Subjective evaluation input section 26 Perceptual measurement value estimation section 27 Perceptual measurement value recording section 28 Evaluation target information input section 29 Evaluation target identification section 30 Evaluation target recording section 31 Statistics recording section 32 Category classification filter

Claims (3)

An emotion estimation device that estimates an emotional state of a subject placed in a sound environment, comprising:
a sound data acquisition means for acquiring sound data in the sound environment;
an estimation means for estimating an emotional state of the subject when the subject receives an auditory stimulus specified by the sound data based on a predetermined emotion estimation formula;
A classification means for classifying the sound data into categories that are preset for each type of sound source based on a texture feature of the sound data;
a correction means for correcting the emotion estimation formula according to the category into which the sound data has been classified by the classification means ,
The texture feature is calculated based on frequency components and modulation frequency components of the sound data . The emotion estimation device according to claim 1 ,
When the classification means classifies the sound data into a category of a plosive sound or a transportation machine, the correction means corrects the emotion estimation formula in a direction in which the emotional state is estimated to be more unpleasant and more active. The emotion estimation device according to claim 1 ,
When the classification means classifies the sound data into the artificial sound category, the correction means corrects the emotion estimation formula in a direction such that the emotional state is estimated to be more unpleasant.

Images