JP7664960B2 - Performance control system, performance control method, and program - Google Patents

Description

The present invention relates to a performance control system, a performance control method, and a program.

When a speaker (information provider) is concentrating on his/her own remarks in a meeting or presentation, it can be difficult to identify what the audience is feeling. This situation makes it difficult for the speaker to accurately grasp the situation of the audience and adjust the progress. This tendency becomes more pronounced the larger the audience.
In recent years, remote conferences or presentations connected by communication have been frequently carried out. In these conferences or presentations, it is more difficult for a speaker to accurately grasp what an audience is feeling than in a face-to-face conference or presentation, because the audience cannot meet face to face.
As described above, it has been desirable for a speaker to be able to accurately grasp what the audience is feeling, regardless of whether the meeting is face-to-face or over the Internet.
Meanwhile, there is known a technique for calculating a sensory evaluation of an individual from an image of the individual's face or the like (see, for example, Japanese Patent Application Laid-Open No. 2003-233694).

JP 2022-037575 A

However, even if the technology in Patent Document 1 and other technologies could be used to perform a personal emotional evaluation based on an individual's facial expression, it was not easy for the information provider to understand the situation of the audience.

The problem that this invention aims to solve is to provide a production control system, a production control method, and a program that assist information providers in understanding the emotions of their audience.

The production control system of one aspect of the present invention produces an environment in which an information provider provides information to an audience. The production control system includes a control unit that controls the production of the environment in which the information provider provides information to an audience based on the results of emotion analysis that estimates the emotion of the information provider or the audience.

The performance control system includes an output unit that outputs the results of performance control, either sound control, fragrance control, or lighting control, and the control unit causes the output unit to output the results of the performance control based on the results of the emotion analysis.

In the above-mentioned production control system, the control unit changes the environment of the venue through the acoustic control to the extent that it does not affect the transmission of the voice of the speaker who is the information provider.

In the above-mentioned performance control system, the control unit changes the environment of the location through the scent control so that the change in scent reaches the information provider.

In the above-mentioned performance control system, the control unit changes the environment of the venue through the lighting control so that the information provider can identify changes in the lighting of the venue.

In the above-mentioned performance control system, the control unit extends the usage time of the space when the result of the emotion analysis indicates a relatively positive tendency, and shortens the usage time of the space when the result of the emotion analysis indicates a relatively negative tendency.

In the above-mentioned performance control system, when the result of the emotion analysis indicates a relatively negative tendency, the control unit controls the environment of the venue to promote relaxation in the venue.

In the above-mentioned performance control system, when the control unit detects a high tendency toward "anger" or "fear" from the results of the emotion analysis, it controls the environment of the venue to create an environment that encourages relaxation.

In the above-mentioned performance control system, when the control unit detects a relatively high tendency for the subject to be in a "low state of arousal" based on the results of the emotion analysis, the control unit controls the environment of the venue to promote arousal.

In the above-mentioned production control system, the output unit is provided in a room, which is the location where the information provider is present.

The production control method according to one aspect of the present invention is a production control device that produces the environment of a place where an information provider provides information to an audience. The production control method includes a process in which the production control device controls the production of the environment of the place based on the results of an emotion analysis that estimates the emotion of the information provider or the audience.

A program according to one aspect of the present invention is a program for causing a computer of a production control device, which produces the environment of a place where an information provider provides information to an audience, to execute a step of controlling the production of the environment of the place based on the results of an emotion analysis that estimates the emotions of the information provider or the audience.

The present invention provides a production control system, a production control method, and a program that assist information providers in understanding the emotions of their audience.

FIG. 1 is a configuration diagram of a performance control system according to an embodiment. FIG. 2 is a diagram for explaining an example of a usage form of the performance control system according to the embodiment. FIG. 2 is a configuration diagram of an estimation unit according to the embodiment. 10A to 10C are diagrams for explaining various types of information related to emotion analysis processing according to an embodiment. 11 is a flowchart of an emotion analysis process according to the embodiment. 13 is a flowchart of an emotion analysis process according to the second embodiment. FIG. 2 is a diagram for explaining processing related to performance control in an embodiment. 11 is a flowchart of an automatic extension process of a usage time according to an embodiment.

The embodiment of the performance control system, performance control method, and program will be described with reference to the figures.

First, we will explain comparative examples for the performance control system of the embodiment. The systems exemplified as comparative examples are as follows.

(Sound production within the facility)
Conference rooms and presentation environments (such as halls) are often silent, with no special sounds being played for sound effects. Other examples of sound effects include playing sounds that make the environmental noise of the conference room less noticeable. Other examples of sound effects include playing music to entertain participants.

(Scenting in the facility)
Aromas and the like may be released to create a fragrant atmosphere in a conference room or presentation environment. For example, the aroma may be released continuously or at specific times by schedule control (timer control). The times when the aroma is released may be selected to be times when it is expected to be effective, such as working hours.

(Lighting effects within the facility)
Facilities such as conference rooms, presentation venues, and the like are used in a variety of ways, and may be configured to enable lighting effects to be created in a variety of ways.
For example, in a conference room, lighting may be turned on/off and its illuminance and color temperature may be controlled by hand operation. For example, the on/off control of the lighting is performed by manually operating a switch or the like. In addition to the on/off control of the lighting, the illuminance of the room, the output of the luminaire, and the color temperature are adjusted by manually operating an operating device.
In addition, lighting control aimed at saving energy when using a conference room may be linked to a conference room reservation system, etc., so that the lights are turned on only when the conference is to begin and during the time periods before and after the conference, or turned off when the conference is to end.

As a lighting design for a presentation venue, lighting may be turned on/off and illuminance and color temperature may be controlled based on hand movements, similar to the aforementioned conference room.
In addition, when controlling lighting for the purpose of saving energy when using a venue for a presentation, it may be linked to a conference room reservation system, etc., so that the lights are turned on only when the presentation is to be held and during the period before and after the presentation, or turned off when the presentation is to end.

As mentioned above, none of the comparison examples included a conference room/presentation environment with mechanisms based on emotion recognition results.
In contrast, the emotion analysis system of the embodiment estimates the emotion of an information provider or an audience member, and outputs the result of emotion analysis based on this.
Emotions that the emotion analysis system analyzes and identifies may include, for example, emotions such as "fun,""joy,""cheerful,""refreshing,""pleasure,""sad,""pain,""anger,""surprise,""discomfort," and "dislike."
These will be explained in order below.

(First embodiment)
Fig. 1 is a configuration diagram of a performance control system 1 according to an embodiment. Fig. 2 is a diagram for explaining an example of a usage form of the performance control system 1 according to the embodiment.

The production control system 1 uses the production control device 3 of the facility to produce the environment in which an information provider provides information to an audience.
The performance control system 1 includes, for example, a performance control device 3.
The performance control device 3 is an example of a device including a computer system.
The performance control device 3 includes, for example, a control unit 31, a storage unit 32, a remote conference monitor unit 33, and an output unit 34.

The control unit 31 includes a processor such as a CPU. A part or the whole of the performance control device 3 may be a hardware function unit or a software function unit.
The storage unit 32 includes, for example, a semiconductor memory, a magnetic storage device, etc. for storing data. The storage unit 32 stores various data as well as programs required for the control of the control unit 31.
The remote conference monitor unit 33 becomes a client of the remote conference system 7 and communicates with the remote conference system 7 to acquire and monitor video and audio of the remote conference provided by the remote conference system 7. The remote conference monitor unit 33 makes the acquired video and audio available for processing by the control unit 31.
The output unit 34 outputs the result of the performance control, which is any one of the sound control, the scent control, and the lighting control, by the control unit 31 under the control of the control unit 31, and makes the result detectable by the user. For example, the output unit 34 is provided in a room where the information provider is present. The output unit 34 will be described in detail later.
For example, as shown in FIG. 2, the performance control device 3 may be placed in a conference room, an auditorium where a presentation is given, or the like, in a position where it can be seen by the speaker.

FIG. 1 shows the following parts:
The lighting control equipment 2 is an example of a lighting control equipment that controls the lighting status of the above-mentioned place.
The terminal device 4 is, for example, a portable wireless communication device used as an operation unit of the performance control system 1. The terminal device 4 may be composed of one or more terminal devices 4. The type of the multiple terminal devices 4 may be the same or different. The terminal device 4 may be formed by executing processing by application software.
The camera 5 includes an imaging device, an optical system, and the like, and transmits image data via the network NW.
The microphone 6 is an example of a sound collecting device.

The performance control system 1 may be configured to include, for example, a lighting control facility 2, a terminal device 4, a camera 5, and a microphone 6. The lighting control facility 2, the terminal device 4, the camera 5, and the microphone 6 may each be provided in a conference room or the like.
The performance control device 3, the lighting control equipment 2, the terminal device 4, and the camera 5 are connected to each other via a network NW so that they can communicate with each other.
Furthermore, the performance control device 3 may be connected to, for example, a remote conference system 7 and a remote terminal device 8 so that they can communicate with each other via a network NW.

The details of each part of the performance control system 1 are explained below.

(Lighting control equipment)
The lighting control device 2 includes, for example, a first lighting control device 21 and a second lighting control device 22 having different uses.
The first lighting control device 21 of the lighting control device 2 includes a luminaire 211 and its controller 212 .

The luminaire 211 of the first lighting control device 21 is placed in a specific area such as a conference room. The type of luminaire 211 may be one of the common types such as ceiling type, downlight type, pendant type, spotlight type, bracket type, etc., or may be a combination of multiple types.

For example, the controller 212 of the first lighting control device 21 and the luminaire 211 are connected so as to be able to communicate with each other. The controller 212 of the first lighting control device 21 controls the lighting state of the associated luminaire 211.

The controller 212 of the first lighting control equipment 21 can adjust the lighting state to dim the light, in addition to controlling the on/off of the above-mentioned luminaires 211. "Dimming" here may include adjusting the amount of light emitted by the luminaires 211, i.e., adjusting the output of the luminaires 211, adjusting the color temperature of the light emitted by the luminaires 211, or a combination of the above.

The second lighting control device 22 includes a luminaire 221 and its controller 222. The luminaire 221 of the second lighting control device 22 is placed in the specific area or an area other than the specific area. The type of luminaire 221 may be one of the general types such as ceiling type, downlight type, pendant type, spotlight type, bracket type, etc., or may be a combination of multiple types.

The controller 222 of the second lighting control device 22 controls the on/off of the above-mentioned luminaires 221, but does not have to be configured to adjust the lighting state to adjust the dimming.

(Terminal device)
The terminal device 4 is an example of a device including a so-called computer system, such as a smartphone. For example, the terminal device 4 enables the setting of each part of the performance control system 1 and the control of the first lighting control facility 21 by executing an application program activated on an OS.

(Remote conference system)
The remote conference system 7 is a server device that provides a remote conference service and the like to a user who uses the remote terminal device 8 by communicating with the remote terminal device 8. This remote conference service can be used from the remote terminal device 8. The remote conference system 7 selects and relays images, audio, and the like sent from a specific remote terminal device 8 to other remote terminal devices 8.

(Remote Terminal Device)
The remote terminal device 8 is an example of a device including a so-called computer system, such as a smartphone, a personal computer, etc. For example, the remote terminal device 8 is configured to be able to use a remote conference service by communicating with the remote conference system 7 through execution of an application program activated on the OS. The remote terminal device 8 may be equipped with a microphone, a camera, etc. for collecting sound, and may be configured to be able to acquire information from an external microphone or camera.
The remote terminal device 8 is positioned as a client of the remote conference system 7 .
As will be described later, the performance control device 3 is also positioned as a client of the remote conference system 7.

(Performance control device)
As described above, the performance control device 3 of the embodiment includes the control unit 31, the storage unit 32, the remote conference monitor unit 33, and the output unit 34.

The control unit 31 includes, for example, a facility use reservation information management unit 311, a performance control unit 312, and an estimation unit 313. The facility use reservation information management unit 311, the performance control unit 312, and the estimation unit 313 are examples of functional units formed by execution of a program.
The control unit 31 has a clock unit (not shown) and can control the lighting control apparatus 2 using time information from the clock unit.

Below, we will first explain the facility usage reservation information management unit 311 and the performance control unit 312.

The facility reservation information management unit 311 manages the reservation information for each facility in the building. The facility reservation information management unit 311 acquires the reservation information for each facility in the building from the terminal device 4, adds it to the storage unit 32, manages it as reservation information for each facility, and uses it.

For example, when the facility reservation information management unit 311 newly acquires reservation information for each facility in the building, it adds the reservation information for each facility in the building acquired from the terminal device 4 to the storage unit 32 .
In addition, when the facility usage reservation information management unit 311 detects a change or a request for a change to the usage reservation information for each facility in the building stored in the memory unit 32 from the acquired information, it updates the usage reservation information for each facility in the building stored in the memory unit 32 to the changed information.

As described above, the facility usage reservation information management unit 311 makes the usage reservation information for each facility in the building, which has been newly added to the storage unit 32, available to the performance control unit 312.
For example, the facility reservation information management unit 311 may notify the performance control unit 312 that there has been a change in the reservation information for each facility in the building. For example, when making this notification, the facility reservation information management unit 311 may notify the performance control unit 312 of reservation information that has been newly added to the storage unit 32. This notification may be sent in the form of a command or mail, or may be sent via a shared memory in the storage unit 32.

The performance control unit 312 cooperates with the facility usage reservation information management unit 311 and the estimation unit 313 to perform dimming control of the lighting control device 2 and control of the output control unit 314.

Referring to Figure 3, we will explain the process of estimating the situation within a facility that can be applied to the first scenario.

First, the control unit 31 of the performance control system 1 uses the estimation unit 313 to detect the situation within the facility.
In the embodiment, the estimation unit 313 estimates the emotions of the audience from the venue conditions of the facility, etc., based on either the collected audio or images, using either a predetermined identification criterion or a trained model.

FIG. 3 is a configuration diagram of the estimation unit 313 according to the embodiment.
The estimation unit 313 includes, for example, a sensor information acquisition unit 3131 , a voice and sound detection unit 3132 , a natural language information understanding unit 3133 , an image and video information understanding unit 3134 , an inter-modal common understanding unit 3135 , and a state estimation unit 3136 .

The sensor information acquisition unit 3131 acquires information from each camera 5 installed within the facility. The cameras 5 may be fixedly installed in each facility.

Alternatively, the camera 5 may be provided in the terminal device 4. For example, a user may bring a terminal device 4 into the facility, and the terminal device 4 and the performance control device 3 may be placed in a state in which they can communicate with each other. The sensor information acquisition unit 3131 may treat information from the terminal device 4 as information from the camera 5.

The voice and sound detection unit 3132 extracts features indicated by the voice and sound information based on the voice and sound information within the facility detected by the microphone 6. For example, features indicated by the voice and sound information include the volume of the voice, the volume of the environmental sounds within the facility, fluctuations, identification of the speaker's voice quality (male/female, etc.), the speaker's speech frequency, etc.

The natural language information understanding unit 3133 extracts natural language information from the voice and audio information within the facility detected by the microphone 6. For example, the features indicated by the voice and audio information may include the meaning of the word or sentence, as well as the volume of the voice or sound at the time the word or sentence was identified, the volume of the environmental sound, and fluctuations.

The image video information understanding unit 3134 extracts information about the facility usage status from the image information within the facility detected by the camera 5. For example, the information about the facility usage status includes the number of users, user identification, user behavior identification, and speaker identification.

By associating information regarding the situation within the facility during use, extracted by each of the voice and sound detection unit 3132, the natural language information understanding unit 3133, and the image and video information understanding unit 3134, with the time detected by the camera 5, it is possible to combine information written using the time information as a key.
For example, the inter-modal common understanding unit 3135 uses time information as a key to combine information related to the situation within the facility being used, thereby extracting information that is significant for the intended analysis.
For example, changes in "emotions" are detected as a later stage of processing, and the inter-modal common understanding unit 3135 extracts information suitable for this purpose.

The state estimation unit 3136 detects a change in "emotion" by using the information extracted by the inter-modal common understanding unit 3135. The "emotion" to be detected may include voice quality, volume, meaning of words, speaker, intonation of speech, and the like.
The estimation unit 313 improves the accuracy of identifying such changes in "emotion" by comprehensively determining the "emotion."

For example, the estimation unit 313 estimates "emotions" using either the sound collected by the microphone 6 or the image collected by the camera 5, and a predetermined discrimination criterion. The estimation unit 313 estimates the actual usage status of the facility from the estimated "emotions."

More specifically, "emotions" can be estimated by analyzing audio information acquired through the microphone 6 and image information acquired through the camera 5. There are times when it is possible to determine from "emotions" that it is not appropriate to continue the meeting or presentation as it is. In such cases, it is better to determine that it is more appropriate to end the current state before the permitted usage time expires, rather than continuing until that time expires.

The performance control system 1 uses the estimation unit 313 to detect ongoing situations within the facility and controls the performance of the conference room environment based on the results.

First Scenario:
First, the first scenario will be described.
In this first scenario, an example of a production control system 1 applied to a system that enables an information provider of a conference or presentation to casually know the reactions of the participants (hereinafter referred to as a "system for knowing reactions"). Note that the above "information provider" includes the speaker, organizer, presenter, facilitator, etc. Many of the "participants" will be listeners. The "participants" may include the speaker (information provider). Participants in a conference, etc. will be listeners except when they are speaking.

A camera 5 installed in the conference room or a camera installed in a terminal device 4 such as a personal computer captures the facial expressions of each participant. For participants participating online, a camera installed in a remote terminal device 8 captures the facial expressions of each participant.

The control unit 31 acquires video showing the facial expressions of each individual participant and analyzes it in real time to classify the emotions of each individual participant into "boredom," "anger," "sadness," "joy," "surprise," etc. Known technology may be applied to identify such individual emotions.

The performance control system 1 of this embodiment is an example of forming a "system that knows reactions" by identifying "group emotions" and controlling the performance of the event.

The "system to know reactions" (control unit 31) identifies "group emotions" by, for example, utilizing technology to identify individual emotions. For example, the "system to know reactions" (control unit 31) counts the number of people expressing a specific emotion among the subjects of emotion analysis for each type of emotion, determines the "group emotion" as the atmosphere of the meeting or presentation, and casually conveys the "group emotion" to the organizer of the meeting or presentation by using lighting effects that involve changes in illuminance and color. In the "system to know reactions" (control unit 31), it is advisable to set in advance specific illuminance and color that characterize the environment for each type of "group emotion."

In addition, the lighting control equipment 2 in the embodiment may be configured to include not only general ceiling lighting, but also lighting installed on the walls and floor, indirect lighting, projectors, etc. that illuminate the ceiling, walls, and floor.

As a more specific example, we will explain a case where one speaker is giving a lecture to 19 listeners using both a conference room and a remote conference.
The speaker and some of the audience gather in a conference room, while the rest of the audience participate in a remote conference online using a remote conference service. During the lecture, the "reaction recognition system" (control unit 31) collects the "individual emotions" of each audience member.

<Step 1>
In the "system for knowing a reaction" (control unit 31) of the embodiment, initial values of various information related to the emotion analysis process are set. The various information related to the emotion analysis process of the embodiment will be described with reference to FIG.
4 is a diagram for explaining various information related to the emotion analysis process of the embodiment. An example of a table used by the "system for knowing reactions" of the embodiment is shown in FIG. This table includes, for example, "boredom,""anger,""sadness,""fun,""surprise," and "fear" as types of "group emotions" identified by the "system for knowing reactions."
Some or all of these emotions are classified in advance into negative and positive emotions. For example, "boredom,""anger,""sadness," and "fear" are classified as negative emotions, and "fun" and "surprise" are classified as positive emotions. For example, negative emotions are marked with a "▼" in the classification column of FIG. 4, and positive emotions are marked with a "△" in the classification column of FIG. 4. A positive emotion is an example of an emotion that shows a relatively positive tendency.

The table shown in Fig. 4 includes items of various data such as "volume judgment,""illuminance,""color,""scent,""soundeffects,""number of people," and "judgment." Data indicating whether or not judgment based on volume is required is added to the "volume judgment" item. Furthermore, data for control according to "group emotion" determined based on the aggregated results of emotion analysis of each audience member is added to this table. The control items include "illuminance,""color,""scent,""soundeffects," etc.
In addition to the above, this table also has the fields "Number of people" and "Judgment." Data on the results of sentiment analysis is added to these "Number of people" and "Judgment" fields. For example, the "Number of people" field is added with data on the total number of attendees and the aggregated results of sentiment analysis for each attendee. The "Judgment" field is added with data showing that the majority of attendees expressed the same emotion out of the total number of attendees.

For example, this is applied to a lecture with an audience of 19 people. The majority of the audience is 10 people or more. In the table shown in Fig. 4, the effect of "boredom" expressed by the "system for knowing reactions" is preset as "illuminance 600 lux, color blue". Similarly, desired data can be preset for other types of emotions and the "scents" and "sound effects" to be controlled.
The "system for knowing reactions" may use the data in this table for sentiment analysis processing and control based on the results of that processing.

In the following explanation, we will use the table in Figure 4 to explain an example in which some or all of the types of "group emotions" are classified into two categories: negative emotions and positive emotions. There are no limitations to the types of emotions, their number, or the number of categories. For example, instead of classifying into two categories, emotions may be classified into three or more categories, such as negative emotions, positive emotions, and neutral emotions (leaving the category column in Figure 4 unmarked).

<Step 2>
FIG. 5 is a flowchart of the emotion analysis process according to the embodiment.
As described above, after data initialization is completed, the "reaction detection system" (estimation unit 313) detects and counts people expressing various emotions from images taken by camera 5 or the like (SA11).
The “system for knowing the reaction” (estimation unit 313) identifies, as a result of the counting, whether or not there is any type of “group emotion” that exceeds the majority (type of emotion) (SA12). For example, the estimation unit 313 notifies the performance control unit 312 of the result.
If there is a type of emotion that exceeds the majority among the types of "group emotions," the "system to know the reaction" (performance control unit 312) identifies that there is a specific tendency in the "emotions common to the group" of the audience (SA13), and carries out processing according to the type of emotion identified as a result (SA14), thereby completing the series of processes.
On the other hand, if there is no type of "group emotion" that exceeds the majority (type of emotion), the "system for knowing the reaction" (performance control unit 312) ends the series of processes.

A more specific example will be given for explanation.
It is assumed that, as a result of the counting in step SA11, 10 or more of the 19 people express the emotion of "boredom" as shown in FIG.
In such a situation, the "system for knowing the reaction" (estimation unit 313) detects that 10 or more of the 19 people express the emotion of "boredom." In this case, the "system for knowing the reaction" (estimation unit 313) determines that the "emotion common to the group" is classified as "boredom."

When the "group emotion" is determined to be "boredom," the "system for knowing the reaction" (performance control unit 312) controls the lighting control equipment 2 to set the lighting in the conference room to 600 lux and to increase the proportion of blue light compared to white light, gradually changing the environment (light) in the conference room.

The "system to know the reaction" (performance control unit 312) will not determine the "group emotion" to be "boredom" even if less than 10 people out of 19 people express the emotion of "boredom." In this case, the "system to know the reaction" (performance control unit 312) will not change the environment (lighting) of the conference room from its normal state.

According to the above embodiment, the production control system 1 produces an environment in which a speaker, who is an information provider, provides information to an audience. The control unit 31 of the production control device 3 controls the production of the environment of the venue based on at least the results of emotion analysis that estimates the emotions of the audience. In this way, the production control system 1 can assist the speaker, who is an information provider, in understanding the situation of the audience.

As described above, the control unit 31 may change the environment of the venue by controlling the lighting using the lighting control equipment 2 so that information providers such as speakers, organizers, speakers, and facilitators can identify changes in the lighting in the venue. This allows each of the information providers to understand the situation of the audience.

By controlling the presentation in this way, the speaker can discern changes in the conference room environment (lighting) and casually learn in real time that the audience is feeling bored. Then, upon detecting this, the speaker can adjust the content of the presentation and the pace of the announcement.

The above process also increased the blue component ratio of the conference room environment (lighting) compared to white light, and light with a high blue component ratio has the effect of waking people up. Therefore, by using light with a high blue component ratio, it is possible to not only inform the speaker of the situation (emotions) of the participants, but also to wake up the audience in such a lighting environment, thereby reducing their sense of boredom.

In this way, when the control unit 31 detects a high tendency for a "boredom" state, which is identified as a relatively "low level of arousal" from the results of emotion analysis, it is preferable for the control unit 31 to control the environment of the location to promote arousal.

(Modification in which sound-related processing is applied in the "first scenario")
Furthermore, in addition to or instead of the image analysis by the control unit 31, the volume of voice/sound (sound pressure level) is detected by a microphone 6 installed in the conference room or a microphone installed in the remote terminal device 8, and the "emotion of the group" is judged based on the detected volume of voice/sound (sound pressure level). For example, when the "system for knowing the reaction" (performance control unit 312) detects a voice/sound with a noise level exceeding a predetermined value (about 90 dB), it may infer that there are people in the conference room expressing the emotion of "anger" or "fear," and judge the "emotion of the group" to be "anger" or "fear" without counting the number of people.
In this way, when using a microphone 6 or the like as a sensor, just as when using a camera 5 or the like as a sensor, it is advisable to adjust the illuminance and color rendering of the conference room according to the results of identifying the "emotions of the group" using the "system for knowing the reaction" (performance control unit 312).
When the result of the emotion analysis indicates a relatively negative tendency (negative emotion), the performance control unit 312 of the control unit 31 controls the environment of the venue to promote relaxation. For example, when the result of the emotion analysis indicates a high tendency of "anger" or "fear," the performance control unit 312 may control the environment of the venue to promote relaxation. The tendency of "anger" or "fear" may be identified based on the volume of sound. For example, the control unit 31 may control the performance of the environment of the venue based on the result of the emotion analysis that estimates the emotion of the person who raised his voice or yelled (the information provider). This enables each information provider to grasp the situation of the audience.

Second Embodiment
The second scenario will now be described with reference to the above figures and FIG.
The performance control system 1 of the second embodiment uses the estimation unit 313 to detect the ongoing situation in the facility and adjusts the scheduled usage time of the conference room based on the result. An example of this is shown in the second scenario.

Second Scenario:
In the second scenario illustrated here, as in the first scenario described above, a process is described according to the result of determining whether the "emotion of the group" is a "negative emotion" or a "positive emotion." After the above-mentioned "first scenario <Stage 1>" is completed, the following process is executed.

In this second scenario, the performance control system 1 provides a function to change (extend or shorten) the duration of a meeting or presentation by including the emotions of the participants in the conditions for adjusting the time (hereinafter referred to as the "duration change function").
The usage time reduction process by the performance control system 1 described in this embodiment includes the following two processing aspects.
The first mode is carried out in the middle of the usage time, and will be described in <Phase 2-A>.
The second mode is implemented towards the end of the usage time and will be described in <Phase 2-B>.
In addition, the performance control system 1 may implement either or both of the first and second aspects described above after the initialization process shown in "<Step 1>" as described above. As a method of providing information used here, an example in which the following lighting control is applied will be described.

<Step 2-A>
(Emotion analysis processing in the middle of usage time and processing to reduce usage time)
FIG. 6 is a flowchart of the emotion analysis process according to the embodiment.

When a predetermined time has elapsed since the start of the conference (SB11), the control unit 31 identifies the "emotion of the group" (SB12A) and, based on the result of the determination, determines whether or not the duration of the conference, in other words, the usage time of the facility to be used for the conference, needs to be adjusted (SB12B).
If the result of this determination indicates that adjustment of the facility usage time is necessary, the "system for knowing reactions" (performance control unit 312) identifies that there is a specific tendency in the "emotions common to the group" of audience members (SB13), and adjusts the facility usage time as processing according to the type of emotion identified (SB14), thereby completing the series of processes.
On the other hand, if there is no need to adjust the facility usage time, the "system for knowing the reaction" (performance control unit 312) ends the series of processes.

The above processing can be arranged as follows.
For example, a predetermined time from the start of a meeting may be associated with a configurable minimum meeting time, which may be preset, such as 20 minutes from the start of the meeting.

In addition, in step SB12B above, at any timing after the set shortest meeting time, the "system to know reactions" (control unit 31) determines whether the "group emotion" is a negative emotion and detects this. If this condition is met, the process may proceed immediately to step SB13 as described above, or alternatively, if a preset fixed time (e.g., 2 minutes) continues thereafter, the process may proceed to step SB13. After this fixed time has elapsed, the control unit 31 may repeat steps SB12A and AS12B, and depending on the result, may notify the participants that the meeting or presentation should be ended as processing in step SB14. Details of this production control will be described later.

This will be described in relation to specific times with reference to FIG.
FIG. 7 is a diagram for explaining the processing related to the performance control of the embodiment.
7 shows a model of the change in the environment caused by the performance control. For example, the horizontal axis shows the passage of time, and the vertical axis shows the quantified state of the environment.
When controlling the performance using lighting equipment, the information may include the illuminance of the room, the output of the lighting fixtures, color temperature, etc.

An example of a conference scheduled to be held from time t10 (e.g., 9:00) to time t30 (e.g., 10:00) is given below. In this example, attention is focused on the "emotion of the group" at time t50 (not shown: e.g., 9:20).
Assume that the "group emotion" determined by the "reaction recognition system" (control unit 31) at 9:20 was a negative emotion. The emotion was still determined to be negative at 9:22 (time t51). Therefore, the "conference time change function" (control unit 31) determines that it would be better to shorten the conference time beyond the scheduled end time of 10 o'clock, and notifies the participants that the conference or presentation should be ended.

If the "reaction detection system" (control unit 31) determines that the "group emotion" is not a negative emotion, it decides to continue the meeting as scheduled and does not implement the usage time reduction process.
At this time, the control unit 31 determines that it would be better to shorten the duration of the conference from the scheduled end time, and adjusts the output from the output control unit 314 or adjusts the lighting state of the lighting fixtures of the lighting equipment 2. When adjusting the lighting state of the lighting fixtures of the lighting equipment 2, the output control unit 314 controls the output unit 34 to reduce the illuminance of the room, reduce the output of the lighting fixtures, or change the color temperature to create a room environment, and notify participants of this situation.
The above is an example of control for adjusting the illuminance of the room, the output of the lighting fixture, and the color temperature, and the present invention is not limited to this.

<Step 2-B>
(Emotion analysis and usage time extension processing at the end of usage time)
If the above "<Step 2-A>" processing is omitted, or if the result of the "<Step 2-A>" processing is that it is decided to continue the conference as scheduled, a determination is made as to whether or not to perform usage time extension processing at the end of the usage time.

Figure 8 is a flowchart of the automatic extension process for usage time in an embodiment.

When a predetermined amount of time has elapsed since the start of the conference and before the end of the conference period, the control unit 31 determines that the "group emotion" is positive, and obtains the reservation information for the conference room at that time.

At that time, the "session time change function" (controller 31) determines whether another meeting is scheduled to take place immediately after the time indicated by the reservation information (SC15). If another meeting is scheduled, the controller 31 determines that extended use of the conference room is not possible, and guides the user in the direction of ending the meeting (SC16).

If no other meetings are scheduled, the control unit 31 determines that the conference room can be used for an extended period of time, and automatically extends the conference for a certain period of time (SC26). For example, the control unit 31 rewrites the conference room use time on the reservation system to the extended period of time accordingly. This certain period of time is preferably determined in advance.
The above "another meeting" may be defined as, for example, "a meeting registered by another employee, etc."

This procedure makes it possible to carry out emotion analysis processing towards the end of usage time and to extend usage time.

(Automatic re-extension of usage time)
As described above, after the usage time has been extended by the usage time extension process, further extension may be permitted. An example of this will be described.
After the above-mentioned step SC26, the "emotion of the group" is identified after a certain time has elapsed, and if the "emotion of the group" is still determined to be positive after the certain time has elapsed, the meeting time is automatically extended for a further certain time.
For example, a maximum extension time for a meeting may be preset, and participants may be notified that the meeting or presentation should end five minutes before the maximum extension time is reached or five minutes before the end of the extended meeting time.

For example, in a meeting scheduled to take place from 9:00 to 10:00, at 9:50, the "reaction detection system" (control unit 31) determines whether the "group emotion" is positive or not.

As a result, the "group emotion" determined by the "system to know reactions" (control unit 31) was a positive emotion. At this time, when the "function to change the conference time" (control unit 31) read the conference reservation information, it found that there was no next reservation, so it automatically rewrote the conference reservation information to extend it by 15 minutes. The conference room reservation time was automatically changed to 9:00 to 10:15. At 10:05, the "group emotion" determined by the "system to know reactions" (control unit 31) was a positive emotion again. When the "function to change the conference time" read the conference reservation information, it found that there was no next reservation after 10:15, so it automatically rewrote the conference reservation information to extend it by 15 minutes. The conference room reservation time was automatically changed to 9:00 to 10:30. Since the maximum extension time had been set to 30 minutes in advance, the wrap-up performance that had been separately applied for was performed at 10:25.

According to the above embodiment, the performance control system 1 uses the estimation unit 313 to detect the ongoing situation within the facility and adjusts the scheduled use time of the conference room based on the result, thereby achieving the same effect as the first embodiment.

Third Embodiment
In the third embodiment, a presentation function using light, sound, and fragrance (hereinafter, referred to as "presentation function using light, sound, and fragrance") will be described.

Third Scenario:
In the third scenario illustrated here, similar to the first scenario described above, a process is described according to the result of determining whether the "emotion of the group" is a "negative emotion" or a "positive emotion." In this process, a performance function using light, sound, and scent is provided.

<Step 2-C>
After completing the above-mentioned "First Scenario <Stage 1>", the following procedure is executed.
For example, the control unit 31 periodically performs the following determination, regardless of the amount of time that has elapsed since the start of the conference or presentation.
The control unit 31 identifies whether or not the time during which the "group emotion" is a negative emotion continues for a preset certain period of time (for example, three minutes) or more.
If the "group emotion" remains negative for a certain period of time,
The control unit 31 operates some or all of the following functions installed in the conference room or presentation environment.

Relaxation music is played from a speaker installed in the room. It is advisable to select the type of music to be played in advance. The speaker is an example of the output unit 34.
The output unit 34 controls the aroma to be diffused from an aroma diffuser to promote relaxation. The type of aroma to be diffused may be selected in advance. The aroma diffuser is an example of the output unit 34.
- To create an environment that encourages relaxation by implementing lighting control that gradually increases and/or decreases the illuminance and output of the luminaires in the room. For example, to create an environment that encourages relaxation by implementing lighting control that temporarily decreases the illuminance and output of the luminaires in the room from the normal state. It is advisable to select the illuminance and output of the luminaires in the room to be implemented in advance.
- To create an environment that promotes relaxation by implementing lighting control that gradually raises and/or lowers the color temperature of the lighting. For example, to create an environment that promotes relaxation by implementing lighting control that gradually and temporarily lowers the color temperature of the lighting. It is advisable to select the color temperature to be implemented in advance.
Using a projector, an image that promotes relaxation is projected onto the wall of the room. It is advisable to select the type of image to be projected in advance. This projector may be an example of the output unit 34, and may be integrated into the lighting control device 2.

There are no limitations on the sensing method used to execute the above-mentioned environmental rendering process, and it may be based on the results detected from an image or the results detected from sound characteristics.

For example, during the above-mentioned production control, the control unit 31 may change the environment of the venue through acoustic control within a range that does not affect the transmission of the voice of the speaker who is the information provider.
During the above-mentioned performance control, the control unit 31 may change the environment of the location by executing scent control so that the change in scent reaches the information provider.
During the above-mentioned performance control, the control unit 31 may change the environment of the venue by controlling the lighting using a projector so that the information provider can identify changes in the lighting of the venue.

(Fourth embodiment)
In the fourth embodiment, the awakening support function will be described.
In the fourth scenario illustrated here, a process will be described depending on whether the "emotion of the group" is in a "state of low arousal."

Fourth Scenario:
In this fourth scenario, a wakefulness support function is provided that takes emotions into account through lighting and scents to enable meetings to be held efficiently and comfortably.

<Step 1-A>
An initial state for classifying "emotions of the group" is set so as to identify the "state of arousal." For example, some or all of the "emotions of the group" are classified in advance into two types: "low arousal state" and "not low arousal state." For example, "boredom" is set to correspond to "low arousal state."

<Step 2-E>
For example, the control unit 31 periodically performs the following determination, regardless of the amount of time that has elapsed since the start of the conference or presentation.
The control unit 31 identifies whether the "emotion of the group" is in a "state of low arousal" for a period of time that is longer than a preset certain period of time (for example, 3 minutes).
If the "emotion of the group" remains in a "state of low arousal" for a certain period of time or longer, the control unit 31 will activate some or all of the following functions installed in the conference room or presentation environment:

- Music that promotes awakening is played from a speaker installed in the room. It is advisable to select the type of music to be played in advance. The speaker is an example of the output unit 34.
The output unit 34 controls the aroma to be diffused from an aroma diffuser to stimulate the user's awakening. The type of aroma to be diffused may be selected in advance. The aroma diffuser is an example of the output unit 34.
- To create an environment that encourages awakening by gradually increasing or decreasing the illuminance and/or output of the luminaire in the room. For example, to create an environment that encourages awakening by temporarily increasing the illuminance and output of the luminaire in the room from the normal state. It is advisable to select the illuminance and output of the luminaire in the room to be created in advance.
- To create an environment that encourages wakefulness by implementing lighting control that gradually raises and/or lowers the color temperature of the lighting. For example, to create an environment that encourages wakefulness by implementing lighting control that temporarily raises the color temperature of the lighting. It is advisable to select the color temperature to be implemented in advance.
Using a projector or the like, an image that promotes awakening is projected onto the wall of the room. It is advisable to select the type of image to be projected in advance. This projector may be an example of the output unit 34, and may be integrated into the lighting control device 2.

There are no limitations on the sensing method used to execute the above-mentioned environmental rendering process, and it may be based on the results detected from an image or the results detected from sound characteristics.

According to the above embodiment, "group emotions" can contribute to resolving issues caused by a "state of low arousal."

(Effects common to all embodiments)
・The "Reaction Monitoring System" allows organizers and speakers to easily know the reactions of the audience, even in meetings and presentations that use remote meetings. Even if organizers and speakers have difficulty reading people's emotions due to disabilities or other reasons, it is expected that they will be able to visually obtain the evaluation results of the audience's reactions.

- The "reaction monitoring system" makes it possible to carry out objective evaluations even when the number of attendees is large and it is difficult for the organizers, speakers, etc. to grasp the emotions of each individual.
In the comparative example, the more attendees there were, the more difficult it became to grasp the emotions of each individual. However, by applying this embodiment, it becomes possible to carry out an objective evaluation.

- With the "change meeting time feature," if a discussion or presentation is in full swing and extending it would increase team productivity, the discussion or presentation can be extended without being interrupted at the scheduled end time, even when the scheduled end time for the meeting or presentation has come. This is expected to improve the team's results that day. At the same time, it eliminates the need for meeting or presentation participants to check the reservation status of the conference room or presentation environment themselves, visually confirm that there are no further reservations and that the meeting can be extended, and then decide to extend the meeting, improving efficiency.

- The "change meeting time feature" can be used to shorten meetings that evoke negative emotions. Meetings that evoke negative emotions are generally considered to be less productive. By shortening the time required for such meetings, you can reduce unnecessary work by your team.

・The "light, sound and scent production function" increases the likelihood that a meeting will be held in a comfortable, productive and creative atmosphere, without relying solely on the facilitator's personal skills to change the atmosphere when one participant intimidates or annoys other participants, such as by yelling loudly. This reduces the chances of the meeting ending up being inefficient and unpleasant.
In particular, the "light effect function" can induce the behavior of lowering the volume of one's voice by making the person see the red light effect.

- The "Awakening Support Function" is expected to help resolve low awakening states.

The production control system 1 according to any of the above embodiments produces an environment in which an information provider provides information to an audience. The production control system includes a production control unit that controls the production of the environment in which the information provider provides information based on the results of emotion analysis that estimates the emotion of the information provider or the audience, thereby helping the information provider to understand the emotion of the audience.

The embodiment of the present invention has been described above, but the performance control system 1 has an internal computer system. The series of steps related to the above-mentioned processes are stored in the form of a program on a computer-readable storage medium, and the above-mentioned processes are performed by the computer reading and executing this program. Here, computer-readable storage medium refers to a magnetic disk, magneto-optical disk, CD-ROM, DVD-ROM, semiconductor memory, etc. Also, this computer program may be distributed to a computer via a communication line, and the computer that receives this distribution may execute the program. Also, the "computer system" here includes an OS, etc.

Although one embodiment of the present invention has been described above, the embodiment of the present invention is not limited to the above. For example, the methods exemplified in each embodiment and its modified examples may be combined in combinations other than those exemplified. Furthermore, the embodiment of the present invention may be a modification of the above embodiment as follows.

REFERENCE SIGNS LIST 1: performance control system 2: lighting control equipment 21: first lighting control equipment 22: second lighting control equipment 3: performance control device 31: control unit 311: facility use reservation information management unit 312: performance control unit 313: estimation unit 32: storage unit 33: remote conference monitor unit 34: output unit 4: terminal device 5: camera 7: remote conference system 8: remote terminal device

Claims (10)

A production control system for producing an environment of a presentation in which an information provider provides information to an audience, the participants of the presentation including the information provider of the presentation, a first audience member participating in the presentation online, and a second audience member participating in the presentation in a venue where the presentation is held , the system producing an environment of the venue using a result of emotion analysis of the audience members,
The correspondence between the emotion analysis results and the performance of the controlled object is preset,
obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener , and estimating the emotion of the group of listeners based on the number of listeners;
a control unit that controls the presentation of the environment of the venue based on a result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship;
The control unit is
controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location;
Production control system. The control unit is
identifying emotions of the audience for two or more categories to estimate emotions of the audience;
an output unit that outputs a result of sound control, and a sound control method selected from two or more sound control methods based on the estimation result of the emotion of the group and the correspondence relationship is used to change the environment of the audience by the sound control of the sound control method.
The production control system according to claim 1 . The control unit is
identifying emotions of the audience for two or more categories to estimate emotions of the audience;
an output unit that outputs a result of the scent control performance control, and changing the environment of the venue for the listeners by the scent control of a performance method selected from two or more performance methods based on the estimation result of the emotion of the group and the correspondence relationship;
The production control system according to claim 1 . The control unit is
identifying emotions of the audience for two or more categories to estimate emotions of the audience;
an output unit that outputs a result of lighting control, and changing an environment of the place by the lighting control of a lighting control method selected from two or more lighting methods based on the estimation result of the emotion of the group and the correspondence relationship;
The production control system according to claim 1 . The control unit is
When the estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively positive tendency, the usage time of the place is extended;
shortening a usage time of the venue when an estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively negative tendency;
A performance control system according to any one of claims 1 to 4 . The control unit is
The emotions of the listeners are identified for two classification items including items with a relatively positive tendency and items with a relatively negative tendency, and the number of people who showed the same classification item emotion is counted for each classification item, and emotions common to the listeners are identified from the counting results.
When the estimation result of the emotion of the group based on the result of the emotion analysis indicates a relatively negative tendency, the environment of the place is controlled so as to promote relaxation of the place.
A performance control system according to any one of claims 1 to 4 . The control unit is
The performance control system according to any one of claims 1 to 4 , wherein when a high tendency of "anger" or "fear" is detected from the results of the emotion analysis, the environment of the venue is controlled to create an environment that encourages relaxation. The control unit is
The performance control system according to any one of claims 1 to 4 , wherein when the result of the emotion analysis detects a high tendency for a "low level of arousal," the environment of the venue is controlled to promote arousal. A method for controlling presentation using a presentation control device, the method comprising the steps of: participants in a presentation in which an information provider provides information to an audience include the information provider of the presentation, a first audience participating in the presentation online , and a second audience participating in the presentation in person at a venue where the presentation is held ; and the method for controlling a presentation environment using a result of emotion analysis of the audience, the method comprising the steps of:
The correspondence between the emotion analysis results and the performance of the controlled object is preset,
obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener , and estimating the emotion of the group of listeners based on the number of listeners;
A step of controlling the presentation of the environment of the venue based on a result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship;
A step of controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location ;
A production control method including: Participants of a presentation in which an information provider provides information to an audience include the information provider of the presentation, a first audience member who participates in the presentation online , and a second audience member who gathers at a venue where the presentation is held , and a computer of a production control device that produces an environment of the venue using a result of emotion analysis of the audience ,
The correspondence between the emotion analysis results and the performance of the controlled object is preset,
obtaining the number of listeners who show the same emotion from the result of emotion analysis of each listener including the first listener and the second listener , and estimating the emotion of the group of listeners based on the number of listeners;
controlling the presentation of the environment of the venue based on a result of emotion analysis, which is an estimation result of the emotion of the group of listeners, and the correspondence relationship;
A step of controlling an output from an output unit provided in the location where the information provider is present, to produce an environment of the location;
A program for executing.

Images