JP7705176B2 - Video Analysis System - Google Patents

Description

The present invention relates to a video image analysis system that analyzes the biological reactions of participants based on video images obtained through an online session conducted with multiple participants.

There is known a technique for analyzing the emotions felt by others in response to a speaker's utterance (see, for example, Patent Document 1). There is also known a technique for chronologically analyzing changes in a subject's facial expression over a long period of time and estimating the emotions felt during that time (see, for example, Patent Document 2). There is also known a technique for identifying the factor that most influenced changes in emotions (see, for example, Patent Documents 3 to 5). There is also known a technique for comparing a subject's usual facial expression with their current facial expression and issuing an alert if the facial expression is gloomy (see, for example, Patent Document 6). There is also known a technique for comparing a subject's normal (expressionless) facial expression with their current facial expression to determine the degree of the subject's emotions (see, for example, Patent Documents 7 to 9). There are also known techniques for analyzing organizational emotions and the atmosphere within a group as felt by individuals (see, for example, Patent Documents 10 and 11).

JP 2019-58625 A JP 2016-149063 A JP 2020-86559 A JP 2000-76421 A JP 2017-201499 A JP 2018-112831 A JP 2011-154665 A JP 2012-8949 A JP 2013-300 A JP 2011-186521 A WO15/174426 publication

All of the above technologies are merely secondary functions in situations where communication in the real world is the main focus. In other words, they were not created in a situation where communication for work, classes, etc. is primarily conducted online, in response to the recent digital transformation (DX) of work and the global pandemic.

The present invention aims to objectively evaluate communications in situations where online communication is the norm, such as meetings and lectures, in order to make such communications more efficient.

According to the present invention,
A video analysis system for analyzing a reaction of a user based on a video obtained by photographing the user in an environment in which the user is performing an online session, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
An analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit;
a providing unit that provides a result of the analysis to other users;
A matching unit that performs matching with the other users.
A video image analysis system is obtained.

According to the present disclosure, by analyzing and evaluating the video footage of a video session, evaluation, particularly with regard to content, can be made objectively.

In particular, the present invention makes it possible to objectively evaluate communication in order to communicate more efficiently in situations where online communication is the norm.

FIG. 1 is a diagram showing an overall system according to an embodiment of the present invention. FIG. 2 is a functional block diagram of an evaluation terminal according to an embodiment of the present invention; FIG. 2 is a diagram illustrating a first example of a functional configuration of an evaluation terminal according to an embodiment of the present invention. FIG. 11 is a diagram illustrating a second example of a functional configuration of the evaluation terminal according to the embodiment of the present invention. FIG. 11 is a diagram illustrating a third example of a functional configuration of the evaluation terminal according to an embodiment of the present invention. FIG. 13 is a diagram illustrating another configuration of the functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. FIG. 13 is a diagram illustrating another configuration of the functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. FIG. 1 is a functional block diagram of a system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of how a system is used in an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of how a system is used in an embodiment of the present invention.

The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration.
[Item 1]
A video analysis system for analyzing a reaction of a user based on a video obtained by photographing the user in an environment in which the user is performing an online session, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
An analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit;
a providing unit that provides a result of the analysis to other users;
A matching unit that performs matching with the other users.
Video image analysis system.
[Item 2]
2. The video analysis system according to claim 1,
A content/service providing unit that provides content or services to the matched other users,
Video image analysis system.
[Item 3]
The video analysis system according to claim 1 or 2,
An emotion evaluation unit that evaluates a degree of emotion of the user according to an evaluation criterion that is standardized among a plurality of users based on the change in the biological reaction of the user analyzed by the analysis unit,
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, the degree being adjusted according to the likelihood of the user experiencing the same emotion;
Video image analysis system.
[Item 4]
A moving image analysis system according to any one of items 1 to 3,
a uniqueness determination unit that determines whether or not the change in the biological reaction analyzed for the user in one online session is unique compared to the change in the biological reaction analyzed for the user in an online session temporally preceding the one online session;
A clustering unit that clusters the change pattern of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specific determination unit and the magnitude of the change from the past.
Video image analysis system.
[Item 5]
A moving image analysis system according to any one of items 1 to 4,
an emotion evaluation unit that evaluates the degree of emotion of the subject according to an evaluation criterion that is standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit;
The emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the subject having the same emotion.
Video image analysis system.

A preferred embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

<Basic functions>
The video session evaluation system of the present embodiment is a system that analyzes and evaluates a unique emotion (feelings caused by one's own or other's words and actions, such as pleasantness or unpleasantness, or the degree of pleasantness) of an analysis subject among a plurality of people in an environment where the video session (hereinafter, one-way and two-way are both referred to as online sessions) is held by the plurality of people. The online session is, for example, an online conference, an online class, an online chat, etc., and terminals installed in a plurality of places are connected to a server via a communication network such as the Internet, and moving images can be exchanged between the plurality of terminals through the server. The moving images handled in the online session include facial images and voices of users using the terminals. The moving images also include images of materials shared and viewed by a plurality of users. It is possible to switch between a facial image and a material image on the screen of each terminal to display only one of them, or to divide the display area to display a facial image and a material image simultaneously. It is also possible to display an image of one of the plurality of people in full screen, or to display images of some or all of the users divided into small screens. It is possible to designate one or more of the multiple users who participate in the online session using a terminal as the analysis target. For example, the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) designates one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of the multiple users who participate in the online session, but may be a different person who does not participate in the online session. It is also possible to designate all participants as the analysis target without designating an analysis target. It is also possible for the leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer) to designate one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of the multiple users who participate in the online session, but may be a different person who does not participate in the online session.

In the video session evaluation system according to the present embodiment, when a video session is established between multiple terminals, at least a moving image acquired from the video session is displayed. The displayed moving image is acquired by the terminal, and at least a facial image contained in the moving image is identified for each predetermined frame unit. An evaluation value for the identified facial image is then calculated. The evaluation value is shared as necessary. In particular, in the present embodiment, the acquired moving image is stored in the terminal, analyzed and evaluated on the terminal, and the result is provided to the user of the terminal. Therefore, even if a video session includes personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency, etc. Also, if necessary, the evaluation result (evaluation value) alone can be provided to an external terminal, allowing the results to be visualized and cross-analysis, etc. to be performed.

As shown in FIG. 1, the video session evaluation system of this embodiment includes user terminals 10, 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10, 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

<Hardware configuration example>
Each functional block, functional unit, and functional module described below can be configured by, for example, hardware, DSP (Digital Signal Processor), or software provided in a computer. For example, when configured by software, the computer is actually configured with a CPU, RAM, ROM, etc., and is realized by the operation of a program stored in a recording medium such as a RAM, ROM, hard disk, or semiconductor memory. A series of processes by the system and terminal described in this specification may be realized by using any of software, hardware, and a combination of software and hardware. A computer program for realizing each function of the information sharing support device 10 according to this embodiment can be created and implemented in a PC or the like. In addition, a computer-readable recording medium in which such a computer program is stored can also be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. In addition, the above computer program may be distributed, for example, via a network, without using a recording medium.

The evaluation terminal in this embodiment acquires moving images from a video session service terminal, identifies at least facial images contained in the moving images for each predetermined frame unit, and calculates an evaluation value for the facial images (details will be described later).

<How to get the video>
As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter, sometimes simply referred to as "this service") enables two-way communication with user terminals 10 and 20 using images and audio. This service is capable of displaying video images captured by the camera unit of the other user terminal on the display of the user terminal, and outputting audio captured by the microphone unit of the other user terminal from the speaker. This service is also configured to enable both or any of the user terminals to record video images and audio (collectively referred to as "video images, etc.") in the memory unit of at least one of the user terminals. The recorded video information Vs (hereinafter, referred to as "recorded information") is cached in the user terminal that started the recording, and is recorded only locally in one of the user terminals. If necessary, the user can view the recorded information himself or share it with others within the scope of use of this service.

<Functional configuration example 1>
Fig. 3 is a block diagram showing an example of the configuration according to this embodiment. As shown in Fig. 4, the video session evaluation system of this embodiment is realized as a functional configuration of a user terminal 10. That is, the user terminal 10 includes, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

The video acquisition unit 11 acquires video from each terminal, obtained by photographing multiple people (multiple users) with a camera provided in each terminal during an online session. The video acquired from each terminal may or may not be set to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including video being displayed and video not being displayed on each terminal.

The bioreaction analysis unit 12 analyzes changes in bioreactions for each of the multiple people based on the video acquired by the video acquisition unit 11 (whether or not it is displayed on the screen). In this embodiment, the bioreaction analysis unit 12 separates the video acquired by the video acquisition unit 11 into a set of images (a collection of frame images) and audio, and analyzes changes in bioreactions from each of them. For example, the bioreaction analysis unit 12 analyzes changes in bioreactions related to at least one of facial expression, gaze, pulse, and facial movement by analyzing the face image of the user using the frame images separated from the video acquired by the video acquisition unit 11. In addition, the bioreaction analysis unit 12 analyzes changes in bioreactions related to at least one of the user's remarks and voice quality by analyzing the audio separated from the video acquired by the video acquisition unit 11.

When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, gaze, pulse rate, facial movements, speech, and voice quality. In this embodiment, the change in the user's emotions is analyzed by analyzing the change in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort/discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies the change in the biological reaction according to a predetermined standard, thereby calculating a biological reaction index value that reflects the content of the change in the biological reaction.

Analysis of changes in facial expressions is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image, and the identified facial expressions are classified into multiple categories according to an image analysis model that has been trained by machine learning in advance. Then, based on the classification results, it is analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and the extent of the facial expression change, and a facial expression change index value according to the analysis results is output.

Analysis of changes in eye line is performed, for example, as follows. That is, for each frame image, the eye area is identified from within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, at the shared material being displayed, or looking outside the screen. It may also be possible to analyze whether the eye line movement is large or small, and whether the movement is frequent or infrequent. Changes in eye line are also related to the user's level of concentration. The biological response analysis unit 12 outputs an eye line change index value according to the analysis result of the eye line change.

The analysis of changes in pulse rate is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image. Then, using a trained image analysis model that captures the numerical values of facial color information (G in RGB), changes in the G color of the face surface are analyzed. The results are arranged along the time axis to form a waveform representing changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate decreases. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of changes in pulse rate.

The analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, the facial area is identified from within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker being displayed, the shared material being displayed, or looking outside the screen. It is also possible to analyze whether the facial movement is large or small, and whether the movement occurs frequently or infrequently. It is also possible to analyze the facial movement and the eye movement together. For example, it is possible to analyze whether the user is looking straight at the speaker being displayed, looking up or down, or looking at an angle. The biological response analysis unit 12 outputs a facial direction change index value according to the analysis result of the change in facial direction.

The analysis of the speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts the speech into a string of characters by performing a known speech recognition process on the speech for a specified period of time (for example, about 30 to 150 seconds), and performs morphological analysis on the string of characters to remove words that are unnecessary for expressing the conversation, such as particles and articles. The remaining words are then vectorized, and an analysis is performed to determine whether a positive or negative emotional change has occurred, and to what extent the emotional change has occurred, and a speech content index value corresponding to the analysis result is output.

Voice quality analysis is performed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic characteristics of the voice by performing a known voice analysis process on the voice for a specified period of time (for example, about 30 to 150 seconds). Then, based on the acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred and to what extent the voice quality change has occurred, and outputs a voice quality change index value according to the analysis result.

The biological reaction analysis unit 12 calculates a biological reaction index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological reaction index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, face direction change index value, speech content index value, and voice quality change index value.

The uniqueness determination unit 13 determines whether the change in the biological reaction analyzed for the subject of analysis is unique compared to the change in the biological reaction analyzed for other people other than the subject of analysis. In this embodiment, the uniqueness determination unit 13 determines whether the change in the biological reaction analyzed for the subject of analysis is unique compared to other people based on the biological reaction index values calculated for each of the multiple users by the biological reaction analysis unit 12.

For example, the uniqueness determination unit 13 calculates the variance of the biological reaction index values calculated for each of multiple individuals by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the subject of analysis with the variance, determines whether or not the changes in the biological reactions analyzed for the subject of analysis are unique compared to others.

There are three possible cases where the changes in the analyzed biological reactions of the subject are unique compared to others. The first is when no particularly significant changes in the biological reactions of others occur, but a relatively large change in the biological reaction of the subject occurs. The second is when no particularly significant changes in the biological reactions of the subject occur, but a relatively large change in the biological reaction of others occurs. The third is when a relatively large change in the biological reaction occurs in both the subject and others, but the content of the change differs between the subject and others.

The related event identification unit 14 identifies an event occurring with respect to at least one of the subject of analysis, other people, and the environment when a change in a biological reaction determined to be unique by the uniqueness determination unit 13 occurs. For example, the related event identification unit 14 identifies the behavior of the subject of analysis himself/herself from a video when a unique change in a biological reaction occurs in the subject of analysis. The related event identification unit 14 also identifies the behavior of other people from the video when a unique change in a biological reaction occurs in the subject of analysis. The related event identification unit 14 also identifies the environment from the video when a unique change in a biological reaction occurs in the subject of analysis. The environment is, for example, shared materials being displayed on the screen, or something that appears in the background of the subject of analysis, etc.

The clustering unit 15 analyzes the degree of correlation between a change in a biological reaction determined to be unique by the unique determination unit 13 (e.g., one or more combinations of eye movement, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in a biological reaction occurs (an event identified by the related event identification unit 14), and if it is determined that the correlation is at or above a certain level, it clusters the person or event being analyzed based on the results of the correlation analysis.

For example, if a specific change in a biological reaction corresponds to a negative emotional change, and the event occurring when the specific change in the biological reaction occurs is also a negative event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or the event into one of multiple pre-segmented classifications according to the content of the event, the degree of negativity, the magnitude of correlation, etc.

Similarly, if a specific change in biological reaction corresponds to a positive emotional change, and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or the event into one of multiple pre-segmented classifications according to the content of the event, the degree of positivity, the magnitude of correlation, etc.

The analysis result notification unit 16 notifies the person who designated the analysis subject (the analysis subject or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the uniqueness determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

For example, the analysis result notification unit 16 notifies the analysis subject of the analysis subject's own words and actions as an event occurring when a unique change in the analysis subject's biological reaction that differs from that of others occurs (any of the three patterns described above; the same applies below). This allows the analysis subject to understand that when he or she behaves in a certain way, he or she has different emotions than others. At this time, the analysis subject may also be notified of the unique changes in the analysis subject's biological reaction identified. Furthermore, the analysis subject may also be notified of the changes in the biological reaction of the other person being compared.

For example, if the emotions felt by others in response to words or actions made by the subject without any particular awareness and with normal emotions, or words or actions made by the subject with particular emotions and awareness differ from the emotions felt by the subject himself at the time of the words or actions, the subject will be notified of the words or actions of the subject at that time. This makes it possible to discover words or actions that are well-received by others or not well-received by others despite one's own awareness.

Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events that occur when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, along with the unique change in the biological reaction. This allows the organizer of the online session to know what events are influencing what emotional changes as a phenomenon unique to the specified analysis subject. Then, it becomes possible to take appropriate measures for the analysis subject depending on the content thus understood.

Furthermore, the analysis result notification unit 16 notifies the organizer of the online session of events occurring when a unique change in the biological reaction of the analysis subject occurs that is different from that of others, or the clustering results of the analysis subject. This allows the organizer of the online session to grasp the behavioral tendencies unique to the analysis subject and predict possible future behaviors and conditions, etc., depending on which classification the specified analysis subject is clustered into. It then becomes possible to take appropriate measures for the analysis subject.

In the above embodiment, a bioreaction index value is calculated by quantifying the change in bioreaction according to a predetermined standard, and based on the calculated bioreaction index value for each of a plurality of people, it is determined whether or not the change in the bioreaction analyzed for the subject of analysis is unique compared to others. However, the present invention is not limited to this example. For example, the following may be used.

That is, the biological reaction analysis unit 12 analyzes the eye movement of each of the multiple people to generate a heat map showing the direction of the eye movement. The peculiar determination unit 13 compares the heat map generated by the biological reaction analysis unit 12 for the analysis subject with the heat map generated for other people to determine whether the change in the biological reaction analyzed for the analysis subject is unique compared to the change in the biological reaction analyzed for other people.

In this manner, in this embodiment, the video of the video session is stored in the local storage of the user terminal 10, and the above-mentioned analysis is performed on the user terminal 10. Although it may depend on the machine specifications of the user terminal 10, it is possible to analyze the video information without providing it to an external party.

<Functional configuration example 2>
As shown in FIG. 4, the video session evaluation system of the present embodiment may include a video image acquisition unit 11, a biological reaction analysis unit 12, and a reaction information presentation unit 13a as functional configurations. The reaction information presentation unit 13a presents information indicating changes in biological reactions analyzed by the biological reaction analysis unit 12a, including participants not displayed on the screen. For example, the reaction information presentation unit 13a presents information indicating changes in biological reactions to a leader, facilitator, or manager of the online session (hereinafter, collectively referred to as the organizer). The organizer of the online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is usually one of multiple users participating in the online session, but may be another person who does not participate in the online session.

In this way, the organizer of an online session can keep track of participants who are not displayed on the screen in an environment where an online session is being held with multiple people.

<Functional Configuration Example 3>
Fig. 5 is a block diagram showing a configuration example according to this embodiment. As shown in Fig. 5, in the video session evaluation system of this embodiment, functions similar to those of the first embodiment described above may be assigned the same reference numerals as in the functional configuration, and the description thereof may be omitted. The system according to this embodiment includes a camera unit for acquiring video of the video session, a microphone unit for acquiring audio, an analysis unit for analyzing and evaluating the video, an object generation unit for generating a display object (described later) based on information obtained by evaluating the acquired video, and a display unit for displaying both the video of the video session and the display object during the execution of the video session.

As explained above, the analysis unit includes a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

Based on the result of the analysis of the video acquired from the video session by the analysis unit, the object generation unit displays an object indicating the recognized face part and information indicating the above-mentioned analyzed and evaluated contents superimposed on the video, as necessary. When the faces of multiple people are moving in the video, the object may identify and display the faces of all the multiple people. In addition, even if the camera function of the video session is stopped on the other party's terminal (i.e., the camera is not physically covered, but is stopped by software in the video session application), if the other party's face is recognized by the other party's camera, the object may be displayed in the part where the other party's face is located. This makes it possible for both parties to confirm that the other party is in front of the terminal even if the camera function is turned off. In this case, for example, in the video session application, the information acquired from the camera may be hidden, while an object corresponding to the face recognized by the analysis unit may be displayed. In addition, the video information acquired from the video session and the information recognized and obtained by the analysis unit may be divided into different display layers, and the layer related to the former information may be hidden. When there are areas for displaying a plurality of video images, the object may be displayed in all areas or only in a part of the areas. For example, the object may be displayed only in the video image on the guest side.

The embodiments of the invention described in the above-described basic configuration examples 1 to 3 may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) partially or entirely connected via a network. For example, the control unit 110 and the storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, the present system includes user terminals 10 and 20, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video session, and the following variations and combinations of configurations are possible.
(1) All processing is done solely on the user's terminal. As shown in FIG. 6, by performing processing by the analysis unit on the terminal performing the video session, it is possible to obtain analysis and evaluation results simultaneously (in real time) with the time the video session is being performed (although a certain level of processing power is required).
(2) Processing in the user terminal and the evaluation terminal As shown in Fig. 7, an analysis unit may be provided in the evaluation terminal connected by a network or the like. In this case, the video acquired by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, information on the objects 50 and 100 is shared with the user terminal together with or separately from the video data (i.e., information including at least the analysis data) and displayed on the display unit.

The following system is realized using each of the configurations of functional configuration example 1 to functional configuration example 3 described above or a combination of them.

<Embodiment>
The moving image analysis system according to this embodiment is a moving image analysis system that analyzes user reactions based on moving images obtained by photographing a user in an environment in which multiple users are participating in online sessions, regardless of whether the user is displayed on the screen during the online session.

The concept underlying this system will be explained with reference to FIG. 8. In conventional psychology, there is a concept that one becomes aware of something, becomes self-aware through that awareness, and finally resolves and realizes oneself by disclosing the self-awareness to others. On the other hand, in the present invention, the above-mentioned analysis unit (see FIG. 3 to FIG. 7) makes it easy to detect one's own state (become aware), and it also makes it possible to disclose oneself to others (even if one is not self-aware). In other words, in the conventional psychology process, one was in a state where one could not disclose oneself without self-awareness, but according to the present invention, it is now possible to self-awareness (make oneself aware) by disclosing one's "awareness" to others before self-awareness and relying on the help of others.

As shown in Figure 9, the system includes a provision unit that acquires video data of user A, analyzes changes in the user's biological reactions based on the acquired video data (see Figures 3 to 7), and provides the results of the analysis to users B to D. Users B to D can promote (all or part of) the changes in user A's biological reactions provided by the provision unit, and match users who have been requested to be matched with the evaluation results.

For example, if user A's biological reaction changes are negative, user B, who has discovered the negative situation, will request a match and the two will be matched. User B will be able to take some kind of action (encouragement, advice, etc.) for user A, who is unaware of the negative biological reaction.

Also, as shown in FIG. 10, a mechanism may be used in which content or services aimed at user A are provided in exchange for user B taking some action toward user A. In this case, for example, user B can send user A some kind of gift or some kind of content. User B can also introduce appropriate counseling and recommend the counseling service to user A.

The above-described embodiments may be implemented in appropriate combinations. Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

10, 20 User terminal 30 Video session service terminal 40 Evaluation terminal

Claims (5)

A video analysis system for analyzing a reaction of a user based on a video obtained by photographing the user in an environment in which the user is performing an online session, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
An analysis unit that analyzes a change in a biological reaction of the first user based on the moving image acquired by the moving image acquisition unit;
A providing unit that provides a result of the analysis to another second user;
and a matching unit that performs matching so that an action can be taken from the second user to the first user in response to a request from the second user who has received the result of the analysis .
Video image analysis system. The video analysis system according to claim 1 ,
the action being the provision of content or a service;
a content service providing unit that provides the content or service to the first user in response to a request from the second user,
Video image analysis system. The video analysis system according to claim 1 or 2,
An emotion evaluation unit that evaluates a degree of emotion of the user according to an evaluation criterion that is standardized among a plurality of users based on the change in the biological reaction of the user analyzed by the analysis unit,
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, the degree being adjusted according to the likelihood of the user experiencing the same emotion;
Video image analysis system. The video analysis system according to any one of claims 1 to 3,
a uniqueness determination unit that determines whether or not the change in the biological reaction analyzed for the user in one online session is unique compared to the change in the biological reaction analyzed for the user in an online session temporally preceding the one online session;
A clustering unit that clusters the change pattern of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specific determination unit and the magnitude of the change from the past.
Video image analysis system. The video analysis system according to any one of claims 1 to 4,
an emotion evaluation unit that evaluates the degree of emotion of the subject according to an evaluation criterion that is standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit;
The emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the subject having the same emotion.
Video image analysis system.

Images