JP7745906B2 - Video analysis system - Google Patents

Description

The present invention relates to a video image analysis system that analyzes the biological responses of participants based on video images obtained during an online session conducted with multiple participants.

Technology is known that analyzes the emotions felt by others in response to a speaker's comments (see, for example, Patent Document 1). Technology is also known that analyzes changes in a subject's facial expression over a long period of time and estimates the emotions felt during that time (see, for example, Patent Document 2). Technology is also known that identifies the factors that most influenced changes in emotions (see, for example, Patent Documents 3 to 5). Technology is also known that compares a subject's usual facial expression with their current facial expression and issues an alert if the facial expression is gloomy (see, for example, Patent Document 6). Technology is also known that compares a subject's normal (expressionless) facial expression with their current facial expression to determine the subject's level of emotion (see, for example, Patent Documents 7 to 9). Technology is also known that analyzes organizational emotions and the atmosphere felt by individuals within a group (see, for example, Patent Documents 10 and 11).

JP 2019-58625 A JP 2016-149063 A Japanese Patent Application Laid-Open No. 2020-86559 Japanese Patent Application Laid-Open No. 2000-76421 Japanese Patent Application Laid-Open No. 2017-201499 JP 2018-112831 A JP 2011-154665 A JP 2012-8949 A Japanese Patent Application Laid-Open No. 2013-300 JP 2011-186521 A WO15/174426 publication

All of the above technologies are merely secondary functions in situations where communication in the real world is the primary mode of communication. In other words, they were not created in response to the recent digital transformation (DX) of work and the global pandemic, in which communication for work, classes, etc. is primarily conducted online.

The present invention aims to objectively evaluate communication in situations where online communication is the norm, such as meetings and lectures, in order to ensure more efficient communication.

According to the present invention,
A video analysis system that analyzes reactions of a user based on video images obtained by photographing the user in an environment where an online session is held by a plurality of users, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
an analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit,
the plurality of users include at least a patient who is a target of receiving counseling using the online session and a counselor;
a storage unit that stores, as a counseling record, information on questions posed by the counselor to the target patient, information on the target patient's answers to the questions, and information on the target patient's biological reactions at the time of the answers, in association with each other;
and a comparison unit that compares the counseling records in chronological order to evaluate the effect of the counseling on the target patient.

According to the present disclosure, by analyzing and evaluating the video footage of a video session, evaluation can be made objectively, particularly with regard to content.

In particular, the present invention makes it possible to objectively evaluate communication in order to communicate more efficiently in situations where online communication is the norm.

1 is a diagram showing an overall system according to an embodiment of the present invention; FIG. 2 is a functional block diagram of an evaluation terminal according to an embodiment of the present invention; FIG. 2 is a diagram illustrating a first example of a functional configuration of an evaluation terminal according to an embodiment of the present invention. FIG. 10 is a diagram illustrating a second example of a functional configuration of the evaluation terminal according to the embodiment of the present invention. FIG. 10 is a diagram illustrating a third example of a functional configuration of an evaluation terminal according to an embodiment of the present invention. FIG. 10 is a diagram illustrating another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. FIG. 10 is a diagram illustrating another configuration of functional configuration example 3 of the evaluation terminal according to the embodiment of the present invention. FIG. 1 is a functional block diagram of a system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an image of how a system is used in an embodiment of the present invention. FIG. 1 is a diagram illustrating an image of how a system is used in an embodiment of the present invention. FIG. 1 is a diagram illustrating an image of how a system is used in an embodiment of the present invention.

The present disclosure will be described below with reference to the following embodiments.
[Item 1]
A video analysis system that analyzes reactions of a user based on video images obtained by photographing the user in an environment where an online session is held by a plurality of users, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
an analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit,
the plurality of users include at least a patient who is a target of receiving counseling using the online session and a counselor;
a storage unit that stores, as a counseling record, information on questions posed by the counselor to the target patient, information on the target patient's answers to the questions, and information on the target patient's biological reactions at the time of the answers, in association with each other;
a comparison unit that compares the counseling records in chronological order and evaluates the effectiveness of the counseling on the target patient.
[Item 2]
Item 1, a video analysis system according to item 1,
The comparison unit compares, in time series, the biological responses of the target patient when the target patient gives answers that can be evaluated as the same to the same questions asked by the counselor multiple times.
Video image analysis system.
[Item 3]
Item 1 or Item 2, a video analysis system according to item 1 or 2,
an emotion evaluation unit that evaluates a level of emotion of the user in accordance with evaluation criteria that are standardized among a plurality of users based on the change in the biological reaction of the user analyzed by the analysis unit;
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the user experiencing the same emotion.
Video image analysis system.
[Item 4]
Item 3. The video analysis system according to any one of items 1 to 3,
a uniqueness determination unit that determines whether the change in the biological reaction analyzed for the user in one online session is unique compared to the change in the biological reaction analyzed for the user in an online session temporally earlier than the one online session;
and a clustering unit that clusters the change patterns of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specific determination unit and the magnitude of the change from the past.
Video image analysis system.
[Item 5]
Item 4. The video analysis system according to any one of items 1 to 4,
an emotion evaluation unit that evaluates the degree of emotion of the subject in accordance with evaluation criteria that are standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit;
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the subject experiencing the same emotion.
Video image analysis system.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and drawings, components having substantially the same functional configuration will be assigned the same reference numerals, and redundant explanations will be omitted.

<Basic functions>
The video session evaluation system of this embodiment is a system that analyzes and evaluates the unique emotions (e.g., feelings of pleasure or discomfort in response to one's own or another's words or actions, or the degree of such feelings) of a subject of analysis among multiple people in an environment where the multiple people are engaged in a video session (hereinafter, both one-way and two-way sessions are referred to as online sessions) that are different from those of the other people. The online session may be, for example, an online conference, online class, or online chat. Terminals installed in multiple locations are connected to a server via a communication network such as the Internet, allowing video images to be exchanged between the multiple terminals through the server. The video images handled in the online session include facial images and audio of the users using the terminals. The video images also include images of documents shared and viewed by multiple users. It is possible to switch between facial images and document images on the screen of each terminal, displaying only one of them, or to divide the display area and display both facial images and document images simultaneously. It is also possible to display an image of one of the multiple people in full screen, or to split the images of some or all of the users into smaller screens and display them. It is possible to designate one or more of the multiple users participating in an online session using terminals as analysis targets. For example, the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the "organizer") designates one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a coaching session. The organizer of the online session is typically one of the multiple users participating in the online session, but may also be a different person who does not participate in the online session. Note that it is also possible to designate no analysis target and to use all participants as analysis targets. It is also possible for the leader, facilitator, or administrator of the online session (hereinafter collectively referred to as the "organizer") to designate one of the users as the analysis target. The organizer of the online session is, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a coaching session. The organizer of the online session is typically one of the multiple users participating in the online session, but may also be a different person who does not participate in the online session.

In the video session evaluation system according to this embodiment, when a video session is established between multiple terminals, at least a video image acquired from the video session is displayed. The displayed video image is acquired by the terminal, and at least facial images contained in the video image are identified for each predetermined frame. An evaluation value for the identified facial image is then calculated. The evaluation value is shared as needed. In particular, in this embodiment, the acquired video image is stored on the terminal, analyzed and evaluated on the terminal, and the results are provided to the user of the terminal. Therefore, even if a video session contains personal information or confidential information, for example, the video itself can be analyzed and evaluated without providing the video itself to an external evaluation agency, etc. Furthermore, if necessary, the evaluation results (evaluation values) can be provided to an external terminal, allowing the results to be visualized and cross-analysis, etc. to be performed.

As shown in Figure 1, the video session evaluation system of this embodiment includes user terminals 10 and 20 having at least an input unit such as a camera unit and a microphone unit, a display unit such as a display, and an output unit such as a speaker, a video session service terminal 30 that provides a two-way video session to the user terminals 10 and 20, and an evaluation terminal 40 that performs part of the evaluation of the video session.

<Hardware configuration example>
Each functional block, functional unit, and functional module described below can be configured using, for example, hardware, a DSP (Digital Signal Processor), or software provided in a computer. For example, when configured using software, the computer is actually configured with a CPU, RAM, ROM, and the like, and is realized by running a program stored in a recording medium such as RAM, ROM, a hard disk, or a semiconductor memory. The series of processes performed by the system and terminal described herein can be realized using software, hardware, or a combination of software and hardware. A computer program for implementing each function of the information sharing support device 10 according to this embodiment can be created and installed on a PC or the like. A computer-readable recording medium on which such a computer program is stored can also be provided. Examples of such recording media include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The computer program may also be distributed, for example, via a network, without using a recording medium.

The evaluation terminal in this embodiment acquires moving images from a video session service terminal, identifies at least facial images contained in the moving images for each specified frame unit, and calculates an evaluation value for the facial images (details will be described later).

<How to obtain the video>
As shown in FIG. 2, the video session service provided by the video session service terminal (hereinafter, sometimes simply referred to as "this service") enables two-way image and audio communication between user terminals 10 and 20. This service displays video captured by the camera of the other user terminal on the display of the user terminal, and can output audio captured by the microphone of the other user terminal from the speaker. This service is also configured to enable both or either user terminal to record video and audio (collectively referred to as "video, etc.") in the memory of at least one of the user terminals. The recorded video information Vs (hereinafter referred to as "recorded information") is cached on the user terminal that initiated the recording and is recorded only locally on one of the user terminals. If necessary, users can view the recorded information themselves or share it with others within the scope of their use of this service.

<Functional configuration example 1>
Fig. 3 is a block diagram showing an example of the configuration according to this embodiment. As shown in Fig. 4, the video session evaluation system of this embodiment is realized as a functional configuration possessed by a user terminal 10. That is, the user terminal 10 has, as its functions, a video image acquisition unit 11, a biological reaction analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16.

The video acquisition unit 11 acquires video from each terminal, obtained by capturing images of multiple people (multiple users) using a camera equipped on each terminal during an online session. The video acquired from each terminal may or may not be configured to be displayed on the screen of each terminal. In other words, the video acquisition unit 11 acquires video from each terminal, including video that is currently being displayed and video that is not currently being displayed on each terminal.

The biological response analysis unit 12 analyzes changes in biological responses for each of multiple people based on the video images acquired by the video image acquisition unit 11 (whether or not they are currently being displayed on a screen). In this embodiment, the biological response analysis unit 12 separates the video images acquired by the video image acquisition unit 11 into an image set (a collection of frame images) and audio, and analyzes changes in biological responses from each. For example, the biological response analysis unit 12 analyzes changes in biological responses related to at least one of facial expression, eye contact, pulse rate, and facial movement by analyzing the user's facial image using the frame images separated from the video images acquired by the video image acquisition unit 11. The biological response analysis unit 12 also analyzes changes in biological responses related to at least one of the user's speech content and voice quality by analyzing the audio separated from the video images acquired by the video image acquisition unit 11.

When a person's emotions change, this is reflected in changes in biological reactions such as facial expressions, eye movements, pulse rate, facial movements, speech content, and voice quality. In this embodiment, changes in the user's emotions are analyzed by analyzing changes in the user's biological reactions. In this embodiment, the emotion analyzed is, for example, the degree of comfort/discomfort. In this embodiment, the biological reaction analysis unit 12 quantifies changes in biological reactions according to predetermined criteria, thereby calculating a biological reaction index value that reflects the content of the changes in biological reactions.

Analysis of facial expression changes is performed, for example, as follows: For each frame image, a facial region is identified within the frame image, and the identified facial expressions are classified into multiple categories according to an image analysis model that has been trained in advance by machine learning. Based on the classification results, it is then analyzed whether a positive or negative facial expression change has occurred between consecutive frame images, and the extent of the change, and a facial expression change index value corresponding to the analysis results is output.

Analysis of changes in eye direction is performed, for example, as follows. That is, for each frame image, the eye area is identified within the frame image, and the direction of both eyes is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the speaker currently being displayed, at the shared material currently being displayed, or looking off-screen. It is also possible to analyze whether the eye movements are large or small, and whether the movements are frequent or infrequent. Eye direction changes are also related to the user's level of concentration. The biological response analysis unit 12 outputs an eye direction change index value according to the results of the analysis of eye direction changes.

Analysis of pulse rate changes is performed, for example, as follows. That is, for each frame image, the facial area is identified within the frame image. Then, using a trained image analysis model that captures the numerical values of facial color information (G in RGB), changes in the G color of the facial surface are analyzed. The results are arranged along a time axis to form a waveform representing changes in color information, and the pulse rate is identified from this waveform. When a person is nervous, their pulse rate increases, and when they feel calm, their pulse rate slows down. The biological response analysis unit 12 outputs a pulse rate change index value according to the analysis results of pulse rate changes.

Analysis of changes in facial movement is performed, for example, as follows. That is, for each frame image, a facial area is identified within the frame image, and the facial direction is analyzed to analyze where the user is looking. For example, it is analyzed whether the user is looking at the face of the currently displayed speaker, at the currently displayed shared material, or looking off-screen. It may also be analyzed whether the facial movements are large or small, or whether the movements are frequent or infrequent. It may also be analyzed by combining facial movements with eye movements. For example, it may be analyzed whether the user is looking straight at the currently displayed speaker's face, looking up or down, or looking at an angle. The biological response analysis unit 12 outputs a facial direction change index value according to the analysis results of changes in facial direction.

The analysis of speech content is performed, for example, as follows. That is, the biological response analysis unit 12 converts speech into a string of characters by performing known speech recognition processing on a specified period of speech (for example, approximately 30 to 150 seconds), and then performs morphological analysis on the string of characters to remove words unnecessary for expressing conversation, such as particles and articles. The remaining words are then vectorized, and an analysis is performed to determine whether a positive or negative emotional change has occurred, and the extent of the emotional change, and a speech content index value corresponding to the analysis results is output.

Voice quality analysis is performed, for example, as follows. That is, the biological response analysis unit 12 identifies the acoustic characteristics of the voice by performing known voice analysis processing on the voice for a specified period of time (for example, approximately 30 to 150 seconds). Then, based on the acoustic characteristics, it analyzes whether a positive or negative voice quality change has occurred, and the extent of the voice quality change, and outputs a voice quality change index value according to the analysis results.

The biological response analysis unit 12 calculates a biological response index value using at least one of the facial expression change index value, eye gaze change index value, pulse rate change index value, facial direction change index value, speech content index value, and voice quality change index value calculated as described above. For example, the biological response index value is calculated by weighting the facial expression change index value, eye gaze change index value, pulse rate change index value, facial direction change index value, speech content index value, and voice quality change index value.

The uniqueness determination unit 13 determines whether the changes in biological reactions analyzed for the subject of analysis are unique compared to changes in biological reactions analyzed for other people other than the subject of analysis. In this embodiment, the uniqueness determination unit 13 determines whether the changes in biological reactions analyzed for the subject of analysis are unique compared to other people based on the biological reaction index values calculated for each of multiple users by the biological reaction analysis unit 12.

For example, the uniqueness determination unit 13 calculates the variance of the biological reaction index values calculated for each of multiple people by the biological reaction analysis unit 12, and by comparing the biological reaction index value calculated for the person being analyzed with the variance, determines whether the changes in the biological reactions analyzed for the person being analyzed are unique compared to others.

There are three possible cases where changes in the biological reactions analyzed for the subject are unique compared to others. The first is when no particularly significant changes in biological reactions occur in others, but a relatively large change in biological reactions occurs in the subject. The second is when no particularly significant changes in biological reactions occur in the subject, but a relatively large change in biological reactions occurs in others. The third is when relatively large changes in biological reactions occur in both the subject and others, but the nature of the change differs between the subject and others.

The related event identification unit 14 identifies events occurring with respect to at least one of the subject, others, and the environment when a change in a biological reaction determined to be unique by the uniqueness determination unit 13 occurs. For example, the related event identification unit 14 identifies from video images the words and actions of the subject when a unique change in a biological reaction occurs in the subject. The related event identification unit 14 also identifies from video images the words and actions of others when a unique change in a biological reaction occurs in the subject. The related event identification unit 14 also identifies from video images the environment when a unique change in a biological reaction occurs in the subject. The environment may be, for example, shared documents displayed on the screen or something that appears in the background of the subject.

The clustering unit 15 analyzes the degree of correlation between a change in a biological reaction determined to be unique by the unique determination unit 13 (for example, one or more combinations of eye contact, pulse rate, facial movement, speech content, and voice quality) and an event occurring when the unique change in the biological reaction occurs (an event identified by the related event identification unit 14), and if the correlation is determined to be at a certain level or above, clusters the person or event being analyzed based on the results of the correlation analysis.

For example, if a specific change in a biological reaction corresponds to a negative emotional change and the event occurring when the specific change in the biological reaction occurs is also a negative event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or the event into one of multiple pre-segmented categories depending on the content of the event, the degree of negativity, the magnitude of correlation, etc.

Similarly, if a specific change in biological reaction corresponds to a positive emotional change and the event occurring when the specific change in biological reaction occurs is also a positive event, a correlation above a certain level is detected. The clustering unit 15 clusters the subject of analysis or event into one of multiple pre-segmented categories depending on the content of the event, the degree of positivity, the magnitude of correlation, etc.

The analysis result notification unit 16 notifies the person designating the subject of analysis (the subject of analysis or the organizer of the online session) of at least one of the changes in biological reactions determined to be specific by the specificity determination unit 13, the events identified by the related event identification unit 14, and the classifications clustered by the clustering unit 15.

For example, the analysis result notification unit 16 notifies the subject of analysis of the subject's own words and actions as an event occurring when a unique change in biological reaction occurs in the subject that is different from that of others (one of the three patterns described above; the same applies below). This allows the subject of analysis to understand that when he or she behaves in a certain way, he or she is feeling differently from others. At this time, the subject of analysis may also be notified of the unique changes in biological reaction identified for the subject of analysis. Furthermore, the subject of analysis may also be notified of changes in the biological reaction of others to be compared.

For example, if the emotions felt by others in response to words or actions made by the subject without any particular awareness and with normal emotions, or words or actions made by the subject with particular awareness and with a certain emotion, differ from the emotions felt by the subject himself at the time of the words or actions, the subject will be notified of his or her own words or actions at that time. This makes it possible to discover words or actions that are well-received or unwell by others, despite the subject's own awareness.

The analysis result notification unit 16 also notifies the organizer of the online session of events occurring when a unique change in biological response occurs in the analysis subject that is different from that of others, along with the unique change in biological response. This allows the organizer of the online session to know what events are influencing what emotional changes as a phenomenon unique to the designated analysis subject. It is then possible to take appropriate measures for the analysis subject based on the information obtained.

The analysis result notification unit 16 also notifies the organizer of the online session of events occurring when a unique change in the analysis subject's biological response occurs that is different from that of others, or of the clustering results for the analysis subject. This allows the organizer of the online session to understand the analysis subject's unique behavioral tendencies and predict possible future behaviors and conditions, depending on which category the specified analysis subject is clustered into. This then makes it possible to take appropriate measures for the analysis subject.

In the above embodiment, an example was described in which biological reaction index values are calculated by quantifying changes in biological reactions according to predetermined criteria, and whether or not the changes in biological reactions analyzed for the subject of analysis are unique compared to others is determined based on the biological reaction index values calculated for each of multiple people, but this example is not limiting. For example, the following may be used.

That is, the biological response analysis unit 12 analyzes the eye movements of each of multiple people and generates a heat map showing the direction of the eyes. The peculiar determination unit 13 compares the heat map generated by the biological response analysis unit 12 for the analysis subject with the heat maps generated for other people, and determines whether the changes in the biological responses analyzed for the analysis subject are unique compared to the changes in the biological responses analyzed for other people.

In this way, in this embodiment, the video footage of the video session is stored in the local storage of the user terminal 10, and the above-mentioned analysis is performed on the user terminal 10. Although this may depend on the machine specifications of the user terminal 10, it is possible to analyze the video footage without providing it to an external party.

<Functional configuration example 2>
As shown in FIG. 4 , the video session evaluation system of this embodiment may include, as its functional configuration, a video image acquisition unit 11, a biological response analysis unit 12, and a response information presentation unit 13a. The response information presentation unit 13a presents information indicating changes in biological responses analyzed by the biological response analysis unit 12a, including participants not displayed on the screen. For example, the response information presentation unit 13a presents information indicating changes in biological responses to a leader, facilitator, or manager of the online session (hereinafter collectively referred to as the organizer). The organizer of the online session may be, for example, a lecturer of an online class, a chairperson or facilitator of an online conference, or a coach of a session for coaching purposes. The organizer of the online session is typically one of multiple users participating in the online session, but may also be a different person who does not participate in the online session.

By doing this, the organizer of an online session can keep track of participants who are not displayed on the screen in an environment where an online session is being held with multiple people.

<Functional Configuration Example 3>
5 is a block diagram showing an example of a configuration according to this embodiment. As shown in FIG. 5, the video session evaluation system of this embodiment has a functional configuration in which functions similar to those of the first embodiment described above are assigned the same reference numerals, and descriptions thereof may be omitted. The system according to this embodiment includes a camera unit that acquires video of the video session, a microphone unit that acquires audio, an analysis unit that analyzes and evaluates the video, an object generation unit that generates display objects (described later) based on information obtained by evaluating the acquired video, and a display unit that displays both the video of the video session and the display object during execution of the video session.

As explained above, the analysis unit includes a video image acquisition unit 11, a biological response analysis unit 12, a peculiar determination unit 13, a related event identification unit 14, a clustering unit 15, and an analysis result notification unit 16. The functions of each element are as described above.

Based on the analysis results of the video acquired from the video session by the analysis unit, the object generation unit, if necessary, superimposes and displays an object representing the recognized face and information representing the analyzed and evaluated content on the video. When multiple faces appear in the video, the object may identify and display all of the faces. Furthermore, even if the camera function of the other party's device is disabled (i.e., the camera is disabled by software within the video session application, rather than by physically covering it), the object may be displayed in the area where the other party's face is located if the other party's face is recognized by the camera. This allows both parties to confirm that the other party is in front of the device, even if the camera function is turned off. In this case, for example, the video session application may hide the information acquired from the camera, while displaying an object corresponding to the face recognized by the analysis unit. Furthermore, the video information acquired from the video session and the information recognized by the analysis unit may be displayed on different display layers, with the layer related to the former information hidden. When there are areas for displaying multiple videos, the object may be displayed in all areas or only in some areas. For example, the object may be displayed only in the video on the guest side.

The embodiments of the invention described above in Basic Configuration Examples 1 to 3 may be realized as a single device, or may be realized by a plurality of devices (e.g., cloud servers) partially or entirely connected via a network. For example, the control unit 110 and storage 130 of each terminal 10 may be realized by different servers connected to each other via a network. That is, this system includes user terminals 10 and 20, a video session service terminal 30 that provides two-way video sessions to the user terminals 10 and 20, and an evaluation terminal 40 that evaluates the video sessions. The following variations and combinations of configurations are possible:
(1) All processing is done on the user's terminal. As shown in Figure 6, by performing processing by the analysis unit on the terminal where the video session is being conducted, it is possible to obtain analysis and evaluation results simultaneously (in real time) with the time the video session is being conducted (although a certain level of processing power is required).
(2) Processing in the User Terminal and the Evaluation Terminal As shown in Fig. 7, an analysis unit may be provided in an evaluation terminal connected via a network, etc. In this case, the video captured by the user terminal is shared with the evaluation terminal simultaneously with or after the video session, and after being analyzed and evaluated by the analysis unit in the evaluation terminal, information on the objects 50 and 100 is shared with the user terminal together with or separately from the video data (i.e., information including at least the analysis data) and displayed on the display unit.

The following system is realized using each of the configurations of functional configuration example 1 to functional configuration example 3 described above or a combination of them.

<Embodiment>

The video analysis system of this embodiment is a video analysis system that analyzes user reactions based on video images obtained by photographing users in an environment where online sessions are held by multiple users, regardless of whether the users are displayed on the screen during the online session.

As shown in FIG. 10, the system includes a video image acquisition unit that acquires video images of each of a plurality of users by photographing the user during an online session.
The system includes an analysis unit that analyzes changes in the user's biological reactions based on video images acquired by the video image acquisition unit. The multiple users include at least a target patient receiving counseling using an online session and a counselor. The system includes a memory unit that associates and stores, as a counseling record, information about questions posed by the counselor to the target patient, information about the target patient's answers to the questions, and information about the target patient's biological reactions at the time of the answers. The system includes a comparison unit that compares the counseling records chronologically to evaluate the effectiveness of counseling on the target patient.

As shown in Figures 11 to 13, the comparison unit compares, in chronological order, the biological responses of each patient when the counselor asks the same question multiple times and the patient responds in a way that can be evaluated as identical. That is, as shown in Figure 11, the response data to the questions asked by the counselor to the patient is associated with the changes in the biological responses at the time of the response and is stored.

As shown in Figure 13, even if a patient always answers the counselor's question, "How are you?" with the same answer, "I'm fine," the happiness score of the biological response may differ, as shown in Figure 12. By capturing such changes in biological response, it is possible to estimate whether the counseling treatment is progressing well.

The embodiments described above may be implemented in appropriate combinations. Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above effects.

10, 20 User terminal 30 Video session service terminal 40 Evaluation terminal

Claims (4)

A video analysis system that analyzes reactions of a user based on video images obtained by photographing the user in an environment where an online session is held by a plurality of users, regardless of whether the user is displayed on a screen during the online session, comprising:
a video acquisition unit that acquires, for each of the plurality of users, a video obtained by photographing the user during the online session;
an analysis unit that analyzes a change in a biological reaction of the user based on the moving image acquired by the moving image acquisition unit,
the plurality of users include at least a patient who is a target of receiving counseling using the online session and a counselor;
a storage unit that stores, as a counseling record, information on questions posed by the counselor to the target patient, information on the target patient's answers to the questions, and information on the target patient's biological reactions at the time of the answers, in association with each other;
and a comparison unit that, when the target patient gives answers that can be evaluated as the same to each of the same questions asked by the counselor multiple times, calculates index values corresponding to changes in the biological reaction at the time of each of the answers based on the counseling record, compares the index values in chronological order , and evaluates the effectiveness of the counseling on the target patient. The video analysis system according to claim 1 ,
an emotion evaluation unit that evaluates a level of emotion of the user in accordance with evaluation criteria that are standardized among a plurality of users based on the change in the biological reaction of the user analyzed by the analysis unit;
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the user experiencing the same emotion.
Video image analysis system. 3. The video analysis system according to claim 1 ,
a uniqueness determination unit that determines whether the change in the biological reaction analyzed for the user in one online session is unique compared to the change in the biological reaction analyzed for the user in an online session temporally earlier than the one online session;
and a clustering unit that clusters the change patterns of the biological reaction based on the content of the change in the biological reaction determined to be specific by the specificity determination unit and the magnitude of the change from the past.
Video image analysis system. 4. The video analysis system according to claim 1,
an emotion evaluation unit that evaluates the degree of emotion of the subject in accordance with evaluation criteria that are standardized among a plurality of subjects based on the change in the biological reaction analyzed by the analysis unit;
the emotion evaluation unit evaluates the degree of emotion based on the magnitude of difference between a current biological response and a normal biological response, and the degree of emotion is adjusted according to the likelihood of the subject experiencing the same emotion.
Video image analysis system.