JP7589438B2 - Information processing device, information processing system, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing system, an information processing method, and an information processing program.

Systems that support learning and the acquisition of work skills by learners are known as prior art. For example, Patent Document 1 describes a technology that monitors a learner's rereading frequency and eye fixation, compares the results with those of other learners, and presents the comparison results to the learner and instructor.

JP 2005-338173 A (published on December 8, 2005)

In recent years, more and more people have been reviewing videos of learning or work in various situations to help improve said learning or work. It is preferable for such technology to be able to automatically identify important scenes from the recorded videos that are useful for improving learning or work.

In addition to learning and work, there is a great need for technology that can identify and present important information from sequential information.

In view of the above problems, the present invention aims to provide a technology that can appropriately identify and present important information from among information that has sequence characteristics.

To solve the above problem, an information processing device according to one aspect of the present invention includes a sequence information acquisition unit that acquires annotated sequence information in which annotations are added to each element included in sequence information related to a subject, an importance discrimination unit that refers to the annotated sequence information related to the subject and discriminates the importance of each element included in the annotated sequence information of the subject, a discrimination integration unit that refers to the discrimination results by the importance discrimination unit related to multiple subjects and generates discrimination integration information, and a presentation information generation unit that generates presentation information by referring to at least one of the discrimination results by the importance discrimination unit and the discrimination integration information.

The above configuration makes it possible to appropriately identify and present important information from among information that has a sequence.

In the information processing device according to the above aspect, the importance determination unit may update the determination logic by referring to feedback information from a user.

According to the above configuration, the judgment logic in the importance judgment unit is updated by referring to feedback information from the user, so that the importance judgment of the series information can be made more accurately each time the importance judgment is repeated.

In the information processing device according to the above aspect, the sequence information may include information regarding the subject's line of sight.

With the above configuration, the importance can be determined more appropriately by referring to information about the subject's line of sight.

In the information processing device according to the above aspect, the discrimination integration unit may extract common information from the discrimination results by the importance discrimination unit for each of the multiple subjects, and include the common information in the discrimination integration information.

With the above configuration, even if the series information differs depending on the subject, common information is extracted, so the importance can be determined more appropriately.

In the information processing device according to the above aspect, the importance determination unit may determine the degree of concentration of the subject as the importance.

With the above configuration, the importance of series information can be determined by referring to the concentration level of the subject, so the importance can be determined more appropriately.

In order to solve the above problem, a terminal device according to one aspect of the present invention includes a sequence information acquisition unit that acquires sequence information, an annotation assignment unit that generates annotated sequence information by assigning annotations to each element included in the acquired sequence information, a presentation information acquisition unit that acquires presentation information generated by referring to the annotated sequence information, and a presentation unit that presents the presentation information.

The above configuration allows for the acquisition of presentation information that integrates each series of information and presents it to the user.

The terminal device according to the above aspect may further include a discrimination integration information acquisition unit that acquires discrimination integration information generated by referring to discrimination results regarding annotated series information of multiple subjects, and the annotation assignment unit may update the annotation assignment logic by referring to at least one of feedback information from a user and the discrimination integration information.

According to the above configuration, annotation can be more efficiently performed by updating the annotation logic with reference to each piece of information.

In the terminal device according to the above aspect, the annotation assignment unit may assign, in addition to the annotation, information indicating the reliability of the annotation to each of the elements.

The above configuration allows each process that references annotations to be executed more efficiently.

In order to solve the above problem, an information processing system according to one aspect of the present invention includes an acquisition unit that acquires sequence information, an annotation assignment unit that generates annotated sequence information by assigning annotations to each element included in the acquired sequence information, an importance discrimination unit that refers to the annotated sequence information about a certain subject and determines the importance of each element included in the annotated sequence information about the subject, a discrimination integration unit that refers to the discrimination results by the importance discrimination unit about multiple subjects and generates discrimination integration information, and a presentation information generation unit that generates presentation information by referring to at least one of the discrimination results by the importance discrimination unit and the discrimination integration information.

The above configuration provides the same effects as the information processing device according to one aspect of the present invention.

In order to solve the above problem, an information processing method according to one aspect of the present invention includes an annotation adding step of generating annotated series information by adding annotations to each element included in series information related to a subject, an importance determination step of referring to the annotated series information and determining the importance of each element included in the annotated series information of the subject, a discrimination and integration step of generating discrimination and integration information by referring to the discrimination results in the importance determination step related to multiple subjects, and a presentation information generating step of generating presentation information by referring to at least one of the discrimination results in the importance determination step and the discrimination and integration information.

The above configuration provides the same effects as the information processing device according to one aspect of the present invention.

An information processing program according to one aspect of the present invention is an information processing program for causing a computer to function as any of the information processing devices described above, and causes the computer to function as the sequence information acquisition unit, importance discrimination unit, discrimination integration unit, and presentation information generation unit.

The above configuration provides the same effects as the information processing device according to one aspect of the present invention.

According to one aspect of the present invention, important information from information that has a sequence can be appropriately identified and presented.

1 is a block diagram illustrating a schematic configuration example of an information processing system according to an embodiment of the present invention. FIG. 2 is a sequence diagram illustrating an example of the operation of an information processing system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a test question to be recorded according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of presentation information according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of presentation information according to an embodiment of the present invention. 1 is a diagram showing an example of a visual target of a subject while learning a customer service task in one embodiment of the present invention; FIG. 1 is a table showing an example of data on a visual target and speech of a learner according to an embodiment of the present invention. 11 is a table showing an example of data of a veteran attendant according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of difference information between a learner and an experienced customer service representative according to an embodiment of the present invention. 11 is a table showing an example of data on a visual target and an action of a learner according to an embodiment of the present invention. 1 is a table illustrating an example of experienced assembler data according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of difference information between a learner and an experienced assembler according to an embodiment of the present invention. 11 is a table showing an example of data on a visual target and an action of a learner according to an embodiment of the present invention. 1 is a table showing an example of data on an experienced chef according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of difference information between a learner and an experienced chef in one embodiment of the present invention. 11 is a table showing an example of data on visual recognition targets and actions of Player A according to one embodiment of the present invention. 13 is a table showing an example of data of player B according to one embodiment of the present invention. FIG. 11 is a diagram showing an example of difference information between player A and player B according to one embodiment of the present invention. FIG. 11 is a block diagram illustrating a schematic configuration example of an information processing system according to another embodiment of the present invention. FIG. 11 is a sequence diagram illustrating an example of the operation of an information processing system according to another embodiment of the present invention. FIG. 11 is a block diagram illustrating a schematic configuration example of an information processing system according to still another embodiment of the present invention. FIG. 11 is a sequence diagram illustrating an example of the operation of an information processing system according to still another embodiment of the present invention.

Below, an embodiment of one aspect of the present invention (hereinafter also referred to as "this embodiment") will be described with reference to the drawings.

<Embodiment 1>
§1. Application Example First, an example of a situation in which the present invention is applied will be described with reference to Fig. 1. Note that, here, an example will be given in which the subject of the present invention is a learner who takes a lecture or a mock exam at a cram school or the like, but this does not limit the present embodiment, and the subject is not particularly limited as long as he or she has some kind of relevance to sequence information.

1 is a block diagram illustrating a schematic configuration example of an information processing system 1 according to an embodiment of the present invention. The information processing system 1 is an information processing system used, for example, in a cram school or the like. As shown in FIG. 1, the information processing system 1 according to this embodiment includes a server 100, learner terminal devices 200A, 200B, 200C, and instructor terminal device 1000, which are information processing devices. The server 100, learner terminal devices 200A, 200B, 200C, and instructor terminal device 1000 are connected via a network and can communicate with each other. The type of network may be any type, such as the Internet, a telephone network, or a dedicated network. In the following, the learner terminal devices 200A, 200B, and 200C may be collectively referred to as the learner terminal device 200.

The learner terminal devices 200A, 200B, and 200C according to this embodiment are, for example, computers assigned to each learner (hereinafter also referred to as the subject) A, B, and C in a cram school. Each learner terminal device 200A, 200B, and 200C is equipped with a camera 241, a microphone 242, and the like, and can record the appearance and speech of each subject during a lecture or a mock test. This recorded video is an example of sequence information in this embodiment. Here, sequence information means information in general in which the order of each element contained in the information is meaningful. In addition, sequence information may include video data, audio data, text data, and data showing changes in numerical values over time. In addition, each terminal device 200 can generate annotated sequence information by adding annotations to the sequence information and record it in the respective storage units 230. Here, annotations are meta information added to each element included in the sequence information. As an example, the annotation includes meta-information regarding the subject's state at each time point or each period during the recording of the series information. Note that the type of annotation is not limiting to this embodiment, but as an example, an index indicating the subject's concentration level at that time point can be used as the annotation.

Then, each of the learner's terminal devices 200A, 200B, and 200C transmits this annotated series information for each subject to the server 100.

The server 100 determines the importance of each element included in the series information based on the annotated series information transmitted from each terminal device 200. The server 100 then integrates the results of the importance determination based on the annotated series information transmitted from each terminal device 200 to generate presentation information. The server 100 stores the generated presentation information in the storage unit 130 and transmits it to each terminal device 200 or 1000.

Each terminal device 200 or 1000 presents the presentation information transmitted from the server 100 and transmits feedback information to the server 100. At this time, the subject may check the presentation information together with the instructor, and transmit feedback information regarding updates to the importance determination logic or the determination logic used in the determination integration process described below to the server 100 via each terminal device 200 or 1000. In other words, feedback information as a result of correcting and confirming the presentation information may be transmitted to the server 100 from each terminal device 200 or 1000. The server 100 updates the importance determination logic based on the feedback information transmitted from each terminal device 200 or 1000.

As described above, the amount of information that can be used for determining the importance level and the like increases each time transmission and reception of series information is repeated between the server 100 and each of the terminal devices 200 or 1000, and the importance level determination logic and the determination logic used in the determination and integration process are improved. As a result, the importance level and the like of series information can be determined more appropriately. This allows the instructor to review the video by focusing on information that is likely to lead to improvements in the content of instruction, and thus allows the instructor to use time efficiently to improve instruction.
§2. Configuration Example The configurations of the server 100, learner terminal device 200, and instructor terminal device 1000, which are information processing devices constituting an information processing system 1 in this embodiment, will be described in detail below with reference to FIG.

<Student's Terminal Device 200>
The learner's terminal devices 200A, 200B, and 200C according to the embodiment have the same configuration, for example. For example, the learner's terminal device 200 includes a control unit 210, a communication unit 220, a storage unit 230, a camera 241, a microphone 242, an operation reception unit 243, a display unit (presentation unit) 244, and a speaker 245.

The communication unit 220 performs communication processing with an external device such as the server 100. The memory unit 230 is a storage device that stores various data. The operation reception unit 243 is an interface that receives input operations from the subject, and is, for example, a button such as a keyboard. The display unit 244 is a display panel that displays moving images. The operation reception unit 243 and the display unit 244 may be realized as a touch panel that receives input operations from the subject and displays moving images. The control unit 210 is a control device that controls the entire learner terminal device 200, and includes a series information acquisition unit 212 that acquires series information via the camera 241, an annotation assignment unit 214 that generates annotated series information by assigning annotations to each element included in the acquired series information, a presentation information acquisition unit 216 that acquires presentation information generated by referring to the annotated series information via the communication unit 220, and a discrimination integration information acquisition unit (not shown) that acquires discrimination integration information generated by referring to the discrimination results regarding the annotated series information of one or more subjects.

<Series information acquisition unit 212>
The sequence information acquisition unit 212 acquires sequence information related to the subject. Examples of the sequence information acquired by the sequence information acquisition unit 212 are as follows. These data are, for example, obtained during a mock exam or a lecture. However, this does not limit the present embodiment, and the information may be acquired in other situations.
A subject captured by the camera 241 (which may include the subject's face)
The subject's voice collected by the microphone 242 and the subject's operation input accepted by the operation acceptance unit 243 (which may include text data)
Image, audio, text, and other data acquired via the communication unit 220 Image, audio, text, and other data read from the storage unit 230 Changes in temperature over time acquired from a temperature sensor (not shown) and pulse data obtained from a sensor (not shown) for reading the subject's level of tension, stress, concentration, etc.

The series information may also include information regarding the subject's line of sight. For example, the series information acquisition unit 212 acquires facial information of the subject from an image acquired from the camera 241. The facial information includes positional information indicating the position of each part of the subject's face (e.g., eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, and size information indicating the size, as well as the subject's line of sight as a measure of the subject's state. The line of sight is particularly important as an indicator of the subject's degree of concentration on a task, etc. Detection of the subject's line of sight will be described later.

<Annotation Assignment Unit 214>
The annotation assigning unit 214 generates annotated series information by automatically assigning annotations to each element included in the acquired series information or by assigning annotations based on a user instruction. Annotations are assigned to elements included in the series information. As one example, annotation A is assigned to the section from 00:01 to 00:02 in the video data, and annotation B is assigned to the section from 00:05 to 00:06. As another example, annotation X is assigned to phrase AA in sentence A included in the text data, and annotation Y is assigned to phrase BB.

The annotation assigning unit 214 may also assign information indicating whether or not the subject has viewed a specific area as an annotation to the series information. The operation receiving unit 243 also accepts an operation to assign an annotation to the series information, and an operation to input the importance (described below) for each part of the series information. For example, when a user issues an instruction via the operation receiving unit 243 to assign an annotation to a certain part in the series information, the annotation assigning unit 214 assigns the instructed annotation to that part of the series information.

As a more specific example, the annotation assignment unit 214 evaluates the subject's level of concentration at each point in the recorded video on a five-point scale, and assigns the level of concentration as an annotation to each point in the video, which is series information. As another example, from a video of the subject during a mock test, the problem statement or question that the subject is looking at at each point in time is identified, and information indicating which problem statement or question the subject is looking at is assigned as an annotation to each point in the video.

In this way, it is possible to annotate information indicating which problem statement or question is being viewed by, for example, identifying the coordinates indicating the subject's line of sight as coordinates on the mock test question paper.

The annotation assignment process by the annotation assignment unit 214 is not limited to the above example, and may be configured to, for example, determine points to which annotations should be assigned using other image analysis algorithms or audio analysis algorithms, and assign annotations to the determined locations.

As another example, an annotation may be defined by a start time, an end time, a tag, and a reliability. Here, the tag is information indicating the type of annotation, such as information indicating the concentration level and understanding level of the subject. Furthermore, the reliability is information indicating the likelihood of the target annotation. The annotation assignment unit 214 can calculate the reliability of each annotation assigned by the annotation assignment process, which indicates the degree to which the annotation is reliable. Furthermore, the annotation assignment unit 214 may assign information indicating the reliability of the annotation to each element in the series information in addition to the annotation.

In addition, for example, the concentration level may be set to a relatively high value when the variation in the subject's visual target is small (when there is little variation in the gaze), and set to a relatively low value when the variation in the subject's visual target is large (when there is large variation in the gaze). The annotation assignment unit 214 can identify the subject's concentration level by acquiring and referring to the gaze information of the subject.

Also, for example, the level of understanding may be set to a high value when the time that the subject views the object to be understood is relatively long (or short).

For example, when the sequence information is information that records the object visually recognized by the subject during the mock test, the annotation unit 214 may set the level of understanding by referring to the mock test score input by the subject after the mock test via the operation reception unit 243, or information indicating which question was answered correctly. For example, when the subject satisfies a predetermined condition of performing a specific action at a specific timing, the annotation unit 214 may set a high level of understanding for the target part in the sequence information. The information indicating the above-mentioned predetermined condition may be stored in advance in the memory unit 230, etc.

The annotation assignment unit 214 may also update the determination logic (annotation assignment logic) used when assigning annotations to series information, for example, by referring to information included in the presentation information.

When the annotation assignment unit 214 assigns annotations based on user input, the annotation assignment unit 214 may be configured to update the annotation assignment logic by using the annotations assigned by the user as feedback information. The annotation assignment unit 214 may also update the annotation assignment logic by referring to the discrimination integration information. That is, the annotation assignment unit 214 may update the annotation assignment logic by referring to at least one of the feedback information from the user and the discrimination integration information.

<Presentation information acquisition unit 216, display unit 244>
The presentation information acquisition unit 216 acquires the presentation information generated by the presentation information generation unit 118 of the server 100 with reference to the annotated series information.

The display unit 244 displays the presentation information acquired by the presentation information acquisition unit 216. Note that the presentation information is not limited to a moving image presented via the display unit 244, but may be audio presented via the speaker 245. The learner's terminal device 200 may also be configured to accept feedback information from the user via the operation acceptance unit 243, similar to the instructor's terminal device 1000 described below.

More specifically, as an example, the subject and the instructor check the presentation information displayed by the display unit 244 and, while having a dialogue, check the importance determination result included in the presentation information. Then, if there is any point that needs to be corrected regarding the importance,
Information including which part should be corrected and how it should be corrected is input to the operation reception unit 243. Here, the point to be corrected in terms of importance is, for example, an importance determination result corresponding to certain presentation information indicating a relatively high importance, but the presentation information is determined by the subject to be unimportant. The operation reception unit 243 may be configured to generate feedback information including the input information and transmit it to the server 100 via the communication unit 220.

<Server 100>
1, the server 100 of the information processing system 1 includes a control unit 110, a communication unit 120, and a storage unit 130. The communication unit 120 enables communication with other information processing devices (the learner terminal device 200, the instructor terminal device 1000) included in the information processing system 1. The storage unit 130 can store information transmitted from the other information processing devices, information integrated by the server, and the like.

The control unit 110 is a control device that controls the entire server 100, and includes a series information acquisition unit 112, an importance discrimination unit 114, a discrimination integration unit 116, and a presentation information generation unit 118. The communication unit 120 performs communication processing with external devices such as the learner's terminal device 200. The memory unit 130 is a memory device that stores various data.

<Series information acquisition unit 112>
The series information acquiring unit 112 acquires annotated series information in which annotations are added to each element included in the series information on the learner. The information in which annotations have been added to the series information for each subject is acquired via the communication unit 120.

<Importance determination unit 114>
The importance determining unit 114 refers to the annotated series information about the subject, and determines the importance of each element included in the annotated series information about the subject.

The importance determination unit 114 may determine the importance based on the concentration level of the subject. As an example, when the concentration level is equal to or greater than a predetermined value, or the concentration level is equal to or less than a predetermined value for a predetermined period of time, the importance determination unit 114 determines that the section has a non-zero importance.

For example, it is assumed that the server 100 acquires annotated sequence information such as that shown in the table below from each terminal device 200.
Time 00:01 00:02 00:03 00:04 00:05
Concentration level target A 1 5 5 3 1
Subject B 1 5 5 5 1
Subject C 2 5 5 5 5
In this case, as an example, the importance determination unit 114 determines a section in which a concentration level of 5 continues for 2 seconds or more as a section having an importance level other than 0. For example, since subject A maintains a state of concentration level 5 for 2 seconds from 00:02 to 00:03, the importance determination unit 114 determines that the importance of the section is 1. Furthermore, since subject B maintains a state of concentration level 5 for 3 seconds from 00:02 to 00:04, the importance determination unit 114 determines that the importance of the section is 2. Furthermore, since subject C maintains a state of concentration level 5 for 4 seconds from 00:02 to 00:05, the importance determination unit 114 determines that the importance of the section is 3.

Similarly, the importance determination unit 114 can be configured to determine a section in which a concentration level of 1 or 2 continues for 2 seconds or more as a section having an importance level other than 0.

In addition, for example, the importance determination unit 114 may determine that the lower (or higher) the subject's level of understanding of a given part is, the higher the importance of the part in question is. Alternatively, the importance determination unit 114 may determine that the importance of a part where the subject's gaze begins to disperse (or where the gaze begins to converge) during a lecture or the like is high.

As another example, the importance determination unit 114 may be configured to perform a regression analysis on the annotated series information, and determine the importance by referring to various parameters obtained by performing the regression analysis. For example, regression analysis may be applied to annotated series information related to a certain subject, and parameters indicating a regression curve showing the annotated series information may be derived. The value of the parameter may then be compared with the value of a predetermined parameter, and the importance determination may be performed according to the comparison result.

The importance discrimination unit 114 also updates the discrimination logic used in the importance discrimination process by referring to feedback information acquired via the communication unit 120. As an example, when the importance discrimination unit 114 acquires feedback information indicating that the importance of an element to which the importance discrimination unit 114 has assigned an importance level of 5 should be lowered, the discrimination logic is updated so that a lower importance level is assigned to the element to which the importance discrimination unit 114 has assigned an importance level of 5.

The importance determination unit 114 may determine that the more the gaze or movement of the subject during a certain period of time deviates from those of other subjects, the higher the importance. The importance determination unit 114 may determine that the more the gaze or movement of the subject during a certain period of time deviates from the gaze or movement set as the norm, the higher the importance. The information indicating the normative gaze or movement described above may be stored in advance in the storage unit 130, etc.

The importance determination unit 114 may also perform importance determination processing by determining that a point that contributes highly to improving the determination logic when the annotation assignment unit 214 assigns annotations to series information is of high importance.

<Discrimination and integration unit 116>
The discrimination integration unit 116 generates discrimination integration information by referring to the discrimination results by the importance discrimination unit 114 for a plurality of subjects A, B, C, .... For example, the discrimination integration unit 116 collects the sequence information obtained from each terminal device 200 and integrates the sequence information, thereby making it possible to more appropriately determine the importance of the sequence information.

As an example, the discrimination integration unit 116 may extract common information from the discrimination results by the importance discrimination unit 114 for each of the multiple subjects A, B, and C, and include the common information in the discrimination integration information.

For example, it is assumed that the importance determining unit 114 determines the importance of subjects A, B, and C as follows.
Time 00:01 00:02 00:03 00:04 00:05
Importance Target A 0 1 1 0 0
Subject B 0 2 2 2 0
Subject C 0 3 3 3 3
In this case, as an example, the discrimination integration unit 116 determines that 00:02, which is a time assigned a non-zero importance level, is important to subjects A, B, and C, and includes the determination result in the discrimination integration information.

For example, the discrimination integration unit 116 may include in the discrimination integration information information indicating that the gaze or behavior of a subject over a certain period of time deviates from those of other subjects, and information indicating the degree of deviation. The discrimination integration unit 116 may also include in the discrimination integration information information indicating that the gaze or behavior of a subject over a certain period of time deviates from gaze or behavior set as a norm, and information indicating the degree of deviation.

For example, the discrimination integration unit 116 may include meta-information indicating matters related to the subject, such as the subject's past performance records, or meta-information indicating matters related to the subject's environment, such as mock exams that the subject is taking, in the discrimination integration information. Each of the above-mentioned meta-information may be included in the discrimination result by the importance discrimination unit 114.

As another example, when the importance discrimination unit 114 is configured to perform a regression analysis on the annotated sequence information, the discrimination integration unit 116 may be configured to extract parameters having common properties from the parameters obtained by the regression analysis on the subjects A, B, and C, and include the parameters in the discrimination integration information. The discrimination integration unit 116 may generate a regression model to be commonly used by referring to regression models corresponding to multiple subjects. In other words, the discrimination integration unit 116 may take multiple discrimination algorithms or parameters referenced by them as input data, and output the integrated discrimination algorithm or parameters referenced by them. When the importance can be input to the sequence information by the subjects or instructors, the discrimination integration unit 116 can generate discrimination integration information that reflects, for example, the points that multiple instructors consider important.

The discrimination integration unit 116 also updates the discrimination logic used in the discrimination integration process that generates the discrimination integration process by referring to feedback information acquired via the communication unit 120. As an example, when the discrimination integration unit 116 acquires feedback information indicating that the importance of an element assigned an importance level of 5 should be lowered, the discrimination logic is updated so that a lower importance level is assigned to the element assigned an importance level of 5.

The discrimination integration unit 116 also generates information for updating the importance discrimination logic or the updated discrimination logic, and supplies it to the importance discrimination unit 114. The importance discrimination unit 114 updates the importance discrimination logic by referring to the acquired information. This makes it possible to update the results of the importance discrimination process to more suitable ones, for example, by using series information corresponding to multiple subjects.

The discrimination integration unit 116 also generates information for updating the discrimination logic used to assign annotations, or the updated discrimination logic, and transmits it to the learner's terminal device 200 via the communication unit 120. The annotation assignment unit 214 updates the discrimination logic by referring to the acquired information. This makes it possible to update the annotation assignment process to a more suitable one, for example, by using series information corresponding to multiple subjects. Note that the discrimination integration unit 116 may refer to the discrimination integration information it has generated when updating each of the above-mentioned discrimination logics.

Note that the annotation assignment logic by the annotation assignment unit 214, the importance determination logic by the importance determination unit 114, and the integration logic by the discrimination integration unit 116 are not limited to the above examples. Furthermore, the processing by the annotation assignment unit 214, the importance determination unit 114, and the discrimination integration unit 116 may use rule-based logic, machine learning such as a neural network, or other methods.

For example, any one or a combination of the following machine learning techniques can be used:

・Support Vector Machine (SVM)
・Clustering
・Inductive Logic Programming (ILP)
・Genetic Algorithm (GP: Genetic Programming)
・Bayesian Network (BN)
・Neural Network (NN)
When using a neural network, input data may be preprocessed for input to the neural network. For such processing, in addition to one-dimensional or multidimensional arrangement of data, a method such as data augmentation can be used.

When using a neural network, a convolutional neural network (CNN) including convolution processing may be used. More specifically, a convolutional layer that performs convolutional calculations may be provided as one or more layers included in the neural network, and a filter calculation (product-sum calculation) may be performed on input data input to the layer. When performing the filter calculation, processing such as padding may be used in combination, or an appropriately set stride width may be adopted.

In addition, a multi-layer or ultra-multi-layer neural network with tens to thousands of layers may be used as the neural network. The above-mentioned machine learning may be supervised learning or unsupervised learning.

<Presentation information generation unit 118>
The presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 114 and the discrimination integration information. For example, The information processed for each target person is transmitted to each terminal device 200. The generated integrated information and presentation information are stored in the storage unit 130.

Also, for example, the presentation information generating unit 118 may prioritize inclusion of corresponding information in the presentation information, the higher the importance indicated by the determination result by the importance determining unit 114. Also, the presentation information generating unit 118 may prioritize inclusion of corresponding information in the presentation information, the higher the degree to which the information included in the discrimination integration information indicates that the gaze or behavior of the subject during a certain period of time differs from the gaze or behavior of other subjects. Also, the presentation information generating unit 118 may prioritize inclusion of corresponding information in the presentation information, the higher the degree to which the information included in the discrimination integration information indicates that the gaze or behavior of the subject during a certain period of time differs from the gaze or behavior set as a norm.

<Instructor Terminal Device 1000>
The instructor terminal device 1000 includes a control unit 1010 , an operation receiving unit 1043 , a display unit 1044 , a speaker 1045 , a communication unit 1020 , and a storage unit 1030 .

The control unit 1010 is a control device that controls the entire instructor terminal device 1000, and includes a presentation information acquisition unit 1014 and a feedback information acquisition unit 1012.

The communication unit 1020 performs communication processing with external devices such as the server 100. The memory unit 1030 is a storage device that stores various data. The operation reception unit 1043 is an interface that receives input operations from a lecturer or the like, and is, for example, buttons on a keyboard or the like. The display unit 1044 is a display panel that displays moving images. Note that the operation reception unit 1043 and the display unit 1044 may be configured to be realized as a touch panel that receives input operations from a lecturer or the like and displays moving images.

<Presentation information acquisition unit 1014, display unit 1044>
The presentation information acquisition unit 1014 acquires the presentation information generated by the presentation information generation unit 118 of the server 100 with reference to the annotated series information.

The display unit 1044 displays the presentation information acquired by the presentation information acquisition unit 1014.

<Feedback Information Acquisition Unit 1012>
The feedback information acquisition unit 1012 acquires feedback information from the user. As an example, the instructor checks the presentation information displayed by the display unit 1044 and checks the importance determination result included in the presentation information. If there is any point that needs to be corrected regarding the importance,
The user inputs information including which part should be corrected and how it should be corrected into the operation reception unit 1043. Here, the part to be corrected in terms of importance is, for example, an importance determination result corresponding to a certain piece of presentation information that indicates a relatively high importance, but the instructor has determined that the piece of presentation information is not important. The operation reception unit 1043 generates feedback information including the input information and transmits it to the server 100 via the communication unit 1020.

The importance determination unit 114 of the server 100 refers to feedback information from the user and updates the determination logic.

The feedback information acquisition unit 1012 may be provided in the instructor terminal device 1000 or in the learner terminal device 200.

<Gaze detection>
The following describes how the series information acquisition unit 212 and the annotation assignment unit 214 detect the gaze of a subject.

First, the camera 241 captures the subject during the mock test. The series information acquisition unit 212 acquires facial information of the subject from the video captured by the camera 241. The facial information of the subject includes, for example, position information indicating the position of each part of the face (for example, eyes, nose, mouth, eyebrows, etc.), shape information indicating the shape, and size information indicating the size. In particular, the eye information can be used to evaluate the subject's concentration on the object on which the subject is gazing. Examples of eye information include the corners and corners of the eye, and edges such as the iris and pupil. The series information acquisition unit 212 may appropriately perform correction processing such as noise reduction and edge emphasis on the video acquired from the camera 241. The series information acquisition unit 212 transmits the extracted facial information to the annotation assignment unit 214.

The annotation assignment unit 214 detects the state of the subject based on the face information extracted by the sequence information acquisition unit 212. For example, the state of each part of the subject's face is detected, and at least one of the following is detected: the gaze of the subject, the state of the pupils, the number of blinks, the movement of the eyebrows, the movement of the cheeks, the movement of the eyelids, the movement of the lips, and the movement of the jaw.

The method of detecting the subject's gaze is not particularly limited, but an example is a method in which a point light source (not shown) is provided in the terminal device 200, and the corneal reflection image of light from the point light source is captured by the camera 241 for a predetermined period of time to detect the direction of the subject's gaze. The type of point light source is not particularly limited, and examples include visible light and infrared light, but by using an infrared LED, for example, it is possible to detect the gaze without causing discomfort to the subject. In gaze detection, if the gaze does not move for more than a predetermined period of time, it can be said that the subject is staring at the same place.

Methods for detecting the state of the pupils include, but are not limited to, a method of detecting circular pupils from an image of the eye using a Hough transform. Generally, humans tend to dilate their pupils when concentrating, so the subject's level of concentration can be evaluated by detecting the size of the pupils. For example, the size of the pupils can be detected for a specified period of time, and if the pupils are enlarged within that period of time, it is highly likely that the subject is gazing at an object. A threshold value can be set for pupil size, and pupil size equal to or greater than the threshold can be evaluated as "open," and pupil size less than the threshold can be evaluated as "closed."

Methods for detecting the number of blinks include, but are not limited to, a method of irradiating the subject's eyes with infrared light and detecting the difference in the amount of infrared light reflected when the eyes are open and when they are closed. Generally, when a person is concentrating, they tend to blink at a low frequency and at stable intervals, so by detecting the number of blinks, the subject's level of concentration can be evaluated. For example, if the number of blinks is detected for a specified period of time and blinks are occurring at stable intervals within the specified period, it can be said that there is a high possibility that the subject is gazing at a certain object.

The annotation assignment unit 214 only needs to detect at least one of the subject's gaze, pupil state and blink rate, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement, and chin movement, but it is preferable to combine these. By combining detection methods in this way, the annotation assignment unit 214 can appropriately evaluate the subject's concentration level when viewing a certain object.

In addition to the state of the eyes, examples of the state of each part of the face include eyebrow movement, such as lifting the inner or outer eyebrow, eyelid movement, such as lifting the upper eyelid or tensing the eyelid, nose movement, such as wrinkling the nose, lip movement, such as lifting the upper lip or pursing the lips, cheek movement, such as lifting the cheek, and jaw movement, such as lowering the chin. The state of the subject may be a combination of the states of multiple parts of the face.

In addition to gaze information, as described above, the subject's concentration level can be determined more appropriately by referring to the detection results of pupil state, blink frequency, eyebrow movement, eyelid movement, cheek movement, nose movement, lip movement, and chin movement. The subject's concentration level may also be determined by further referring to the subject's pulse data.

The annotation assigning unit 214 evaluates the concentration level of the identified subject on a five-level scale, for example, from 1 to 5, and assigns the result as an annotation to the subject's series information.
§3. Operation example <Operation of information processing system 1>
Next, a description will be given of the operation of the information processing system 1. Fig. 2 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to the present embodiment. With reference to Fig. 2, a process of generating presentation information in the information processing system 1 according to the present embodiment will be described.

(Step S102)
In step S102, the annotation unit 214 generates annotated series information by annotating each element included in the series information about the subject acquired by the series information acquisition unit 212. The communication unit 220 transmits the generated annotated series information to the server 100.

(Step S104)
In step S104, the series information acquisition unit 112 of the server 100 acquires the series information of the annotated series information.

(Step S106)
In step S106, the importance determining unit 114 refers to the annotated series information related to the subject, and determines the importance of each element included in the annotated series information of the subject.

(Step S108)
In step S108, the discrimination integration unit 116 generates discrimination integration information by referring to the results of the importance discrimination for the multiple subjects A, B, C, . . .

(Step S110)
In step S110, the presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 114 and the discrimination integration information. When the presentation information generating unit 118 generates presentation information by referring only to the discrimination result by the importance discriminating unit 114, the process of step S108 does not necessarily have to be executed, and the process of step S110 may be executed following the process of step S106. Thereafter, the communication unit 120 transmits the generated presentation information to each learner's terminal device 200.

(Step S112)
In step S112, the presentation information acquisition section 216 of each learner's terminal device 200 acquires the generated presentation information.

(Step S114)
In step S114, the presentation unit (display unit) 244 presents the acquired presentation information.

In addition, in this step S114, the annotation assignment unit 214 may refer to the information contained in the presentation information to update the determination logic used when assigning annotations to the series information.

(Step S116)
In step S116, the terminal device 200 (or the instructor terminal device 1000) acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The communication unit 220 of the terminal device 200 transmits the feedback information to the server 100.

(Step S118)
In step S118, the importance discrimination unit 114 or the discrimination integration unit 116 of the server 100 updates each discrimination logic with reference to feedback information from the learner's terminal device 200. For example, the importance discrimination unit 114 updates the importance discrimination logic based on feedback information from the terminal device 200 including the confirmed and corrected importance.

(Step S120)
When the series information acquirer 212 of the terminal device 200 acquires series information again, the annotation assigner 214 generates annotated series information by assigning annotations to the series information. The subsequent steps S120 to S136 are the same as the above steps S102 to S118.

In the information processing system 1, the learner's terminal device 200 repeats the above steps S102 to S118 each time it acquires sequence information.

Therefore, the server 100 updates each discrimination logic each time it obtains series information from the subject. As the amount of series information from the subject increases, each discrimination logic is improved, and each discrimination can be performed more appropriately.

<Another example 1 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.>
The annotation assignment process, the importance assignment process, the presentation information generation process, and the like performed by the information processing system 1 according to the present embodiment are not limited to the above examples. Other examples of the annotation assignment process, the importance assignment process, the presentation information generation process, and the like performed by the information processing system 1 according to the present embodiment will be described below.

In this example, the sequence information is information that records the subject's visual target during the mock test. FIG. 3 is a diagram showing an example of a test question that is the subject of recording. In FIG. 3, areas R1 to R3 are areas that are defined in advance to distinguish each range. Marker 301 is a marker that identifies the coordinates indicating the subject's line of sight as coordinates on the mock test question paper.

The annotation assignment unit 214 in this example assigns information to the sequence information indicating which part of the test question the subject was looking at while the sequence information was being recorded. For example, the annotation assignment unit 214 assigns information to the sequence information indicating that the subject was looking at area R1 at time 00:05.

Figure 4 is a diagram showing an example of information to be presented. Also, Figure 4 (A) is a diagram showing, in a time series, a model way of moving the gaze when solving a target mock test. Figure 4 (B) is a diagram showing, in a time series, the way of moving the gaze of a subject who solved the mock test. Figure 4 (C) is a diagram showing the importance at each time point assigned to the series information by the importance determination unit 114. In each diagram in Figure 4, the horizontal axis indicates time. Also, the vertical axis in Figure 4 (C) indicates the level of importance.

For example, in FIG. 4(C), point 303 indicates a point to which the importance discrimination unit 114 set a high importance because the variation in the subject's gaze was greater than the normative gaze when the subject read through the entire test question. Also, point 304 indicates a point to which the importance discrimination unit 114 set a high importance because the subject's degree of understanding was determined to be low because the subject's visual recognition of area R1, which is the key to the answer, was low when the subject considered his or her answer.

The importance discrimination unit 114 may set a high importance to areas where the difference between the information shown in FIG. 4(A) and the information shown in FIG. 4(B) is large or small, or may set a high importance to a corresponding area in the series information if the subject has a high rate of visual recognition of an area that is the key to the answer. Also, FIG. 4(C) may be configured to include information indicating the importance set in the series information corresponding to other users as the integrated discrimination information generated by the discrimination integration unit 116 so that the user can compare it.

FIG. 5 is a diagram showing an example of presentation information displayed on the display unit 244 of the learner's terminal device 200 (or the display unit 1044 of the instructor's terminal device 1000).
In FIG. 5, objects 311, 312, and 313 correspond to those in FIG. 4(A), FIG. 4(B), and FIG. 4(C), respectively.

Screen 315 is a playback screen for a video recording of the subject's visual target. Also, marker 301 does not have to be displayed on this screen.

Object 317 and object 314 in seek bar 316 indicate which part of the video recording the visual target of the subject is being played on screen 315. The part of the video being played may be changeable by sliding object 316 or object 317 left or right. Object 318 is an operation panel for controlling the playback of the video.

The location where the video is played may be changeable by selecting a portion of the text on screen 315. In other words, when a portion of the text is selected, the video may be configured to start playing from the location where the subject was viewing the portion of the text.

The display unit 244 may also display a pie chart or the like showing the proportion of each part of the test question that the subject looked at and to what extent on the screen exemplified in FIG. 5. The same applies to the pie chart or the like regarding the normative gaze described above.

The operation reception unit 243 also receives input of comments and the like corresponding to each part of the video. The input comments and the like are transmitted as feedback information to the server 100 via the communication unit 220.

<Another example 2 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.>
Other examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Note that the same description as that already described in the above examples will not be repeated.

The series information in this example is information that records the visual target of a subject who is learning customer service work in a restaurant or the like. Also, FIG. 6 is a diagram showing an example of the visual target of a subject who is learning customer service work. The image shown in FIG. 6 is the state inside a restaurant as seen from the perspective of a learner who is a store clerk, and the image shows customers, the learner's hands, other store clerks, tables, etc.

Figure 7 is a table showing an example of data regarding the learner's visual target, speech (content of speech and tone of voice), movement and concentration level (blinking, pupil state, gaze fixation time and facial expression), and is a table showing an example of annotations added to the series information at each time point by the annotation adding unit 214 in this example.

As shown in FIG. 7, the annotation assignment unit 214, as an example, assigns the visual target, speech, action, and concentration level as annotations at each point in time. An event is a series of decisions (speech and action, etc.) made within a specified time from the moment when a learner or an experienced customer recognizes an object. In one example, an event includes timing, visual target, speech, action, and concentration level, and each event can be identified by an ID.

In addition to the state of the eyes, other examples of the state of each part of the face include eyebrow movement, such as lifting the inner or outer eyebrow, eyelid movement, such as lifting the upper eyelid or tensing the eyelid, nose movement, such as wrinkling the nose, lip movement, such as lifting the upper lip or pursing the lips, cheek movement, such as lifting the cheek, and jaw movement, such as lowering the chin. The states of multiple parts of the face may be combined as the state of the learner.

As a method of judging a learner's level of concentration from facial expressions, for example, if the eyebrows move upwards, it can be determined that the learner is concentrating because he or she is paying more attention to the visual target. Also, for example, if the cheeks move upwards while looking at a person's face, it can be determined that the learner is concentrating on the visual target because they are making a facial expression directed at the other person.

Figure 8 is a table showing an example of data of a veteran customer service representative that is pre-stored in the memory unit 130. As an example of the data of a veteran customer service representative, Figure 8 shows the timing at which the object was viewed, the object being viewed, the content of the speech, the tone of voice, the actions, and the level of concentration.

Figure 9 also shows information on the "deviation" between the learner's data shown in Figure 7 and the veteran customer service data shown in Figure 8 (quantitative information shown by the value of (learner - veteran customer) using "+/-" or "+/-" in the table), or information on whether the data matches the veteran customer service data (qualitative information shown by "○/×" in the table). As shown in Figure 9, when the learner's data and the veteran customer service data do not match, the data may include the veteran customer service data ("correct answer" in the table).

The importance determination unit 114 determines the importance of each element included in the annotated series information of the target person by referring to the information shown in Figures 7 to 9. For example, the information corresponding to event ID "4" in Figure 9 indicates that, although this is a point where a veteran customer service representative uttered "I will confirm your order," the target person did not utter such an utterance. The importance determination unit 114 may set a higher importance for such points in the series information where there is a large difference in the actions of the veteran customer service representative and the target person than for other points.

For example, the information corresponding to event ID "3" indicates that there is a difference of 7 seconds between the experienced customer and the target person in the timing of performing the action corresponding to the information. The importance determination unit 114 may set a higher importance to such a portion of the series information where there is a large difference in the timing of the action between the experienced customer and the target person than to other portions. For example, the importance determination unit 114 may set a higher importance to a portion where the target person's concentration level is lower than a predetermined threshold than to other portions, even if the difference in concentration level from the experienced customer is relatively small.

The configuration of the playback screen of the presentation information illustrated in Fig. 5 can also be applied to this example. The presentation information generating unit 118 may generate the presentation information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the behavior between the experienced customer service representative and the target person at the target timing and the content of the difference.
<Another example 3 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Note that the same description as that already described in the above examples will not be repeated.

In this example, the sequence information is, for example, video data capturing the actions of assemblers in factory automation (FA), image data captured by wearable cameras attached to assemblers, assembly sounds, and information recording visual targets. Note that the assemblers may be multiple assemblers who assemble the same product, or assemblers who assemble the same product on different lines.

Figure 10 is a table showing an example of data regarding the subject's visual target, actions, concentration level (blinks, gaze fixation time, and pupil state), and assembly sounds, and is a table showing an example of annotations added to series information at each time point by the annotation adding unit 214 in this example. As shown in Figure 10, as an example, the annotation adding unit 214 adds the visual target, speech, actions, concentration level, and assembly sounds as annotations at each time point.

FIG. 11 is a table showing an example of data of an experienced assembler that is prestored in the storage unit 130. FIG. 11 shows, as an example of the experienced assembler's data, the timing of visual recognition of an object, the visual target, the action, the concentration level, and the sound of assembly. FIG. 12 shows information on the "deviation" between the subject's data shown in FIG. 10 and the experienced assembler's data shown in FIG. 11 (quantitative information shown by the value of (subject - experienced assembler) using "+/-" or "+/-" in the table), or information on whether the data matches the experienced assembler's data (qualitative information shown by "○/×" in the table). As shown in FIG. 12, when the subject's data does not match the experienced assembler's data, the experienced assembler's data ("correct answer" in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the subject's annotated series information by referring to the information shown in Figures 10 to 12. For example, the information corresponding to event ID "3" in Figure 10 indicates that the experienced assembler is holding the end of part B with his left hand, while the subject is holding the center of part B with his right hand. The importance determination unit 114 may set a higher importance for such parts in the series information where the difference between the actions of the experienced assembler and the subject is large than for other parts.

For example, the information corresponding to event ID "7" is a process of assembling part B to part A, and indicates that there is an error of 18 seconds between the experienced assembler and the subject. This indicates that the time required to perform the same assembly operation differs by 18 seconds between the experienced assembler and the subject. For a portion of the series information where the difference in time required to perform the same operation is large, the importance determination unit 114 may set a higher importance than other portions. For a portion where the subject's concentration level is lower than a predetermined threshold, the importance determination unit 114 may set a higher importance than other portions, even if the difference in concentration level from that of the experienced assembler is relatively small. Alternatively, the importance of each event may be specified by the assembler.

The presentation information generation unit may integrate the results of the importance determination and generate presentation information such as the following:

- Presentation information may be generated for the target person, showing areas that need improvement and attention in each process. Based on the presentation information, a video work manual for new assemblers may be created that highlights or plays in slow motion difficult or error-prone processes and other high-importance processes.

- To allow managers to manage assemblers, it may be possible to generate information showing time series changes in each assembler's performance and areas where multiple assemblers make mistakes.

-Furthermore, to improve each process, suggested information may be generated that presents improvements to the process itself, such as changing the order of the processes, separating a process into multiple processes, or integrating multiple processes, for the areas where most assemblers make mistakes.

The configuration of the playback screen of the presentable information illustrated in Fig. 5 can also be applied to this example. The presentable information generating unit 118 may generate the presentable information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the action between the experienced assembler and the subject at the target timing and the content of the difference.
<Another example 4 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

In this example, the series information is, for example, video data capturing the actions of a chef in a cooking situation, image data captured by a wearable camera attached to the chef, cooking sounds, and information recording visual targets. The chefs may be multiple chefs who cook the same menu, each of whom may be a chef who wants to learn how to cook on their own at home, or may be employees who cook in the kitchens of restaurants or supermarkets.

FIG. 13 is a table showing an example of data regarding the subject's visual target, actions, level of concentration (blinks, gaze fixation time, and pupil state), and cooking sounds, and is a table showing an example of annotations added to series information at each time point by the annotation adding unit 214 in this example. As shown in FIG. 13, as an example, the annotation adding unit 214 adds the visual target, speech, actions, and level of concentration as annotations at each time point.

FIG. 14 is a table showing an example of data of an experienced chef prestored in the storage unit 130. FIG. 14 shows, as an example of the experienced chef's data, the timing of visual recognition of an object, the visual target, the action, the concentration level, and the cooking sound. FIG. 15 shows information on the "deviation" between the subject's data shown in FIG. 13 and the experienced chef's data shown in FIG. 14 (quantitative information shown by the value of (subject - experienced chef) using "+/-" or "+/-" in the table), or information on whether the data matches the experienced chef's data (qualitative information shown by "○/×" in the table). As shown in FIG. 15, when the subject's data does not match the experienced chef's data, the experienced chef's data ("correct answer" in the table) may be included.

The importance determination unit 114 determines the importance of each element included in the annotated series information of the subject by referring to the information shown in Figures 13 to 15. For example, the information corresponding to event ID "3" in Figure 13 indicates that, while an experienced chef holds a knife, he places his index finger on the back of the knife and grasps it with his middle finger and thumb, the subject grasps the knife with his thumb and index finger. The importance determination unit 114 may set a higher importance for such parts of the series information where the difference between the actions of the experienced chef and the subject is large than for other parts.

For example, the information corresponding to event ID "5" indicates that there is an 8-second difference between the experienced cook and the subject when cutting ingredients. This indicates that there is an 8-second difference in the time required to perform the same cooking action between the experienced cook and the subject. The importance determination unit 114 may set a higher importance to a portion of the series information where the difference in the time required to perform the same action is large than to other portions. For example, the importance determination unit 114 may set a higher importance to a portion where the subject's concentration level is lower than a predetermined threshold than to other portions, even if the difference in the concentration level from the experienced cook is relatively small. Alternatively, a high importance may be assigned to a difficult process or a process that affects taste or quality, depending on the cook.

The presentation information generation unit may integrate the results of the importance determination and present, for example, the following presentation information:

- Presentation information may be generated for the target user, showing areas that need improvement and attention in each cooking stage. Based on the presentation information, a cooking manual may be created using videos that focus on difficult steps, steps that are easy to make mistakes in, and other steps that are of high importance, as well as slow-motion playback.

- If the subject is an employee who cooks in the kitchen of a restaurant or supermarket, presentation information may be generated that shows time-series changes in each chef's performance and areas where multiple chefs make mistakes, in order to enable the manager to manage the chefs.

-Furthermore, to improve each process, presentation information may be generated that presents improvements to the process itself, such as changing the order of the processes, separating a process into multiple processes, or integrating multiple processes, for areas where most chefs make mistakes.

The configuration of the playback screen of the presentable information illustrated in Fig. 5 can also be applied to this example. The presentable information generating unit 118 may generate the presentable information so that the screen 315 in Fig. 5 includes text or the like indicating that there was a difference in the action between the experienced assembler and the subject at the target timing and the content of the difference.
<Another example 5 of annotation assignment processing, importance assignment processing, presentation information generation processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

The subjects in this example include players in a sport. The annotations may be provided by a coach, by the annotation providing unit 214, or a combination thereof.

Also, one coach may provide annotations to multiple players,
A plurality of coaches may provide annotations to one player, and further, a plurality of coaches may provide annotations to a plurality of players.

Alternatively, the system may be configured to include multiple annotation assigning units 214 that have been trained separately, and multiple annotation assigning units may assign annotations to one player. Furthermore, multiple annotation assigning units may assign annotations to multiple players.

In this example, the series information is, for example, image, audio, and body sensing data of the player. The player may be a player of any sport, but here, a baseball player (batter) will be used as an example. The player may also be a professional athlete, a student, or a working player.

Figure 16 is a table showing an example of data on the visual target, actions, and concentration level (blinks, gaze fixation time, and pupil state) of Player A (a player with a low batting average), and is a table showing an example of annotations added to the series information at each time point by the annotation adding unit 214 in this example. As an example, the annotation adding unit 214 adds the visual target, speech, actions, and concentration level as annotations at each time point.

Figure 17 also shows an example of data for player B (a player with a high batting average), including the timing at which the object was viewed, the object being viewed, the action, the level of concentration, and the sound of the play.

Also, FIG. 18 shows information on the "deviation" between the data of player A shown in FIG. 16 and the data of player B shown in FIG. 17 (quantitative information shown by the value of (player A - player B) using "+/-" or "+/-" in the table), or information on whether the data matches player B's data (qualitative information shown by "○/×" in the table). As shown in FIG. 18, if the data of player A and the data of player B do not match, the data may include player B's data ("correct answer" in the table).

The importance determination unit 114 determines the importance of each element included in the annotated series information of each player by referring to the information shown in Figures 16 to 18. For example, the information corresponding to event ID "1" in Figure 16 indicates that player B holds the bat short, whereas player A holds it long. The importance determination unit 114 may set a higher importance for such parts of the series information where there is a large difference in the actions between players than for other parts.

For example, the information corresponding to event ID "3" indicates that player A's swing speed is 135 Km/hour, while player B's swing speed is 148 Km/hour, indicating an error of 13 Km/hour. The importance determination unit 114 may set a higher importance to a portion of the series information where there is a large difference in speed at which the same action is performed than to other portions. For example, the importance determination unit 114 may set a higher importance to a portion where the player's concentration level is lower than a predetermined threshold than to other portions. Alternatively, a high importance may be specified for actions that vary greatly depending on the player.

The presentation information generation unit may integrate the results of the importance determination and, for example, generate presentation information for each player that integrates information annotated by multiple coaches to present areas for improvement in each player's play and parts of the play that need improvement (image scenes).

In addition, information annotated by one or more coaches for multiple players may be integrated to generate presentation information for coaches and other instructors that suggests areas for improvement in instruction. Furthermore, based on the presentation information, a video instruction manual may be created that shows close-ups and slow-motion playback of important scenes for successful batting.

In the above example, we have taken the case where plays by individual players, such as batting, are input as sequence information, but sequence information may also be directed to plays and actions by multiple players, such as formations in soccer.

Furthermore, the input data may include information about spectators of the match. For example, the input data may include video data capturing images of the spectators, gaze information of the spectators, and sounds emitted by the spectators.

Based on this data about spectators, the spectators' level of attention and excitement for each scene during the game can be annotated.

The presentation information generation unit may generate presentation information for players and coaches that presents scenes that attracted a lot of attention from spectators.

Other examples of the annotation assignment process, the importance assignment process, and the presentation information generation process will be described below.
<Another Example 6 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

In this example, we take electronic sports (e-sports) as an example, in which characters operated by players existing in the real world compete against each other in a virtual space. Note that the players may be multiple players playing the same e-sport.

In this example, the series information includes, as an example, inputs by the player to a personal computer, game operation unit, etc. In addition, the series information in this example may include information regarding the movements of a character in a virtual space that is generated based on these inputs. The series information may also include image, audio, and body sensing data of the player. Furthermore, the series information in this example may include information regarding spectators of e-sports, as in the case of sports described above. It may also include information regarding the level of attention and excitement of the spectators.

The annotation assignment unit 214 refers to the series information and, as an example, assigns annotations to the character's movements, the spectator's level of attention, and the level of excitement at each point in time.

The importance determination unit 114 refers to the annotated series information of multiple players and determines the importance of each element included in this annotated series information. For example, the importance determination unit 114 may set a higher importance for a portion of the series information where there is a large difference in behavior between characters than for other portions.

For example, the importance determination unit 114 may set a higher importance to areas where the player's concentration level is lower than a predetermined threshold than to other areas. Alternatively, a higher importance may be assigned to actions that vary greatly depending on the player.

The presentation information generation unit may then generate presentation information for the player that presents scenes that attracted high attention from e-sports spectators.

Furthermore, the presentation information generation unit may integrate the results of the importance determination and generate presentation information for each player, which presents, for example, points in each player's play that need improvement and parts of the play that need improvement.
<Another Example 7 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

The series information in this example is information that records video data capturing the actions of office workers during office work, image data captured by a wearable camera attached to the office worker, work sounds, and visual targets. The series information may also include operation histories of terminal devices such as keyboards and mice, and application usage histories.

As an example, the annotation assignment unit 214 assigns, as annotations, information about the visual target, speech, movement, concentration level, work sounds, operation history of the terminal device, and application usage history at each point in time.

As an example, the importance determination unit 114 determines the importance of each element included in the annotated series information of the target office worker (subject) by referring to the annotated series information of the target office worker and the reference annotated series information. Here, the reference annotated series information may include annotated series information of other office workers, such as veterans.

For example, the importance determination unit 114 may set a higher importance level when the subject spends more time inputting text than other office workers. Also, the importance determination unit 114 may set a higher importance level when the subject spends more time working on spreadsheets than other office workers.

The presentation information generation unit may integrate the results of the importance determination and present, for example, the following presentation information:

- Presentation information may be generated for the target person to show areas that need improvement and attention in each work situation. Based on the presentation information, a work manual may be created using videos that show close-ups or slow-motion playback of difficult tasks, tasks that are prone to mistakes, and other tasks that are of high importance.

Furthermore, in order to improve the content of each task, for tasks that most office workers make mistakes on, presentation information may be generated that presents improvements that should be made to the task itself, such as changing the order of tasks, separating a task into multiple tasks, or integrating multiple tasks.
<Another Example 8 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

In this example, the series information includes, for example, video data capturing the actions of the telephone operator (subject), image data captured by a wearable camera attached to the telephone operator, audio data of the conversation with the other party on the phone, sounds of the work, and information that records the visual target. The series information may also include the operation history of terminal devices such as a keyboard and mouse.

As an example, the annotation assignment unit 214 assigns, as annotations, information regarding the subject's visual target, speech, movement, concentration level, work sounds, terminal device operation history, application usage history, etc. at each time point. The annotation assignment unit 214 may also estimate the subject's level of tension at each time point and assign an index indicating the level of tension as an annotation. Note that a publicly known algorithm can be used as the tension level estimation algorithm.

As an example, the importance determination unit 114 determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject and the reference annotated series information. Here, the reference annotated series information may include annotated series information of other telephone operators, such as veterans.

For example, the importance determination unit 114 may set a higher importance level when the target person's level of tension is higher than that of other telephone operators. Also, the importance determination unit 114 may set a higher importance level when the target person's level of tension toward a specific telephone caller is higher than that of other telephone callers.

The presentation information generation unit may integrate the results of the importance determination and present, for example, the following presentation information:

- Presentation information may be generated for the target user, showing areas that need improvement and points that require attention in each telephone response situation. Based on the presentation information, a work manual may be created using videos that provide close-ups or slow-motion playback of important tasks such as difficult telephone responses or telephone responses that are prone to mistakes.

Furthermore, presentation information may be generated for the administrator that indicates telephone response contents and callers that cause most telephone operators to feel nervous.
<Another Example 9 of Annotation Assignment Processing, Importance Assignment Processing, Presentation Information Generation Processing, etc.>
Further examples of the annotation assignment process, the importance assignment process, and the presentation information generation process performed by the information processing system 1 according to the present embodiment will be described below. Again, the same description of the matters already described in the above examples will not be repeated.

The sequence information in this example is, for example, video data capturing the movements of students (subjects) at a driving school, image data captured by a wearable camera attached to the subject, audio data of interactions with an instructor, sounds of driving movements, and information recording visual targets. The sequence information may also include sensing data of operating parts such as the steering wheel and brakes.

As an example, the annotation assignment unit 214 assigns, as annotations, information about the visual target, action, concentration level, speech, driving operation sounds, operation history of the operation unit, etc. at each time point. The annotation assignment unit 214 may also estimate the subject's level of tension at each time point and assign an index indicating the level of tension as an annotation. Note that a publicly known algorithm can be used as the tension level estimation algorithm.

As an example, the importance determination unit 114 determines the importance of each element included in the subject's annotated series information by referring to the subject's annotated series information and the reference annotated series information. Here, the reference annotated series information may include the annotated series information of an exemplary driver such as an instructor.

For example, the importance determination unit 114 may set a higher importance when the timing at which the subject starts to turn the steering wheel when turning a curve and the timing at which the exemplary driver starts to turn the steering wheel are greater than a predetermined value. Also, the importance determination unit 114 may set a higher importance when the degree of tension on a particular course is higher than the degree of tension on other courses.

The presentation information generation unit may integrate the results of the importance determination and present, for example, the following presentation information:

- Presentation information may be generated for the target person, presenting areas for improvement and points to pay attention to in each driving scene. Based on the presentation information, a driving manual may be created using videos that show close-ups or slow-motion playback of difficult driving scenes and other high-importance driving scenes.

-Furthermore, presentation information may be generated for instructors showing the courses and driving scenes that cause the most tension for most students.

<Embodiment 2>
The configuration of the information processing system in the present invention is not limited to the above. Fig. 10 is a block diagram showing a schematic configuration example of the information processing system 1 according to the second embodiment. The information processing system 1 of the second embodiment is different from the information processing system 1 of the first embodiment in that the importance determination unit 215 is not provided in the server 100 but in the learner's terminal device 200, but the other configurations are the same.

Next, the operation of the information processing system 1 according to the second embodiment will be described. FIG. 11 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to the second embodiment. The process of generating presentation information in the information processing system 1 according to the second embodiment will be described with reference to FIG. 11.

(Step S202)
In step S202, the annotation assigning unit 214 generates annotated series information by assigning annotations to each element included in the series information related to the subject acquired by the series information acquiring unit 212.

(Step S206)
In step S206, the importance determination unit 215 determines the importance of each element included in the annotated series information of the target person by referring to the annotated series information of the target person generated by the annotation assignment unit 214. The communication unit 220 of the learner's terminal device 200 transmits the importance determination result to the server 100.

(Step S207)
In step S207, the series information acquisition unit 112 of the server 100 receives the importance determination result.

(Step S208)
In step S208, the discrimination integration unit 116 of the server 100 generates discrimination integration information by referring to the results of the importance discrimination for the multiple subjects A, B, C, . . .

(Step S210)
In step S210, the presentation information generating unit 118 generates presentation information by referring to at least one of the discrimination result by the importance discriminating unit 215 and the discrimination integration information. When the presentation information generating unit 118 generates presentation information by referring only to the discrimination result by the importance discriminating unit 215, the process of step S208 does not necessarily have to be executed, and the process of step S210 may be executed following step S207. The communication unit 120 of the server 100 transmits the generated presentation information to each learner's terminal device 200.

(Step S212)
In step S212, the presentation information acquisition section 216 of each learner's terminal device 200 acquires the generated presentation information.

(Step S214)
In step S214, the presentation unit (display unit) 244 presents the acquired presentation information.

In addition, in this step S214, the annotation assignment unit 214 may refer to the information contained in the presentation information to update the determination logic used when assigning annotations to the series information.

(Step S216)
In step S216, the terminal device 200 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The feedback information is transmitted from the terminal device 200 to the server 100.

(Step S218)
In step S218, the importance discrimination unit 215 of the learner's terminal device 200 updates its discrimination logic based on the feedback information. For example, the importance discrimination unit 215 updates its importance discrimination logic based on the feedback information including the confirmed and corrected importance. The discrimination integration unit 116 of the server 100 also updates its discrimination integration logic based on the feedback information.

(Step S220)
When the lineage information acquirer 212 of the terminal device 200 acquires lineage information again, the annotation assigner 214 generates annotated lineage information by assigning annotations to the lineage information in step S220. The subsequent steps S220 to S236 are the same as the above steps S202 to S218.

In the information processing system 1 of embodiment 2, the learner's terminal device 200 repeats the above steps S202 to S218 each time it acquires sequence information.

Therefore, the importance discrimination unit 215 of the learner's terminal device 200 updates the importance discrimination logic each time series information is acquired from the subject. As the amount of series information from the subject increases, the importance discrimination logic improves and the importance can be discriminated more appropriately. The same applies to the discrimination logic used by the discrimination integration unit 116 in the discrimination integration process.

<Embodiment 3>
12 is a block diagram illustrating a schematic configuration example of an information processing system 1 according to embodiment 3. The information processing system according to embodiment 3 is different from the information processing system according to embodiment 1 in that the presentation information generating unit 1013 is not provided in the server 100 but in the instructor terminal device 1000, but the other configurations are the same.

Next, the operation of the information processing system 1 according to the third embodiment will be described. FIG. 13 is a sequence diagram illustrating an example of the operation of the information processing system 1 according to the third embodiment. The process of generating presentation information in the information processing system 1 according to the third embodiment will be described with reference to FIG. 13.

(Step S302)
In step S302, the annotation unit 214 generates annotated series information by annotating each element included in the series information about the subject acquired by the series information acquisition unit 212. The communication unit 220 transmits the generated annotated series information to the server 100.

(Step S304)
In step S304, the series information acquisition unit 112 of the server 100 acquires the annotated series information.

(Step S306)
In step 306, the importance determining unit 114 refers to the annotated series information related to the subject, and determines the importance of each element included in the annotated series information of the subject.

(Step S308)
In step S308, the discrimination integration unit 116 generates discrimination integration information by referring to the discrimination results of the importance levels for the multiple subjects A, B, C, . . . The generated discrimination integration information is transmitted to the instructor terminal device 1000.

(Step S310)
In step S310, the presentation information generating unit 1013 of the instructor terminal device 1000 generates presentation information by referring to at least one of the discrimination result by the importance discrimination unit 114 and the discrimination integration information. When the presentation information generating unit 1013 generates presentation information by referring only to the discrimination result by the importance discrimination unit 114, the process of step S308 does not necessarily need to be executed, and the discrimination result in step S306 may be transmitted to the instructor terminal device 1000, followed by the process of step S310. The generated presentation information is transmitted to each learner terminal device 200.

(Step S312)
In step S312, in each learner's terminal device 200, the presentation information acquisition section 216 acquires the generated presentation information.

(Step S314)
In step S314, the presentation unit (display unit) 244 presents the acquired presentation information.

In addition, in this step S314, the annotation assignment unit 214 may refer to the information contained in the presentation information to update the determination logic used when assigning annotations to the series information.

(Step S316)
In step S316, the instructor terminal device 1000 acquires feedback information. As described above, the feedback information is generated as a result of the subject confirming the presentation information together with the instructor and confirming and correcting the determined importance. The feedback information is transmitted from the instructor terminal device 1000 to the server 100.

(Step S318)
In step S318, the importance discrimination unit 114 or the discrimination integration unit 116 of the server 100 updates each discrimination logic based on feedback information from the instructor terminal device 1000. For example, the importance discrimination unit 114 updates the importance discrimination logic based on feedback information from the instructor terminal device 1000 including the confirmed and corrected importance.

(Step S320)
When the series information acquirer 212 of the terminal device 200 acquires series information again, the annotation assigner 214 generates annotated series information by assigning annotations to the series information. The following steps S320 to S336 are the same as the above steps S302 to S318.

In the information processing system 1, the learner's terminal device 200 repeats the above steps S302 to S318 each time it acquires sequence information.

Therefore, the server 100 updates each discrimination logic each time it obtains series information from the subject. As the amount of series information from the subject increases, each discrimination logic is improved, and each discrimination can be performed more appropriately.

[Software implementation example]
The control block of the information processing device 100 (particularly, the series information acquisition unit 112, the importance discrimination unit 114, the discrimination integration unit 116, and the presentation information generation unit 118) and each terminal device 200 (particularly, the series information acquisition unit 212, the annotation assignment unit 214, and the presentation information acquisition unit 216) may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the information processing device 100 and each terminal device 200 are equipped with a computer that executes instructions of a program, which is software that realizes each function. This computer is equipped with, for example, one or more processors, and a computer-readable recording medium that stores the program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention. The processor can be, for example, a CPU (Central Processing Unit). The recording medium can be a "non-transient tangible medium", such as a ROM (Read Only Memory), tape, disk, card, semiconductor memory, programmable logic circuit, etc. In addition, a RAM (Random Access Memory) that expands the program may also be included. The program may be supplied to the computer via any transmission medium (such as a communication network or broadcast waves) that can transmit the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in different embodiments.

1 Information processing system 100 Information processing device 100 Server 110, 210, 1010 Control unit 112, 212 Series information acquisition unit 114, 215 Importance discrimination unit 116 Discrimination integration unit 118, 1013 Presentation information generation unit 120, 220, 1020 Communication unit 130, 230, 1030 Storage unit 200 Learner's terminal device 214 Annotation assignment unit 216, 1014 Presentation information acquisition unit 241 Camera 242 Microphone 243, 1043 Operation acceptance unit 244 Display unit (presentation unit)
1044 Display unit 245, 1045 Speaker 1000 Lecturer terminal device 1012 Feedback information acquisition unit

Claims (10)

a sequence information acquiring unit that acquires annotated sequence information in which each element included in sequence information related to a subject is annotated based on at least one of a speech sound and a work sound of the subject;
an importance determination unit that determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject;
a discrimination integration unit that generates discrimination integration information by referring to the discrimination results by the importance discrimination unit for a plurality of subjects;
a presentation information generating unit that generates presentation information by referring to at least one of a result of the determination by the importance determining unit and the determination integration information;
Equipped with
The information processing device according to claim 1, wherein the importance determining unit determines, as the importance, a degree of concentration regarding the subject based on movement of the subject's eyebrows, cheeks, nose, lips, or chin . 2 . The information processing apparatus according to claim 1 , wherein the importance determining unit updates a determination logic by referring to at least one of feedback information from a user and the determination integration information. 3. The information processing apparatus according to claim 1, wherein the series information includes information regarding the line of sight of the subject. 4. The information processing device according to claim 1, wherein the discrimination integration unit extracts common information from the discrimination results by the importance discrimination unit for each of the plurality of subjects, and includes the common information in the discrimination integration information. a sequence information acquisition unit for acquiring sequence information relating to a subject;
an annotation unit that generates annotated sequence information by adding annotations to each element included in the acquired sequence information in accordance with at least one of the speech of the subject and the work sound;
an importance determination unit that determines the importance of each element included in the annotated series information of the subject by referring to the annotated series information of the subject;
a presentation information acquisition unit that acquires presentation information generated by referring to the annotated series information and a result of the importance determination unit ;
A presentation unit that presents the presentation information ,
The importance determination unit is a terminal device that determines the degree of concentration regarding the subject as the importance based on the movement of the subject's eyebrows, cheeks, nose, lips, or chin . The method further includes a discrimination integration information acquisition unit that acquires discrimination integration information generated by referring to discrimination results regarding the annotated series information of a plurality of subjects,
The terminal device according to claim 5 , wherein the annotation providing unit updates an annotation providing logic by referring to at least one of feedback information from a user and the discrimination integration information. The terminal device according to claim 5 , wherein the annotation providing unit provides each of the elements with information indicating a reliability of the annotation in addition to the annotation. An acquisition unit that acquires sequence information regarding a subject;
an annotation unit that generates annotated sequence information by adding annotations to each element included in the acquired sequence information in accordance with at least one of the speech of the subject and the work sound;
an importance determining unit that determines the importance of each element included in the annotated series of a given subject by referring to the annotated series of the given subject;
a discrimination integration unit that generates discrimination integration information by referring to discrimination results by the importance discrimination unit for a plurality of subjects;
a presentation information generating unit that generates presentation information by referring to at least one of a result of the determination by the importance determining unit and the determination integration information;
Equipped with
The importance determination unit determines, as the importance, a degree of concentration regarding the subject based on movement of the subject's eyebrows, cheeks, nose, lips, or chin . 1. An information processing method performed by an apparatus, comprising:
an annotation step of generating annotated sequence information by adding annotations to each element included in the sequence information related to the subject in accordance with at least one of a speech sound and a work sound of the subject;
an importance determination step of determining the importance of each element included in the annotated series information of the subject by referring to the annotated series information;
a discrimination integration step of generating discrimination integration information by referring to the discrimination results in the importance discrimination step for a plurality of subjects;
a presentation information generating step of generating at least one of the discrimination result in the importance discrimination step and the discrimination integration information;
Including,
An information processing method characterized in that in the importance determination step, the importance is determined based on the degree of concentration regarding the subject, based on the movement of the subject's eyebrows, cheeks, nose, lips or chin . 5. A program for causing a computer to function as an information processing device according to claim 1, the information processing program causing a computer to function as the series information acquisition unit, the importance discrimination unit, the discrimination and integration unit, and the presentation information generation unit.

Images