JP7659322B2 - Information Extraction Device - Google Patents

Description

This disclosure relates to an information extraction device.

In recent years, distribution services that deliver various types of content have become increasingly popular.

Patent Document 1 discloses a technology that enables a process to detect desired music content by inputting a humming of the song to be searched for, even when the user does not know the title of the music content that he or she wishes to have delivered.

JP 2002-55994 A

However, the technology described in Patent Document 1 limits the content to be distributed to music content, and therefore requires a huge amount of computational processing by a computer to perform processing to detect search targets in all other video content.

Therefore, the present disclosure has been made in consideration of such circumstances, and has an object to provide an information extraction device that can reduce the load associated with calculation processing.

The main invention of the present invention for solving the above problem is characterized by including an extraction unit that extracts a specific group of frames from a moving image made up of a plurality of frames in accordance with predetermined specific conditions instructed from outside.

This disclosure can reduce the load associated with computational processing.

1 is a conceptual diagram illustrating a configuration example of an at-home individualized instruction system 1 according to a first embodiment of the present disclosure. 1 is a diagram illustrating an example of the hardware configuration of a computer that realizes a classroom video distribution device 10 according to a first embodiment of the present disclosure. 2 is a diagram illustrating an example of the hardware configuration of a computer that realizes a student terminal 20 according to the first embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of the software configuration of the classroom video distribution device 10 according to the first embodiment of the present disclosure. FIG. 1 is a flowchart illustrating a processing flow of a classroom video distribution method according to a first embodiment of the present disclosure. A diagram showing an example of the software configuration of a classroom video distribution device 10 according to a second embodiment of the present disclosure. 13 is a flowchart illustrating the process flow of a classroom video distribution method according to a second embodiment of the present disclosure. A diagram showing an example of the software configuration of a classroom video distribution device 10 according to a third embodiment of the present disclosure. 13 is a flowchart illustrating the process flow of a classroom video distribution method according to a third embodiment of the present disclosure. A diagram showing an example of the software configuration of a classroom video distribution device 10 according to a fourth embodiment of the present disclosure. 13 is a flowchart illustrating the process flow of a classroom video distribution method according to a fourth embodiment of the present disclosure.

The contents of the embodiments of the present disclosure will be described below. The present disclosure has the following configuration.
[Item 1]
An acquisition unit that acquires a moving image composed of a plurality of frames;
A storage unit that stores a specific condition for identifying predetermined data included in the video;
an extracting unit that extracts a plurality of specific frame groups from the video in accordance with the specific condition;
a connecting portion that connects the extracted specific frame groups together;
and an output unit that outputs digest information including a plurality of linked frame groups.
Information extraction device.
[Item 2]
Item 1 is an information extraction device according to the present invention,
A waveform registration unit that registers predetermined waveform data in advance,
The specific condition is whether or not waveform data of a sound included in the moving image matches the registered waveform data,
When both waveform data match, the extraction unit extracts a frame group corresponding to the matching waveform from the video as the specific frame group.
Information extraction device.
[Item 3]
Item 2. An information extraction device according to the present invention,
A conversion unit that converts the sounds included in the video into text information by speech recognition,
the conversion unit converts the sound corresponding to an auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group.
Information extraction device.
[Item 4]
The information extraction device according to item 2 or 3,
The information indicated by the surrounding sounds including the subject includes a mixture of conversation information and non-conversation information.
Information extraction device.
[Item 5]
Item 4. An information extraction device according to item 4,
The conversation information includes at least one of words indicating positive emotions and words indicating negative emotions.
Information extraction device.
[Item 6]
Item 4 or Item 5. An information extraction device according to item 4 or 5,
The non-conversational information includes information indicating at least one of clicking of the tongue, sighing, and interjections.
Information extraction device.
[Item 7]
Item 1 is an information extraction device according to the present invention,
A face information registration unit that registers a predetermined face evaluation value related to a facial expression in advance,
the specific condition is whether or not a face evaluation value calculated from a facial expression included in the moving image matches the registered face evaluation value,
when the two face evaluation values match, the extraction unit extracts a frame group corresponding to the matching face evaluation values from the video as the specific frame group.
Information extraction device.
[Item 8]
Item 7: An information extraction device according to item 7,
The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the person.
Information extraction device.
[Item 9]
The information extraction device according to item 7 or 8,
The face evaluation value includes evaluation values of the facial expression of the person, the direction of the gaze of the person, and the direction of the face of the person.
Information extraction device [Item 10]
Item 1 is an information extraction device according to the present invention,
a motion information registration unit that registers in advance a predetermined motion evaluation value related to a motion of a person,
the specific condition is whether or not a motion evaluation value calculated from a person included in the video matches the registered motion evaluation value;
when the two action evaluation values match, the extraction unit extracts a frame group corresponding to the matching action evaluation values from the video as the specific frame group.
Information extraction device.
[Item 11]
Item 11. An information extraction device according to item 10,
The motion evaluation value includes an evaluation value that evaluates at least one of the person's gestures, hand movements, gestures, and body language.
Information extraction device.

[Item 12]
Item 1 is an information extraction device according to the present invention,
a biometric information registration unit for registering in advance a biometric evaluation value relating to predetermined biometric information,
the specific condition is whether or not a biometric evaluation value that can be calculated from a person included in the moving image matches the registered biometric evaluation value,
when the two biometric evaluation values match, the extraction unit extracts a frame group corresponding to the matching biometric evaluation values from the video as the specific frame group.
Information extraction device.
[Item 13]
Item 13. An information extraction device according to item 12,
The biometric evaluation value includes at least one of the person's blood pressure, pulse rate, and pulse pressure.
Information extraction device.
[Item 14]
An information extraction device according to any one of items 1 to 13,
a frame adding unit that adds additional frames that are consecutive in time series to the specific frame group to the specific frame group,
Information extraction device.
[Item 15]
An identification unit that identifies at least a face image or a voice included in the digest information extracted by the information extraction device according to any one of items 1 to 14 for each predetermined frame;
and an evaluation means for calculating an evaluation value for the classified face image.
Video meeting evaluation terminal.
[Item 16]
Item 16. A video meeting evaluation terminal according to item 15,
the video meeting evaluation terminal provides graph information of the evaluation value in time series;
Video meeting evaluation terminal.
[Item 17]
17. The video meeting evaluation terminal according to claim 15,
the video meeting evaluation terminal calculates a plurality of evaluation values by evaluating the face image from a plurality of different viewpoints;
Video meeting evaluation terminal.
[Item 18]
18. A video meeting evaluation terminal according to any one of items 15 to 17,
the video meeting evaluation terminal calculates the evaluation value together with the audio included in the video;
Video meeting evaluation terminal.
[Item 19]
19. A video meeting evaluation terminal according to any one of items 15 to 18,
the video meeting evaluation terminal calculates the evaluation value together with an object other than the face image included in the moving image;
Video meeting evaluation terminal.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configuration are referred to as follows:
The same reference numerals are used to avoid redundant explanation.

In this disclosure, a cram school or educational institution is connected to the student's home via a digital communication line, and each student can watch the lecture video given at the cram school or educational institution from the comfort of their own home.
An example of application of the information extraction device to a home tutoring system where students can take lessons at a cram school or educational institution will be described.

First Embodiment
1 is a conceptual diagram showing a configuration example of a home-based individual instruction system 1 according to a first embodiment of the present disclosure. As shown in the figure, in this home-based individual instruction system 1, a classroom video distribution device 10 provided in a classroom of an instructor T who conducts remote lessons, and student terminals 20A, 20B, and 20C respectively associated with groups of students (students A, B, and C) who receive instruction at home are connected to a network N.
In the following description, the student terminals 20A, 20B, and 20
When there is no need to particularly distinguish between students A, B, and C, they will be simply referred to as student terminal 20. Similarly, when there is no need to particularly distinguish between students A, B, and C, they will be simply referred to as students.

The classroom video distribution device 10 is an example of an information extraction device described in the claims. Note that this configuration is just one example, and one configuration may combine with other configurations or include other configurations. Note that although the case of three students A, B, and C is shown here, the number of students may be greater depending on the number of students that the instructor can teach at the same time or the number of connection lines of the network NW, etc.

In this embodiment, the term "lecturer" refers to a concept that includes professors, teachers, and instructors. The term "classroom" refers to a concept that includes cram schools, cultural centers, and educational institutions (for example, primary, secondary, and higher education institutions, high schools, technical colleges, vocational schools, junior colleges, four-year universities, graduate schools, and other schools registered with the Ministry of Education, Culture, Sports, Science and Technology). The term "student" refers to a concept that includes pupils, students, and auditors.

In this embodiment, the network NW is assumed to be the Internet.
W is constructed, for example, by a public telephone line network, a mobile phone line network, a wireless communication network, Ethernet (registered trademark), and the like.

<Hardware Configuration>
2 is a diagram showing an example of the hardware configuration of a computer that realizes the classroom video distribution device 10 according to this embodiment. The computer includes at least a communication unit 11, an imaging unit 12, a sound collection unit 13, a monitor 14, a memory 15, a storage 16, an input/output unit 17, and a control unit 18. These are electrically connected to each other via a bus 19.

The communication unit 11 connects the classroom video distribution device 10 to a network NW. The communication unit 11 is, for example, a wired LAN (Local Area Network), a wireless LAN, or a Wi-Fi (Wireless Network).
The device communicates with external devices directly or via a network access point using a method such as Bluetooth®, infrared communication, Bluetooth®, short-range or contactless communication.

The imaging unit 12 has a function of electronically capturing images using an imaging element such as a CMOS or a CCD.
The imaging unit 12 captures an image of the lecturer T who is giving a lecture to the students as a subject, and obtains the lecturer video. The imaging unit 12 is preferably configured to capture images of writings on a blackboard or whiteboard used by the lecturer T when proceeding with the lecture, but a separate camera may be provided for the blackboard or whiteboard.

The sound collection unit 13 collects surrounding sounds including the instructor T. The sound collection unit 13 includes a microphone and the like for acquiring surrounding sounds including the voice of the instructor T. Furthermore, the sound collection unit 13 can perform appropriate processing such as converting the acquired sounds into electrical signals.

The monitor 14 can display, in a viewable manner, the student video transmitted from the student terminal 20 and the instructor video acquired by the imaging unit 12. Of course, the monitor 14 may display only the student video alone, or only the instructor video alone.

The memory 15 is a dynamic random access memory (DRAM).
The main memory is made up of a volatile storage device such as a flash memory or a hard disk drive (HDD).
The memory 15 is used as a work area for the control unit 18 and stores a BIOS (Basic Input/Output System) that is executed when the classroom video distribution device 10 is started up.
It also stores various setting information, etc.

The storage 16 stores various programs such as application programs, etc. A database that stores data used in each process may be constructed in the storage 16.

The input/output unit 17 is, for example, an information input device such as a keyboard, a mouse, or a touch panel.

The control unit 18 is a computing device that controls the overall operation of the classroom video distribution device 10, controls data transmission and reception between each element, and performs information processing necessary for application execution and authentication processing. For example, the control unit 18 is a central processing unit (CPU).
The processor 14 is a processor such as a processor for information processing (IT) that executes programs stored in the storage 16 and deployed in the memory 15 to carry out various information processing operations.

A bus 19 is commonly connected to each of the above elements, and transmits, for example, address signals, data signals, and various control signals.

3 is a diagram showing an example of the hardware configuration of a computer that realizes the student terminal 20 according to the present embodiment. The computer includes at least a communication unit 21, an imaging unit 22, and a sound collection unit 23.
The student terminal 20 according to this embodiment includes a computer and other hardware components that realize the student terminal 20, such as a display 23, a display 24, a memory 25, a storage 26, an input/output unit 27, and a control unit 28. These components are electrically connected to each other via a bus 29. The hardware configuration of a computer or the like that realizes the student terminal 20 according to this embodiment is similar to the hardware configuration example of the classroom video distribution device 10 shown in Figure 2, so only the differences will be described.

The communication unit 21 connects the student terminal 20 to the network NW.

The imaging unit 22 captures images of students attending a lecture as subjects and obtains student video.

The sound pickup unit 23 picks up surrounding sounds including those of the students. To transmit and receive voice data, the student terminal 20 may be provided with a headphone with a microphone, or a microphone and a speaker built into the terminal may be used.

The monitor 24 can display, in a viewable manner, the lecturer video transmitted from the classroom video distribution device 10 and the student video captured by the imaging unit 22. Of course, the monitor 24 may display only the lecturer video alone, or only the student video alone.

The control unit 28 is a calculation device that controls the overall operation of the student terminal 20, controls the transmission and reception of data between each element, and performs information processing required for application execution and authentication processing.

<Software configuration>
FIG. 4 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment.
The classroom video distribution device 10 comprises an extraction unit 101, a waveform registration unit 102, a conversion unit 103, a display unit 104, a frame extraction unit 105, and a generation unit 106.

The extraction unit 101, the waveform registration unit 102, the conversion unit 103, the display unit 104, the frame cut-out unit 105, and the generation unit 106 can be realized by the control unit 18 reading out a program stored in the storage 16 into the memory 15 and executing it.

The extraction unit 101 appropriately selects and synthesizes the lecturer video captured by the imaging unit 12 and the student video captured by the imaging unit 22, and generates a classroom video by combining sounds collected by the sound collection unit 13 or the sound collection unit 23. The lecturer video or student video here is an example of a plurality of frames as recited in the claims. The classroom video is an example of a moving image as recited in the claims. The classroom video may include text data, numerical data, graphic data, image data,
This refers to video data, audio data, etc., or a combination of these, which can be stored, edited, searched, etc., and can be exchanged as an individual unit between systems or users, and includes similar data.

The extraction unit 101 has a function of extracting a specific frame group from the classroom video in accordance with a specific condition instructed from outside. For example, the external instruction may instruct the items of the classroom video to be synthesized, the image arrangement, the area occupied by the image, etc. It is preferable that the external instruction is, for example, a simple operation that allows the lecturer T himself to edit the video composition in the middle of a lecture.

The waveform registration unit 102 has a function of registering predetermined waveform data in advance. The predetermined specific condition may be, for example, whether or not the waveform data of the sound collected by the sound collection unit 13 or the sound collection unit 23 matches the waveform data registered in the waveform registration unit 102, and may of course be other conditions. In this embodiment, the waveform data registered in the waveform registration unit 102 can be used to evaluate the overall atmosphere of the lecture.

When a predetermined specific condition is met, for example when both waveform data match, the extraction unit 101 extracts a group of frames corresponding to the matching waveforms as a specific group of frames from the classroom video.

The conversion unit 103 has a function of converting the sound collected by the sound collection unit 13 into text by speech recognition. This text is any character string. If the speech recognition is successful, the conversion unit 103 outputs a speech recognition result including the generated text. The speech recognition result may include success information indicating that the speech recognition was successful.

The display unit 104 displays the text converted by the conversion unit 103 on the monitor 14 or the monitor 24.
The information displayed by the surrounding sounds including the instructor T or the students A, B, and C includes the following:
Conversational information and non-conversational information are mixed in. Conversational information includes, for example, at least one of words indicating positive emotions and words indicating negative emotions.

Examples of words that show positive emotions include when a teacher praises or encourages a student.
Examples of encouraging words include "You did a great job,""You put in effort,""That'samazing,""That'swonderful,""You'reimpressive," and "You're great."

On the other hand, examples of words that indicate negative emotions include instructors using words to belittle, criticize, or slander students, such as "That's no good,""You're no good,""What are you doing,""You'llfail," and "You're stupid."

Non-conversational information is text information other than conversational information. Non-conversational information includes information indicating at least one of clicking the tongue, sighing, and interjections. Such non-conversational information can be a criterion for judging the emotions of students attending a lecture. The emotions of students can be classified into three categories, for example, "happiness,""boredom," and "tension."

The frame extraction unit 105 has a function of extracting at least a group of frames that are consecutive before and after a specific group of frames extracted by the extraction unit 101 in chronological order. The frame extraction unit 105 can extract a group of frames that are consecutive before and after a specific group of frames using any frame rate that is applicable to obtain, for example, context information indicating in what context each word is used. The context information here is, for example, information indicating a character string in an arbitrary range before and after a word, a co-occurrence relationship between words, etc.

The generating unit 106 has a function of generating a digest movie by linking a group of preceding and succeeding frames that are consecutive before and after a specific group of frames extracted by the extracting unit 101 .

Next, the operation of the home individual instruction system 1 configured as above will be described.
1 is a flowchart illustrating a processing flow of a classroom video distribution method according to a first embodiment of the present disclosure.

Here, we will explain an example of a classroom administrator checking whether a lecturer is making inappropriate or problematic remarks to students. Specifically, using waveform data representing the word "baka" (idiot) in a pre-registered lecturer's voice,
An example of a scene where a digest movie is generated will be described.

First, when the time for the lecture to start arrives, each of the students A, B, and C accesses the student terminals 20A, 20B.
, 20C are connected to the home tutoring system 1 via the network NW and wait for the lecture of the instructor T to begin. The instructor T watches the monitor 14 of the classroom video distribution device 10 and
It is determined whether C is ready to attend the lecture, and if C is ready to attend, the lecture is started. That is, the imaging unit 12 and the imaging unit 22 start imaging operations, and the sound collection unit 13 and the sound collection unit 23 start collecting sounds (step S100).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor video captured by the imaging unit 12 and the student video captured by the imaging unit 22, combines the sounds collected by the sound collection unit 13 or the sound collection unit 23 to generate a classroom video, and distributes the generated classroom video to the student terminals 20A, 20B, and 20C via the network NW (step S102).

Next, when the teacher utters the word "idiot", the voice is picked up by the sound pickup unit 13, and the classroom video including the voice data is extracted as a specific frame group by the extraction unit 101 (step S104).

The frame extracting unit 105 extracts a series of frames at time intervals of, for example, about 10 to 20 seconds before and after the classroom video (step S106).

The generating unit 106 then generates a digest movie by connecting these frames together, and performs a predetermined acoustic analysis on the instructor's voice data based on the digest movie (step S108).

This acoustic analysis allows us to understand the context in which the word "baka" was used.

For example, if an instructor uses the word "idiot" in the context of saying (to a third party other than the students) "People who say stupid things like this are no good," it is clear that the instructor did not mean to call the students stupid.

For example, if a teacher uses the word "idiot" in the context of "I'm an idiot," it can be understood that the teacher is calling himself an idiot.

Thus, a classroom administrator or the like can obtain the word "baka" and the context before and after it as a text sentence, for example in the form of a report, and can easily check whether the instructor is making inappropriate or problematic remarks to students.

That is, by using the preregistered waveform data for comparison, it is possible to pinpoint characteristic scenes in the classroom video, including situations where the lecturer is likely to make inappropriate or problematic remarks, while reducing the load of computational processing. For example, if the lecturer utters a relatively large number of words indicating negative emotions (e.g., 10 times) during a lecture time (e.g., 90 minutes), it is possible to judge the lecturer's personality. Of course, the number of times the lecturer utters words indicating positive emotions can also be used to judge the lecturer's personality. Furthermore, not only on the lecturer's side, but also on the students' side, the number of times they click their tongues, sigh, and respond to the lecturer's remarks can be used to judge the students' emotions.

Second Embodiment
Hereinafter, an at-home tutoring system according to a second embodiment will be described with reference to Fig. 6 and Fig. 7. In this embodiment, the same elements as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

As already mentioned, in the above-mentioned first embodiment of the tutoring system, specific frames are extracted from classroom footage by matching with pre-registered waveform data, whereas in the second embodiment of the tutoring system, a process is performed to extract specific frames from classroom footage based on an evaluation value related to a person's facial expression.

<Software configuration>
FIG. 6 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment.
The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 106.
The image processing unit 102 includes a face detection unit 107, a calculation unit 108, and a face information registration unit 109.

The face detection unit 107 has a function of detecting human faces in each of a plurality of frames constituting the classroom video stored in the memory 15. For example, the face detection unit 107 extracts feature points from the classroom video and detects the face region, the size of the face region, the face area, etc. of the instructor T or each of the students A, B, and C.

Examples of the feature points include the end points of the eyebrows, eyes, nose, and lips, the contour points of the face, the vertex of the head, and the bottom end point of the chin. Then, the face detection unit 107 identifies the position information of the face area. For example, the face detection unit 107 calculates the X-coordinate and the Y-coordinate of the pixels included in the face area by taking the horizontal direction of the image as the X-axis and the vertical direction as the Y-axis. Furthermore, the face detection unit 107 can determine the facial expression, age, and the like of the detected instructor or each student by performing a calculation process using the above-mentioned feature points.

The calculation unit 108 has a function of calculating an evaluation value related to the face of the instructor T or each of the students A, B, and C for each of the multiple frames that make up the classroom video. The evaluation values calculated by the calculation unit 108 include the following predetermined evaluation values (1) to (6). These evaluation values (
1) to (6) are registered in advance in the face information registration unit 109.

(1) The smile degree calculation unit 108 calculates the smile degree as an evaluation value for each face detected by the face detection unit 107, using a known technique such as pattern matching. In this embodiment, examples of the degrees include "0: no smile", "1: smile", "2: normal smile", "3:
The level of smile is indicated on a four-point scale, ranging from "laughing loud."

(2) The gaze direction calculation unit 108 calculates the gaze direction as an evaluation value for each face detected by the face detection unit 107 using a known technique. In the present embodiment, as an example,
The direction of gaze is indicated by four types: "1: gaze left/right", "2: gaze right", and "3: gaze detection not possible".

(3) The face direction calculation unit 108 calculates the face direction as an evaluation value for each face detected by the face detection unit 107 using a known technique. In the present embodiment, as an example,
The face direction is indicated in four types: "1: face direction to the left", "2: face direction to the right", and "3: not detectable".

(4) The face area calculation unit 108 calculates the area of the face portion of each face detected by the face detection unit 107 as an evaluation value.

(5) The age calculation unit 108 calculates the age of each person detected by the face detection unit 107 as an evaluation value using a known technique.

(6) The eye-blinking degree calculation unit 108 uses known technology to calculate the degree of eye-blinking as an evaluation value for each face detected by the face detection unit 107. In the present embodiment, as an example, the eye-blinking degree is indicated by four types: "0: no eye-blinking", "1: some eye-blinking", "2: both eyes closed", and "3: eye-blinking not detectable".

These evaluation values (1) to (6) can be used as criteria for judging the emotions of students attending a lecture. The emotions of students can be classified into three categories, for example, "happiness,""boredom," and "tension."

When a predetermined specific condition is satisfied, for example, when the evaluation value calculated by the calculation unit 108 matches an evaluation value pre-registered in the face information registration unit 109, the extraction unit 101 extracts a group of frames corresponding to the matching evaluation value from the classroom video as a specific group of frames.

Next, the operation of the at-home individual instruction system configured as above will be described. Fig. 7 is a flowchart illustrating the process flow of the classroom video distribution method according to the second embodiment of the present disclosure.

Here, we will explain an example of a situation in which a classroom operator responds to a request from a parent or guardian who has their child (student A) attend a cram school or other such institution, who wants to understand student A's learning attitude and learning situation.

First, when the time for the lecture to start arrives, each of the students A, B, and C accesses the student terminals 20A, 20B.
, 20C are connected to the home tutoring system via the network NW and wait for the lecture of the instructor T to start. The instructor T watches the monitor 14 of the classroom video distribution device 10 and the students A, B, C
If the students are ready to attend the lecture, the lecture is started. That is, the imaging units 12 and 22 start imaging operations, and the sound collection units 13 and 23 start collecting sounds (step S200).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor video captured by the imaging unit 12 and the student video captured by the imaging unit 22, combines the sounds collected by the sound collection unit 13 or the sound collection unit 23 to generate a classroom video, and distributes the generated classroom video to the student terminals 20A, 20B, and 20C via the network NW (step S202).

Next, the face detection unit 107 detects the face of student A in the classroom video (step S20
4) The specific method of face detection is similar to known techniques, and therefore a description thereof will be omitted.

Then, the calculation unit 108 calculates the orientation of the face of the student A detected by the face detection unit 107 as an evaluation value using a known technique (step S206).
In the case where the evaluation value calculated by the calculation unit 108 matches any one of the evaluation values "1: face direction to the left" and "2: face direction to the right" registered in advance in the face information registration unit 109, the frame group corresponding to the matching evaluation value is extracted as a specific frame group from the classroom video (
Step S208).

That is, if Student A's face is not facing forward but is facing to the left or right, it can be inferred that Student A's concentration on the lecture is declining.

The frame extraction unit 105 extracts a series of frames, for example, at a time interval of about 10 to 20 seconds, before and after the classroom video in which the face of student A is not facing forward (step S2
10).

The generator 106 then generates a digest video by connecting these frames, and performs a predetermined acoustic analysis on the instructor's voice data based on the digest video (step S212). This acoustic analysis allows a flag to be used to manage only the words that instructor T utters when student A is not facing forward. This allows the administrator of the classroom to collect words that may be the cause of student A's reduced concentration on the lecture, and also efficiently responds to the requests of parents and others who want to understand student A's learning attitude and learning situation.

Of course, the degree of concentration of Student A on the lecture can be determined not only from the direction of Student A's face but also from the direction of Student A's gaze. In other words, if Student A's gaze direction is not forward but toward the left or right, it can be inferred that Student A's degree of concentration on the lecture is decreasing.

Further possible use cases for the at-home individual tutoring system according to this embodiment include the following:

Specifically, a video may be created by editing only scenes in which student A felt positive emotions such as satisfaction, happiness, and fulfillment, and the edited video may be generated as a digest video for student A's parents, etc.

As an example of a specific process for generating such a digest movie, the calculation unit 108
The face detection unit 107 can use a known technique to calculate the degree of smiling as an evaluation value for the face of the student A detected by the face detection unit 107. Next, the extraction unit 101 calculates the degree calculated by the calculation unit 108 and the degree of smiling "2: normal smiling" registered in advance in the face information registration unit 109.
, "3: laughing loudly," then the frame group corresponding to the matching evaluation value can be extracted as a specific frame group from the classroom video.

Furthermore, the frame extraction unit 105 determines that the degree of smile of the student A is “2: normal smile”;
A group of frames that are continuous at a time interval of, for example, about 10 to 20 seconds before and after the classroom video when the subject is in any of the categories of “3: laughing loudly” can be extracted.
The video processing unit 100 generates a digest video by connecting these frames together, as a digest video intended for Student A's parents and guardians.

Third Embodiment
Hereinafter, a home tutoring system according to a third embodiment will be described with reference to Fig. 8 and Fig. 9. In this embodiment, the same elements as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

As already mentioned above, in the at-home tutoring system of the first embodiment, specific frames were extracted from classroom footage by matching with pre-registered waveform data, whereas in the at-home tutoring system of the third embodiment, a process is performed to extract specific frames from classroom footage based on movement information relating to a person's movements.

<Software configuration>
8 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 10
6, a specification unit 110, and a motion information registration unit 111.

The identification unit 110 has a function of identifying motion information related to a person's motion for each of a plurality of frames constituting the classroom video stored in the memory 15. This motion information is, for example, information that captures a person's motion as a sequence of a plurality of postures, and may include information on each joint that forms the skeleton of the human body corresponding to the various postures. The motion information includes, for example, at least one of a person's gestures, hand movements, gestures, and body language.

The identification unit 110 obtains the coordinates of each joint forming the skeleton of the human body from the multiple frames constituting the classroom video, for example, by pattern matching using a human body pattern. The specific method of obtaining the coordinates is similar to known techniques, and therefore will not be described here. The coordinates of each joint expressed in this coordinate system can be, for example, skeleton information for one frame. Furthermore, skeleton information for multiple frames can be predetermined motion information.

Such predetermined motion information is pre-registered in the motion information registration unit 111. That is, the motion information registration unit 111 pre-stores motion information corresponding to various postures, for example, by machine learning using a known artificial intelligence technology. For example, in this embodiment, a gesture corresponding to a student's hand waving movement has been pre-programmed by machine learning as a gesture pattern for a student who is not convinced or has difficulty with the content of a lecture to ask the lecturer for additional explanation.

When a predetermined specific condition is satisfied, for example, when the motion information identified by the identification unit 110 matches the motion information pre-registered in the motion information registration unit 111, the extraction unit 101 extracts a group of frames corresponding to the matching motion information from the classroom video as a specific group of frames.

Next, the operation of the at-home individual instruction system thus configured will be described. Fig. 9 is a flowchart illustrating the process flow of the classroom video distribution method according to the third embodiment of the present disclosure.

Here, an example will be described in which a scene is picked up from a classroom video including a situation in which a student A, who is not convinced by the content of the lecture, asks the lecturer for additional explanation.

First, when the time for the lecture to start arrives, each of the students A, B, and C accesses the student terminals 20A, 20B.
, 20C are connected to the home tutoring system via the network NW and wait for the lecture of the instructor T to start. The instructor T watches the monitor 14 of the classroom video distribution device 10 and the students A, B, C
If the students are ready to attend the lecture, the lecture is started. That is, the imaging units 12 and 22 start imaging operations, and the sound collection units 13 and 23 start collecting sounds (step S300).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor video captured by the imaging unit 12 and the student video captured by the imaging unit 22, combines the sounds collected by the sound collection unit 13 or the sound collection unit 23 to generate a classroom video, and distributes the generated classroom video to the student terminals 20A, 20B, and 20C via the network NW (step S302).

Next, the identification unit 110 obtains the coordinates of each joint forming the skeleton of student A from multiple frames constituting the classroom video by pattern matching using the human body pattern. Furthermore, the identification unit 110 identifies the skeleton information for multiple frames as gestures of student A based on the coordinates of each joint (step S304).

Next, if the gesture identified by the identification unit 110 matches a gesture pattern that has been machine-learned in the action information registration unit 111 (a gesture pattern in which student A requests additional explanation from the instructor), the extraction unit 101 extracts a group of frames corresponding to the matching gesture as a specific group of frames from the classroom video (step S306).

The frame extracting unit 105 extracts a series of frames at time intervals of, for example, about 10 to 20 seconds before and after the classroom video in which Student A requests additional explanation from the instructor (step S308).

The generator 106 then generates a digest movie by connecting these frames (
Step S310).

Thus, since the gesture pattern by which student A requests further explanation from the instructor has already been registered, if a situation similar to the above flow occurs in the future, a specific group of frames can be picked up from the classroom video by matching against the gesture pattern stored in the motion information registration unit 111, and a similar digest video can be generated without increasing the load associated with computational processing.

Fourth Embodiment
Hereinafter, a home tutoring system according to a fourth embodiment will be described with reference to Fig. 10 and Fig. 11. In this embodiment, the same elements as those described in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

As already mentioned above, in the at-home tutoring system of the first embodiment, specific frames were extracted from classroom video footage by matching with pre-registered waveform data, whereas in the at-home tutoring system of the fourth embodiment, a process is performed to extract specific frames from classroom video footage based on a person's biometric information.

<Software configuration>
10 is a diagram showing an example of the software configuration of the classroom video distribution device 10 according to this embodiment. The classroom video distribution device 10 includes an extraction unit 101, a frame extraction unit 105, and a generation unit 106.
06, a biometric information detection unit 112, and a biometric information registration unit 113.

The biometric information detection unit 112 has a function of detecting biometric information of a person for each of a plurality of frames constituting the classroom video stored in the memory 15. The biometric information of a person includes at least one of the person's blood pressure, pulse rate, and pulse pressure. These predetermined biometric information are:
The facial area of the instructor or student reflected in each frame is extracted using general facial detection technology, and then divided into multiple areas along the direction of blood flow, and the facial area can be obtained based on the time-series changes in color images indicating the blood flow in each area.

Such predetermined biometric information is registered in advance in the biometric information registration unit 113. That is, the biometric information registration unit 113 stores in advance, for example, biometric information used to detect mental states such as the presence or absence of tension, and physical states such as physical condition, for example, by machine learning using known artificial intelligence technology. For example, in this embodiment, it is assumed that biometric information capable of reading emotions unconsciously controlled by student A, such as changes in facial expression in milliseconds, pupil dilation, pulse rate (pulse rate), facial flushing, degree of sweating, etc., has been learned.

When a predetermined specific condition is met, for example, when the biometric information detected by the biometric information detection unit 112 matches the biometric information pre-registered in the biometric information registration unit 113, the extraction unit 101 extracts a group of frames corresponding to the matching biometric information from the classroom video as a specific group of frames.

Next, the operation of the home individual instruction system thus configured will be described.
13 is a flowchart illustrating the process flow of a classroom video distribution method according to a fourth embodiment of the present disclosure.

Here, based on the knowledge that humans have a genetic bias to maintain physical safety and that they instantly sense something abnormal when they encounter something unfamiliar or difficult to understand, we will explain by using as an example a scene from a classroom video in which Student A falls into a state of tension.

First, when the time for the lecture to start arrives, each of the students A, B, and C accesses the student terminals 20A, 20B.
, 20C are connected to the home tutoring system via the network NW and wait for the lecture of the instructor T to start. The instructor T watches the monitor 14 of the classroom video distribution device 10 and the students A, B, C
If the students are ready to attend the lecture, the lecture is started. That is, the imaging units 12 and 22 start imaging operations, and the sound collection units 13 and 23 start collecting sounds (step S400).

Then, the extraction unit 101 appropriately selects and synthesizes the instructor video captured by the imaging unit 12 and the student video captured by the imaging unit 22, combines the sounds collected by the sound collection unit 13 or the sound collection unit 23 to generate a classroom video, and distributes the generated classroom video to the student terminals 20A, 20B, and 20C via the network NW (step S402).

Next, the biological information detection unit 112 detects the pulse rate of student A for each of the frames constituting the classroom video using a known technique (step S404).

Next, when the pulse rate detected by the biometric information detection unit 112 matches the pulse rate that has been machine-learned in the biometric information registration unit 113 (the pulse rate when student A is in a state of tension), the extraction unit 101 extracts a group of frames corresponding to the matching pulse rate from the classroom video as a specific group of frames (step S406).

The frame extracting unit 105 extracts a series of frames, for example, at time intervals of about 10 to 20 seconds, before and after the classroom video in which Student A is in a tense state (step S408).

The generator 106 then generates a digest movie by connecting these frames (
Step S410).

Thus, since the pulse rate of student A when he is in a state of tension has already been registered, if a situation similar to that described above occurs in the future, a specific group of frames can be picked out from the classroom video by matching it with the pulse rate stored in the biometric information registration unit 113, and a similar digest video can be generated without increasing the load associated with calculation processing.

Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modified or amended examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally belong to the technical scope of the present disclosure.

In the above-described embodiments, the information extraction device is applied to a home tutoring system that supports remote classes. However, the present invention is not limited to this. For example, the information extraction device may be applied to a system that supports various events, such as meetings and lectures, whose start and end times are determined in advance and which are mainly held in a specific space such as a classroom or a conference room.

The devices described in this specification may be realized as a single device, or may be realized by multiple devices (e.g., cloud servers) partially or entirely connected via a network. For example, the storage 16 or the control unit 18 of the classroom video distribution device 10 may be realized by different servers connected to each other via a network.

The series of processes performed by the device described in this specification may be realized using software, hardware, or a combination of software and hardware. A computer program for realizing each function of the classroom video distribution device 10 according to this embodiment can be created and installed in a PC or the like. A computer-readable recording medium on which such a computer program is stored can also be provided. Examples of the recording medium include a magnetic disk, an optical disk, a magneto-optical disk, and a flash memory. The above computer program may also be distributed, for example, via a network, without using a recording medium.

In addition, the processes described herein using flowchart diagrams do not necessarily have to be performed in the order shown. Some process steps may be performed in parallel. Additional process steps may be employed, and some process steps may be omitted.

In addition, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to a person skilled in the art from the description of this specification, in addition to or in place of the above effects.

1 Home-based individual instruction system 10 Classroom video distribution device (information extraction device)
REFERENCE SIGNS LIST 101 Extraction unit 102 Waveform registration unit 103 Conversion unit 104 Display unit 105 Frame extraction unit 106 Generation unit 107 Face detection unit 108 Calculation unit 109 Face information registration unit 110 Identification unit 111 Motion information registration unit 112 Biometric information detection unit 113 Biometric information registration unit NW Network

Claims (18)

An acquisition unit that acquires a moving image composed of a plurality of frames;
A storage unit that stores a specific condition for identifying predetermined data included in the video;
an extracting unit that extracts a plurality of specific frame groups from the video in accordance with the specific condition;
a connecting portion that connects the extracted specific frame groups together;
an output unit that outputs digest information including a group of multiple linked frames;
an identification means for identifying at least a face image or a voice included in the digest information for each predetermined frame;
and an evaluation means for calculating an evaluation value for the classified face image.
Device. 2. The apparatus of claim 1,
A waveform registration unit that registers predetermined waveform data in advance,
The specific condition is whether or not waveform data of a sound included in the moving image matches the registered waveform data,
When both waveform data match, the extraction unit extracts a frame group corresponding to the matching waveform from the video as the specific frame group.
Device. 3. The apparatus of claim 2,
A conversion unit that converts the sounds included in the video into text information by speech recognition,
the conversion unit converts the sound corresponding to an auxiliary frame group including the specific frame group and a predetermined number of frames before and after the specific frame group.
Device. 4. An apparatus according to claim 2 or claim 3, comprising:
The information provided by the surrounding sounds, including the subject , is a mixture of conversational and non-conversational information.
Device. 5. The apparatus of claim 4,
The conversation information includes at least one of words indicating positive emotions and words indicating negative emotions.
Device. 6. An apparatus according to claim 4 or claim 5,
The non-conversational information includes information indicating at least one of clicking of the tongue, sighing, and interjections.
Device. 2. The apparatus of claim 1,
A face information registration unit that registers a predetermined face evaluation value related to a facial expression in advance,
the specific condition is whether or not a face evaluation value calculated from a facial expression included in the moving image matches the registered face evaluation value,
when the two face evaluation values match, the extraction unit extracts a frame group corresponding to the matching face evaluation values from the video as the specific frame group.
Device. 8. The apparatus of claim 7,
The face evaluation value includes an evaluation value that evaluates the degree of happiness, boredom, or tension of the person .
Device. 9. An apparatus according to claim 7 or claim 8, comprising:
The face evaluation value includes evaluation values of a person 's facial expression, a direction of the person's gaze, and a direction of the person's face.
Device. 2. The apparatus of claim 1,
a motion information registration unit that registers in advance a predetermined motion evaluation value related to a motion of a person,
the specific condition is whether or not a motion evaluation value calculated from a person included in the video matches the registered motion evaluation value;
when the two action evaluation values match, the extraction unit extracts a frame group corresponding to the matching action evaluation values from the video as the specific frame group.
Device. 11. The apparatus of claim 10,
The motion evaluation value includes an evaluation value that evaluates at least one of the person's gestures, hand movements, gestures, and body language.
Device. 2. The apparatus of claim 1,
a biometric information registration unit for registering in advance a biometric evaluation value relating to predetermined biometric information,
the specific condition is whether or not a biometric evaluation value that can be calculated from a person included in the moving image matches the registered biometric evaluation value,
when the two biometric evaluation values match, the extraction unit extracts a frame group corresponding to the matching biometric evaluation values from the video as the specific frame group.
Device. 13. The apparatus of claim 12,
The biometric evaluation value includes at least one of the person's blood pressure, pulse rate, and pulse pressure.
Device. 14. An apparatus according to any one of claims 1 to 13, comprising:
a frame adding unit that adds additional frames that are consecutive to the specific frame group in time series to the specific frame group,
Device. 2. The apparatus of claim 1,
the video meeting evaluation terminal provides graph information of the evaluation value in time series;
Device. 16. The apparatus of claim 15 ,
the video meeting evaluation terminal calculates a plurality of evaluation values by evaluating the face image from a plurality of different viewpoints;
Device. 17. Apparatus according to any one of claims 1 to 16, comprising:
calculating the evaluation value together with the audio included in the moving image;
Device. 18. Apparatus according to any one of claims 1 to 17, comprising:
calculating the evaluation value together with an object other than the face image included in the moving image;
Device.

Images