JP6897095B2 - Image processing program, image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing program, an image processing apparatus, and an image processing method.

In lectures, lectures, and lectures (hereinafter referred to as lectures) that many students take, it is difficult to understand what kind of lecture content and lecture technique is effective in order to concentrate each student on the lecture. , Important for those who provide lectures (hereinafter, lecturers, etc.). Then, in order to grasp the correlation between the lecture content and the lecture technique and the state of each student, it is first required to quantitatively analyze the state of each student during the course.

On the other hand, for example, a technique has been proposed in which an image of a lecture or the like is processed and the state of each student is analyzed. Specifically, by shooting a scene including a large number of students and analyzing the inclination of the face of the person included in the image obtained by the shooting, the state of the person (for example, nodding state, dozing state, etc.) can be obtained. An image processing system that calculates quantitatively has been proposed.

JP-A-2007-97668

However, in order to analyze the state of a person included in an image using the above image processing system, detailed facial information (for example, eyebrow position, facial contour, etc.) of the person is calculated in the captured image. There is a need. Therefore, if it is intended to be applied to a lecture or the like attended by a large number of students, it is necessary to install a large number of high-performance imaging devices, which causes a problem of cost.

For this reason, it is required to be able to analyze the state of a person included in an image of a lecture or the like using a simple image processing system (that is, efficiently).

One aspect is to enable efficient analysis of the state of a person contained in an image.

According to one aspect, the image processing program
The position where the face of the lecturer's face exists is extracted from each of the plurality of images obtained by shooting the lectures held in the same shooting range at different timings.
An image area is specified for each of the plurality of images based on the extracted position where the face exists.
The result according to the presence or absence of the face in the specified image area is associated with the information indicating the timing at which each of the plurality of images was taken, and is output at a plot cycle according to the format of the lecture.
Let the computer perform the process.

The state of the person included in the image can be efficiently analyzed.

It is a figure which shows an example of the system configuration of an image processing system. It is a figure which shows an example of the hardware composition of an image processing apparatus. It is a figure which shows the functional structure of the image processing part of an image processing apparatus. It is a figure which shows the specific example of the processing of a face candidate detection part. It is a figure which shows an example of the detection result information. It is a figure which shows the specific example of the processing of the 1st narrowing-down part. It is a figure which shows the specific example of the processing of the 2nd narrowing-down part. It is a figure which shows the specific example of the processing of the detection number time series data generation part. It is the first figure which shows the specific example of the processing of the change number time series data generation part. It is the 2nd figure which shows the specific example of the processing of the change number time series data generation part. It is a 1st flowchart of image processing by an image processing apparatus. It is a 2nd flowchart of image processing by an image processing apparatus. It is a figure which shows another example of time-series data of the number of face detections.

Hereinafter, each embodiment will be described with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to omit duplicate explanations.

[First Embodiment]
<System configuration of image processing system>
First, the system configuration of the entire image processing system will be described. FIG. 1 is a diagram showing an example of a system configuration of an image processing system.

As shown in FIG. 1, the image processing system 100 includes an image pickup device 110, a moving image data storage device 120, and an image processing device 130. The moving image data storage device 120 and the image processing device 130 are connected via the network 140.

The image pickup device 110 is installed toward the student side in a space where a large number of students take lectures, such as a lecture room of a university, and takes a picture of a range in which the student sits. .. Further, the imaging device 110 transmits the moving image data obtained by taking a picture to the moving image data storage device 120.

The moving image data storage device 120 stores the moving image data transmitted from the imaging device 110. Further, the moving image data storage device 120 transmits the moving image data to the image processing device 130 via the network 140 in response to a request from the image processing device 130.

An image processing program is installed in the image processing device 130. The image processing device 130 functions as the image processing unit 131 by executing the image processing program.

By processing the moving image data in frame units, the image processing unit 131 detects the student's face candidates (referred to as face candidate data) from the image data of each frame, and determines the size of the face candidate data and the face candidates. Extract the location where the data exists. In addition, the image processing unit 131 narrows down the detected face candidate data based on the extracted size and position, and calculates a "face detection number" indicating the number of students raising their faces for each image data. When the student is looking up, the student is concentrating on the lecture. Therefore, by calculating the number of face detections, it is possible to analyze whether or not each student is concentrating on lectures and the like.

The image processing unit 131 stores the number of face detection cases calculated for each image data in the analysis result storage unit 132 as time-series data in association with the time information indicating the timing at which each image data is taken.

Further, the image processing unit 131 calculates the "number of changes" indicating the frequency of raising and lowering the face of the student based on the result of face detection for each image data, associates it with the time information, and analyzes the result as time series data. It is stored in the storage unit 132. The state in which the student frequently raises and lowers his / her face means that the student is taking a lecture while taking notes, and is concentrating on the lecture. Therefore, by calculating the number of changes, it is possible to analyze whether or not each student is concentrating on lectures and the like.

In this way, the image processing device 130 calculates the number of face detection cases and the number of changes in order to analyze the state of the person included in the image. That is, in the case of the image processing device 130, while the face of the person is detected, the detailed face information of the person is not calculated. Therefore, it is not necessary to install a high-performance image pickup device as an image processing system, and the number of image pickup devices installed can be reduced.

That is, a simple image processing system is sufficient, and the state of a person included in the image can be efficiently analyzed.

<Hardware configuration of image processing device>
Next, the hardware configuration of the image processing device 130 will be described. FIG. 2 is a diagram showing an example of the hardware configuration of the image processing device. As shown in FIG. 2, the image processing device 130 includes a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, and a RAM (Random Access Memory) 203. The CPU 201, ROM 202, and RAM 203 form a so-called computer.

Further, the image processing device 130 includes an auxiliary storage device 204, a display device 205, an operation device 206, an I / F (Interface) device 207, and a drive device 208. Each part of the image processing device 130 is connected to each other via a bus 209.

The CPU 201 is a device that executes various programs (for example, an image processing program, etc.) installed in the auxiliary storage device 204.

ROM 202 is a non-volatile memory. The ROM 202 functions as a main storage device for storing various programs, data, and the like necessary for the CPU 201 to execute various programs installed in the auxiliary storage device 204. Specifically, the ROM 202 stores boot programs such as BIOS (Basic Input / Output System) and EFI (Extensible Firmware Interface).

The RAM 203 is a volatile memory such as a DRAM (Dynamic Random Access Memory) or a SRAM (Static Random Access Memory). The RAM 203 functions as a main storage device that provides a work area that is expanded when various programs installed in the auxiliary storage device 204 are executed by the CPU 201.

The auxiliary storage device 204 is an auxiliary storage device that stores various programs, information generated by executing various programs, and information used when various programs are executed. The analysis result storage unit 132 is realized in the auxiliary storage device 204.

The display device 205 “depends on the presence or absence of a face” associated with time information such as information indicating the result of image processing in the image processing device 130 (for example, face detection number time series data, change number time series data, etc., which will be described later). It is a display device that displays information ") indicating the results. The operation device 206 is an input device for the administrator of the image processing device 130 to input various instructions to the image processing device 130.

The I / F device 207 is a communication device for connecting to the network 140 and communicating with the moving image data storage device 120.

The drive device 208 is a device for setting the recording medium 210. The recording medium 210 referred to here includes a medium such as a CD-ROM, a flexible disk, a magneto-optical disk, or the like that optically, electrically, or magnetically records information. Further, the recording medium 210 may include a semiconductor memory or the like for electrically recording information such as a ROM or a flash memory.

The various programs stored in the auxiliary storage device 204 are installed, for example, by setting the distributed recording medium 210 in the drive device 208 and reading the various programs recorded in the recording medium 210 by the drive device 208. Will be done.

<Functional configuration of image processing device>
Next, the functional configuration of the image processing unit 131 of the image processing device 130 will be described. FIG. 3 is a diagram showing an example of the functional configuration of the image processing unit of the image processing device.

As shown in FIG. 3, the image processing unit 131 of the image processing device 130 includes a moving image data acquisition unit 301, a face candidate detection unit 302, a first narrowing unit 303, and a second narrowing unit 304. Further, the image processing unit 131 of the image processing device 130 includes a detection number time series data generation unit 305, a change number time series data generation unit 306, and a display control unit 307.

The moving image data acquisition unit 301 acquires moving image data from the moving image data storage device 120, and notifies the face candidate detection unit 302 of the image data in frame units.

The face candidate detection unit 302 is an example of an extraction unit. The face candidate detection unit 302 receives image data in frame units from the moving image data acquisition unit 301 and processes the received image data to detect the face candidate data of the student (person included in the image). .. The face candidate detection unit 302 detects the student's face candidate data as a rectangular area, and extracts the position (x coordinate, y coordinate) where the detected rectangular area exists and the size (width, height) of the detected rectangular area. To do. Further, the face candidate detection unit 302 stores the extracted position and size (width, height) as detection result information in the analysis result storage unit 132.

When the detection result information is newly stored in the analysis result storage unit 132, the first narrowing-down unit 303 sets a predetermined condition for the size (width, height) of each face candidate data included in the detection result information. Determine if it meets or not. The first narrowing-down unit 303 determines that the face candidate data that does not satisfy the predetermined condition is erroneously detected face candidate data, and deletes the face candidate data from the detection result information.

The second narrowing-down unit 304 is an example of a specific unit. The second narrowing-down unit 304 sets the remaining face candidate data at the position (x coordinate, x-coordinate,) of the detection result information in which the face candidate data determined not to satisfy the predetermined condition is deleted by the first narrowing-down unit 303. Based on the y-coordinate), it is classified into each predetermined image area. Further, the second narrowing-down unit 304 aggregates the number of face candidate data classified in each image area, and stores the aggregated result as the aggregated result information in the analysis result storage unit 132. Further, the second narrowing-down unit 304 calculates the number of face detection cases by subtracting the number of face candidate data classified into the image area equal to or less than a predetermined number in the aggregation result information.

The number of detected cases The time-series data generation unit 305 associates the number of face detection cases calculated for each image data with the time information indicating the timing at which each image data was taken by the second narrowing-down unit 304, so that the number of face detection cases is the time. Series data is generated and stored in the analysis result storage unit 132.

The number of changes time-series data generation unit 306 compares the aggregated result information of each image data with each other, calculates the difference (absolute value) of the number of face candidate data classified into each image area for each image area, and calculates the difference (absolute value). It is stored in the analysis result storage unit 132 as the difference information.

The difference (absolute value) is calculated by the number of changes from the state in which the student is raising his face to the state in which he is lowering, and the state in which the student is raising his face from the state in which his face is lowered. This is to count both the number of changes.

Further, the change number time series data generation unit 306 calculates the change number in the image data by totaling the differences (absolute values) calculated for each image area based on the difference information. Further, the change number time series data generation unit 306 generates time series data of the change number by associating the change number calculated for each image data with the time information, and stores it in the analysis result storage unit 132.

The display control unit 307 is an example of an output unit. The display control unit 307 displays the time-series data of the number of face detections and the time-series data of the number of changes stored in the analysis result storage unit 132 as "results according to the presence or absence of faces" associated with the time information. It is output to the display device 205 as "information". As a result, the display device 205 displays the time-series data of the number of face detection cases and the time-series data of the number of changes.

<Specific example of processing of each part included in the image processing part of the image processing device>
Next, each unit included in the image processing unit 131 of the image processing device 130 (here, the face candidate detection unit 302, the first narrowing-down unit 303, the second narrowing-down unit 304, the number of detections time-series data generation unit 305, changes. A specific example of the processing of the number time series data generation unit 306) will be described.

(1) Specific Example of Processing of Face Candidate Detection Unit FIG. 4 is a diagram showing a specific example of processing of the face candidate detection unit. Of these, FIG. 4A shows an example of moving image data acquired by the moving image data acquisition unit 301. As shown in FIG. 4A, the moving image data 400 includes image data 400_0 to 400_n obtained by taking a picture at each timing from _{time t 0 to t n.} The face candidate detection unit 302 sequentially receives each of the image data 400_0 to 400_n included in the moving image data 400 from the moving image data acquisition unit 301, and performs a process of detecting the face candidate data.

FIG. 4B shows the image data 400_0 received by the face candidate detection unit 302 from the moving image data acquisition unit 301. As shown in FIG. 4B, the face candidate detection unit 302 defines the upper left vertex of the image data 400_0 as the origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis, and specifies the coordinates of each pixel. Performs the process of detecting face candidate data with.

FIG. 4C shows an example of face candidate data detected by the face candidate detection unit 302 from the image data 400_0. The face candidate detection unit 302 detects the face candidate data as a rectangular area from the image data 400_0, and assigns an identifier (ID) to each of the detected face candidate data. Further, the face candidate detection unit 302 has a position (coordinates of the upper left vertex of the rectangular area), a height (length of the rectangular area in the y direction), and a width (length of the rectangular area in the x direction) where each face candidate data exists. ) Is extracted. Further, the face candidate detection unit 302 calculates the number of detected face candidate data (number of face candidate detection cases).

In the example of FIG. 4 (c), "F001" is assigned to the face candidate data 410_1 as an identifier (ID), and the coordinates = (x ₁ , y ₁ ), the height = h ₁ , and the width = w ₁ are extracted. It shows that. In the case of the image data 400_0, the face candidate detection unit 302 has detected m _{0 face candidate data from the face candidate data 410_1 to 410_m 0.}

Of the face candidate data shown in FIG. 4C, the face candidate data (for example, face candidate data 410_3, 410_4) in which the student's face is not included in the rectangular area is an area that is not the student's face. Is mistakenly detected as the student's face.

The face candidate detection unit 302 associates the information (coordinates, height, width) extracted for each detected face candidate data with the identifier (ID) of each face candidate data, and analyzes result storage unit 132 as detection result information. Store in. Further, the face candidate detection unit 302 stores the number of face candidate data (number of face candidate detection cases) of each detected face candidate data in the analysis result storage unit 132 as detection result information.

FIG. 5 is a diagram showing an example of detection result information. As shown in FIG. 5, the detection result information 500 has "identifier (ID)", "horizontal origin (x)", "vertical origin (y)", "width (w)", and "high" as information items. (H) ”,“ Number of face candidate detections ”is included.

The "identifier (ID)" stores the identifier given to the face candidate data. The x-coordinate of the face candidate data (x-coordinate of the upper left vertex of the rectangular area) is stored in the "horizontal starting point (x)". The y-coordinate of the face candidate data (y-coordinate of the upper left vertex of the rectangular area) is stored in the "vertical starting point (y)".

In the "width (w)", the width of the face candidate data (the length of the rectangular area in the x direction) is stored. In the "height (h)", the height of the face candidate data (the length of the rectangular area in the y direction) is stored. The number of detected face candidate data is stored in the "number of detected face candidates".

In FIG. 5, the extracted information for the detected face candidate data in the image data 400_0 of time t _0, which shows a state stored as the detection result information 500. However, the detection result information stored in the face candidate detection unit 302 is not limited to the detection result information 500 shown in FIG. Face candidate detection section 302, for the information extracted for the face candidate data detected respectively in the image data 400_1～ time _{t n} image data 400_M ₀ of time _{t 1,} as well as the detection result information, sequentially, the analysis result storage It shall be stored in the unit 132.

(2) Specific Example of Processing of First Narrowing Section FIG. 6 is a diagram showing a specific example of processing of the first narrowing section. As described above, the first narrowing-down unit 303 deletes the extracted information about the face candidate data determined not to satisfy the predetermined condition from the detection result information together with the identifier (ID).

Specifically, the first narrowing-down unit 303 holds in advance the upper limit value and the lower limit value of the width (w) and the upper limit value and the lower limit value of the height (h), and the width (w) of each face candidate data. ), Height (h) is determined whether or not it is included in the range of the upper limit value to the lower limit value of each.

When it is determined that at least one of the width (w) and the height is not included in the range of the upper limit value to the lower limit value, the first narrowing-down unit 303 determines that the predetermined condition is not satisfied. .. In addition, the first narrowing-down unit 303 deletes the information extracted about the face candidate data from the detection result information together with the identifier (ID).

In the detection result information 500 of FIG. 6A, the width (w) and the height (h) stored in the hatched area are the width (w) and the height (h), respectively, by the first narrowing-down unit 303. It indicates that it was determined that the value was not included in the range from the upper limit value to the lower limit value of.

In this case, the first narrowing-down unit 303 deletes the information extracted from the face candidate data of the identifier (ID) = "F003" and "F004" from the detection result information 500 together with the identifier (ID).

FIG. 6B shows the detected detection result information 600 after the deletion, in which the information extracted from the face candidate data determined not to satisfy the predetermined condition is deleted by the first narrowing-down unit 303. In the example of FIG. 6B, at least the information extracted for the face candidate data of the identifier (ID) = "F003" and "F004" is deleted from the detection result information 600 together with the identifier. Further, in the example of FIG. 6B, the information extracted from the face candidate data determined not to satisfy the predetermined condition is deleted from the detection result information 600, so that the “number of face candidate detections” starts from m _0. It has changed to _{m 0'.}

(3) Specific Example of Processing of Second Narrowing Section FIG. 7 is a diagram showing a specific example of processing of the second narrowing section. As described above, the second narrowing-down unit 304 uses the remaining face candidate data after the information extracted for the face candidate data determined not to satisfy the predetermined condition is deleted, at the position where each face candidate data exists. Based on, it is classified into each predetermined image area. Further, the second narrowing-down unit 304 totals the number of face candidate data classified into each image area and subtracts the number of face candidate data classified into a predetermined number or less of the image area to detect the number of faces. Is calculated.

FIG. 7A is the detection result information 600 processed by the second narrowing-down unit 304. Further, in FIG. 7B, the image data 400_0 is divided into a plurality of image regions, and each face candidate data included in the detection result information 600 is classified into one of the image regions based on the position, and the image data is divided into the respective image regions. The aggregation result information 700 which aggregated the number of the classified face candidate data is shown.

The example of FIG. 7B shows that the image data 400_0 is divided into 12 predetermined image regions. Further, examples of FIG. 7 (b), twelve of the image area, the image area specified by the x-coordinate = _{0 to X} 1, y-coordinate = 0~Y _{1, P} 0 11 pieces of face candidate data Indicates that has been classified. In the example of FIG. 7B, the face candidate data (identifier (ID) = "F001") included in the detection result information 600 is specified by _{x-coordinate = 0 to X 1} and y-coordinate = 0 _{to Y 1.} It is classified into the image area to be used (see arrow 701).

Further, the example of FIG. 7B shows _{that P 0} 12 face candidate data were classified into the image area specified by _{the x-coordinate = 0 to X 1} and the y-coordinate = Y _{1 to Y 2.} ing. In the example of FIG. 7B, the face candidate data (identifier (ID) = "F002") included in the detection result information 600 is based on x-coordinate = 0 to X ₁ , y-coordinate = Y _{1 to Y 2.} It is classified into the specified image area (see arrow 702).

Hereinafter, similarly, the second narrowing-down unit 304 classifies all the face candidate data included in the detection result information 600 into one of the image areas based on the position where the face candidate data exists, and sets each image area. The number of classified face candidate data is totaled (P ₀ 11 _{to P 0} 34).

FIG. 7C shows how the second narrowing-down unit 304 determines whether or not the number of face candidate data aggregated for each image area is equal to or less than a predetermined number. In the case of FIG. 7 (c), the number of face candidate data (P _{0 24) classified into the image area specified by x-coordinate = X 1 to} X ₂ and y-coordinate = Y _{3 to Y 4} is a predetermined number. It is as follows.

In this case, the second narrowing-down unit 304 determines the number of face candidate data (P ₀ 24) _{classified in the image area specified by the x-coordinate = X 1 to} X ₂ and the y coordinate = Y _{3 to Y 4.} It is subtracted from the face candidate detection number _{(= "m '0")} , and calculates a face detection count _{(C 0).}

Total result information 710 in FIG. 7 (c), the second narrowing portion 304, a face detection number of the image data 400_0 _{(C 0),} which indicates that the calculated "m _'0 -P ₀ 24" ..

(4) Specific Example of Processing of Detection Number Time Series Data Generation Unit FIG. 8 is a diagram showing a specific example of processing of the detection number time series data generation unit. As described above, the detection number time series data generation unit 305 associates the number of face detection cases calculated for each image data with the time information by the second narrowing down unit 304.

In FIG. 8A, the detection number time series data generation unit 305 associates the face detection cases C _{0 to} C _n calculated for each image data 400_0 to 400_n by the second narrowing-down unit 304 with the time information. It shows how it looks. Further, FIG. 8B is a graph showing time-series data of the number of face detection cases. In FIG. 8B, the horizontal axis _{represents the elapsed time from time t 0} , and the vertical axis represents the number of face detections in each elapsed time.

The detected number time series data generation unit 305 stores the generated time series data in the analysis result storage unit 132.

(5) Specific Example of Processing of Change Number Time Series Data Generation Unit FIG. 9 is a first diagram showing a specific example of processing of the change number time series data generation unit. As described above, the change number time series data generation unit 306 calculates the difference (absolute value) for each image area from the aggregated result information of the image data one frame before for the aggregated result information of each image data. Further, the change number time series data generation unit 306 calculates the change number by totaling the difference (absolute value) for each image area.

9 (a) and 9 (b) show the aggregated result information 710 of the image data 400_1 and the aggregated result information 910 of the image data 400_1, respectively. Further, FIG. 9C shows the difference information 920 for the image data 400_1 calculated based on the aggregation result information 710 and the aggregation result information 910. In the example of FIG. 9C, it is shown that _{ΔP 0_1} 11 was calculated as the difference (absolute value) for the image region specified by _{the x coordinate = 0 to X 1} and the y coordinate = 0 _{to Y 1.} ..

In addition, ΔP _{0_1} 11 is the number of face candidate data aggregated in the image area in the aggregation result information 710 (P ₀₁₁ ) and the number of face candidate data aggregated in the image area in the aggregation result information 910 (P). obtained by calculating the difference (absolute value) between ₁ 11).

Hereinafter, similarly, change the number time series data generating unit 306, by calculating the difference (absolute value) for each image area, as shown in FIG. 9 (c), to calculate the ΔP _{0_1} 11~ΔP _{0_1} 34 , The difference information 920 is stored in the analysis result storage unit 132. The change number time series data generating unit 306, by aggregating ΔP _{0_1} 11~ΔP _{0_1} 34 calculated, calculates the change number of the image data 400_1 and _{(D 1),} the analysis result storage unit as the difference information 920 132 Store in.

FIG. 10 is a second diagram showing a specific example of processing of the change number time series data generation unit. As described above, the change number time series data generation unit 306 associates the change number (D _{1 to} D _{n ) calculated for each image data (excluding the image data at time t 0} ) with the time information.

_{FIG. 10A shows how the number of changes D 1 to} D _n calculated for each image data 400_1 to 400_n is associated with the time information. Further, FIG. 10B is a graph showing time-series data of the number of changes. The horizontal axis in FIG. 10 (b) represents the time elapsed from the time t _1, the vertical axis represents the change number at each elapsed time.

The change number time series data generation unit 306 stores the generated time series data in the analysis result storage unit 132.

<Flow of image processing>
Next, the flow of image processing by the image processing unit 131 of the image processing device 130 will be described. 11 and 12 are first and second flowcharts of image processing by the image processing apparatus. The image processing program is started by the administrator of the image processing apparatus 130, and the moving image data to be processed is specified, so that the image processing shown in FIGS. 11 and 12 is executed.

In step S1101, the moving image data acquisition unit 301 acquires the designated moving image data from the moving image data storage device 120.

In step S1102, the face candidate detection unit 302 inputs “0” to the counter i that counts the image data in frame units included in the moving image data.

In step S1103, the face candidate detection unit 302 reads out the i-th image data from the moving image data.

In step S1104, the face candidate detection unit 302 detects face candidate data for the i-th image data, and stores the extracted information about the detected face candidate data in the analysis result storage unit 132 as detection result information. Further, the number of detected face candidate data (number of face candidate detection cases) is stored in the analysis result storage unit 132 as detection result information.

In step S1105, the first narrowing-down unit 303 inputs “1” to the counter j that counts the face candidate data detected from the i-th image data.

In step S1106, the first narrowing-down unit 303 determines whether or not the size (width, height) of the j-th face candidate data satisfies a predetermined condition. If it is determined in step S1106 that the predetermined condition is satisfied (Yes in step S1106), the process proceeds to step S1108.

On the other hand, if it is determined in step S1106 that the predetermined condition is not satisfied (No in step S1106), the process proceeds to step S1107. In step S1107, the first narrowing-down unit 303 deletes the information extracted from the j-th face candidate data from the detection result information and changes the number of face candidate detection cases.

In step S1108, the first narrowing-down unit 303 determines whether or not the determination has been performed for all the face candidate data stored in the detection result information. If it is determined in step S1108 that there is face candidate data that has not been determined (No in step S1108), the process proceeds to step S1109.

In step S1109, the first narrowing-down unit 303 increments the counter j and returns to step S1106.

On the other hand, if it is determined in step S1108 that all the face candidate data have been determined (yes in step S1108), the process proceeds to step S1201 in FIG.

In step S1201, the second narrowing-down unit 304 classifies the face candidate data included in the detection result information into each image area predetermined based on the position, and totals the number of face candidate data in each image area. Further, the second narrowing-down unit 304 stores the aggregation result of the i-th image data as the aggregation result information in the analysis result storage unit 132.

In step S1202, the second narrowing-down unit 304 determines an image area in which the number of face candidate data is a predetermined number or less in the aggregation result information.

In step S1203, the second narrowing-down unit 304 subtracts the number of face candidate data included in the image region of a predetermined number or less from the number of face candidate detection cases, and calculates the number of face detection cases in the i-th image data. Further, the second narrowing-down unit 304 stores the calculated number of face detections in the analysis result storage unit 132 as the aggregated result information for the i-th image data.

In step S1204, the change number time series data generation unit 306 reads the aggregation result information for the (i-1) th image data from the analysis result storage unit 132.

In step S1205, the change number time series data generation unit 306 uses the aggregated result information for the (i-1) th image data and the aggregated result information for the i-th image data to generate the i-th image data. Calculate the number of changes in. Further, the change number time series data generation unit 306 stores the calculated change number in the analysis result storage unit 132 as the difference information for the i-th image data.

In step S1206, the face candidate detection unit 302 determines whether or not processing has been performed on all the image data included in the moving image data. If it is determined in step S1206 that there is image data that has not been processed (if No in step S1206), the process proceeds to step S1207. In step S1207, the face candidate detection unit 302 increments the counter i and returns to step S1103 of FIG.

On the other hand, if it is determined in step S1206 that all the image data has been processed (yes in step S1206), the process proceeds to step S1208.

In step S1208, the detection number time series data generation unit 305 associates the time information of each image data with the face detection number calculated for each image data, and generates time series data of the number of face detection cases. In addition, the detected number time series data generation unit 305 stores the generated time series data of the number of face detection cases in the analysis result storage unit 132.

In step S1209, the change number time series data generation unit 306 generates time series data of the change number by associating the time information of each image data with the change number calculated for each image data. Further, the change number time series data generation unit 306 stores the generated time series data of the change number in the analysis result storage unit 132.

As is clear from the above description, the image processing device 130 in the first embodiment calculates the number of face detections and the number of changes in order to analyze the state of the person included in the image. As described above, in the image processing device 130 according to the first embodiment, when analyzing the state of the person, the face of the person is detected, while the detailed face information of the person (for example, the position of the eyebrows and the contour of the face). Etc.) is not calculated. Therefore, it is not necessary to install a high-performance image pickup device as an image processing system, and the number of image pickup devices installed can be reduced.

That is, a simple image processing system is sufficient, and it is possible to efficiently analyze the state of a person included in an image.

[Second Embodiment]
In the first embodiment, the case where the calculated number of face detections and the number of changes for each image data is associated with the time information has been described, but the data associated with the time information is not limited to the number of face detections and the number of changes.

For example, factors that affect the student's condition may be associated with the time information. This makes it possible to analyze the factors that affect the student's condition.

FIG. 13 is a diagram showing another example of time-series data of the number of face detection cases. Of these, FIG. 13A shows how the lecture content is associated with the time information in addition to the number of face detection cases. According to FIG. 13A, it is possible to analyze whether or not there is a correlation between the state of the students and the content of the lecture. In addition, when there is a correlation, it is possible to specify which lecture content is the lecture content that enhances the concentration of the students.

FIG. 13B shows how the size of the characters written on the board by the instructor or the like is associated with the time information in addition to the number of face detection cases. According to FIG. 13B, it is possible to analyze whether or not there is a correlation between the state of the student and the size of the characters written on the board by the instructor or the like. In addition, when there is a correlation, it is possible to specify an appropriate character size that enhances the concentration of students.

FIG. 13C shows how the standing position of the instructor or the like is associated with the time information in addition to the number of face detection cases. According to FIG. 13 (c), it is possible to analyze whether or not there is a correlation between the state of the student and the standing position of the instructor or the like. In addition, if there is a correlation, it is possible to identify an appropriate standing position that enhances the concentration of students.

[Other Embodiments]
In each of the above embodiments, the method of detecting the face candidate data by the face candidate detection unit 302 is not particularly mentioned, but any detection method can be used as the method of detecting the face candidate data by the face candidate detection unit 302.

Further, in each of the above embodiments, it has been described that the first narrowing-down unit 303 holds one type of upper limit value and lower limit value of the width (w) and one type of upper limit value and lower limit value of the height (h). However, the upper limit value and the lower limit value held by the first narrowing-down unit 303 are not limited to one type. For example, a plurality of types of upper limit values and lower limit values are held according to the distance from the installation position of the image pickup device 110 (position on the image data), and the upper limit value and lower limit value differ depending on the position where the face candidate data exists. The value may be read out for determination.

Further, in each of the above embodiments, it has been described that the first narrowing-down portion 303 holds an upper limit value and a lower limit value for each of the width (w) and the height (h), respectively. However, the upper limit value and the lower limit value held by the first narrowing-down unit 303 are not limited to this. For example, the upper limit value and the lower limit value of the area of width (w) × height (h) may be held.

Further, in each of the above embodiments, it has been described that the time-series data of the number of face detections and the time-series data of the number of changes are generated by associating with the time information of each image data. That is, it has been described that the time series data is generated so that the plot period of the number of face detection cases and the number of changes is equal to the frame period (for example, 300 msec) of each image data.

However, it is not necessary to generate time-series data so that the plot period of the number of face detections and the number of changes is equal to the frame period of each image data, and the time-series data is generated so as to be longer than the frame period of each image data. It may be generated (eg, 1 second). Further, the plot cycle of the number of face detection cases and the number of changes may be changed depending on, for example, the lecture format (whether the lecture format is centered on board writing or the lecture format is centered on lectures).

Further, in each of the above embodiments, the image area used by the second narrowing-down unit 304 for classifying the face candidate data has been described as being divided into a predetermined shape and size. It may be divided into shapes and sizes according to the structure of the space to be created.

If the size of the image area is too large, the number of students who changed from the state where the face is raised to the state where the face is lowered and the number of students who changed from the state where the face is lowered to the state where the face is raised The number of changes will be offset, and the accuracy of calculating the number of changes will decrease. Therefore, the image area is required to be divided into an appropriate size.

Further, in each of the above embodiments, the number of changes is calculated by aggregating the differences (absolute values) calculated by the second narrowing-down unit 304 in each image area, and the number of changes time series data is obtained based on the number of changes. Described as being generated. However, the number of changes calculated by the second narrowing-down unit 304 is not limited to this.

For example, the second narrowing-down unit 304 may calculate the number of changes by aggregating the values having a positive sign among the differences calculated in each image area, and generate time-series data on the number of changes. This makes it possible to calculate time-series data on the number of students who have changed from a face-down state to a face-up state.

Alternatively, the second narrowing-down unit 304 may calculate the number of changes by aggregating the values having a negative sign among the differences calculated in each image area, and generate time-series data on the number of changes. This makes it possible to calculate time-series data on the number of students who have changed from a raised face to a lowered face.

Further, in each of the above embodiments, it has been described that the second narrowing-down unit 304 performs the processing after the first narrowing-down unit 303 performs the processing, but the first narrowing-down unit 303 and the second narrowing-down unit 304 perform the processing. The order of processing may be reversed.

Further, in each of the above embodiments, the image processing device 130 has been described as being connected to the moving image data storage device 120 via the network 140, but when a portable terminal is used as the image processing device 130, the moving image It may be directly connected to the data storage device 120. Alternatively, a detachable recording medium is attached to the image pickup apparatus 110 to store the moving image data, and the recording medium in which the moving image data is stored is attached to the image processing apparatus 130, whereby the image processing apparatus 130 is attached to the moving image. Data may be acquired.

Further, in each of the above embodiments, the case where the image processing system 100 includes a set of the image pickup device 110 and the moving image data storage device 120 has been described, but the image pickup device 110 and the moving image data storage device 120 have image processing. A plurality of sets may be included in the system 100.

In addition, in the disclosed technology, a form as described in the appendix described below can be considered.
(Appendix 1)
The position where the face exists is extracted from each of the multiple images obtained by shooting the same shooting range at different timings.
An image area is specified for each of the plurality of images based on the extracted position where the face exists.
The result according to the presence or absence of the face in the specified image area is output in association with the information indicating the timing at which each of the plurality of images was taken.
An image processing program characterized by causing a computer to perform processing.
(Appendix 2)
The image processing program according to Appendix 1, wherein when outputting a result according to the presence or absence of the face, the number of detected faces in the specified image region is output.
(Appendix 3)
When outputting the result according to the presence / absence of the face, the presence / absence of the face in the specified image area is different from the presence / absence of the face in the image area of the image one frame before. The image processing program described in.
(Appendix 4)
When outputting the result according to the presence or absence of the face, the feature is that the number of faces detected in the image of the frame to be processed is output without detecting the face in the image one frame before in the specified image area. The image processing program according to Appendix 3.
(Appendix 5)
When outputting the result according to the presence or absence of the face, in the specified image area, the face is detected in the image one frame before, and the number in which the face is not detected in the image of the frame to be processed is output. The image processing program according to Appendix 3.
(Appendix 6)
The position where the face exists is extracted from each of the multiple images obtained by shooting the same shooting range at different timings.
An image area is specified for each of the plurality of images based on the extracted position where the face exists.
The result according to the presence or absence of the face in the specified image area is output in association with the information indicating the timing at which each of the plurality of images was taken.
An image processing method characterized in that processing is performed by a computer.
(Appendix 7)
An extraction unit that extracts the position where the face exists from each of the multiple images obtained by shooting the same shooting range at different timings,
Based on the extracted position where the face exists, a specific portion for specifying an image area for each of the plurality of images, and a specific portion for specifying the image area,
An image processing apparatus including an output unit that outputs a result according to the presence or absence of a face in the specified image area in association with information indicating the timing at which each of the plurality of images is taken.

The present invention is not limited to the configurations shown here, such as combinations with other elements in the configurations and the like described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form thereof.

100: Image processing system 110: Imaging device 120: Moving image data storage device 130: Image processing device 131: Image processing unit 132: Analysis result storage unit 301: Moving image data acquisition unit 302: Face candidate detection unit 303: First Filtering unit 304: Second filtering unit 305: Number of detected cases Time-series data generation unit 306: Number of changes Time-series data generation unit 307: Display control unit 500, 600: Detection result information 700, 710, 910: Aggregation result information 920: Difference information

Claims (7)

The position where the face of the lecturer's face exists is extracted from each of the plurality of images obtained by shooting the lectures held in the same shooting range at different timings.
An image area is specified for each of the plurality of images based on the extracted position where the face exists.
The result according to the presence or absence of the face in the specified image area is associated with the information indicating the timing at which each of the plurality of images was taken, and output at a plot cycle according to the format of the lecture.
An image processing program characterized by causing a computer to perform processing. The image processing program according to claim 1, wherein when outputting a result according to the presence or absence of the face, the number of detected faces in the specified image region is output. When outputting the result according to the presence / absence of the face, the claim is characterized in that the presence / absence of a face in the specified image region is different from the presence / absence of a face in the image region of the image one plot cycle before. Item 1. The image processing program according to item 1. When outputting the result according to the presence or absence of the face, in the specified image region, the number of detected faces in the image of the frame to be processed is output without detecting the face in the image one plot cycle before. The image processing program according to claim 3, which is characterized. When outputting the result according to the presence or absence of the face, in the specified image region, the face is detected in the image one plot cycle before, and the number in which the face is not detected in the image of the frame to be processed is output. The image processing program according to claim 3. The position where the face of the lecturer's face exists is extracted from each of the plurality of images obtained by shooting the lectures held in the same shooting range at different timings.
An image area is specified for each of the plurality of images based on the extracted position where the face exists.
The result according to the presence or absence of the face in the specified image area is associated with the information indicating the timing at which each of the plurality of images was taken, and is output at a plot cycle according to the format of the lecture.
An image processing method characterized in that processing is performed by a computer. An extraction unit that extracts the position where the face of the lecturer's face exists from each of the multiple images obtained by shooting the lectures held in the same shooting range at different timings.
Based on the extracted position where the face exists, a specific portion for specifying an image area for each of the plurality of images, and a specific portion for specifying the image area,
It has an output unit that associates the results according to the presence or absence of faces in the specified image area with information indicating the timing at which each of the plurality of images was taken, and outputs the results at a plot cycle according to the format of the lecture. An image processing device characterized by.

Images