JP7424940B2 - Person tracking device, person tracking system, person tracking method and person tracking program - Google Patents

Description

One aspect of the present invention relates to a person tracking device, a person tracking system, a person tracking method, and a person tracking program.

Conventionally, various techniques have been proposed for extracting a person's face from a surveillance image obtained by a surveillance camera and identifying the person based on the facial features. Furthermore, in recent years, a technique has been proposed that improves the accuracy of person identification by identifying a person by combining facial feature amounts and whole body feature amounts (for example, see Patent Document 1).

JP2019-023785A

Conventional person identification technology detects the visit of suspicious persons or regular customers when facial images of visitors can be reliably captured, such as with surveillance cameras installed at the entrances of buildings or stores. This is a very effective technique.

On the other hand, in recent years, there has been a strong demand for measures against infectious diseases such as influenza and COVID-19 (new coronavirus infection). Therefore, there is a need for technology to detect the visit of a person suspected of having such an infectious disease, and further to track the person's movement within a facility.

This invention was made in view of the above circumstances, and its purpose is to detect and track people who have prescribed symptoms corresponding to infectious diseases from surveillance images obtained by surveillance cameras. It's about doing.

In order to achieve the above object, one aspect of the person tracking device according to the present invention collects a plurality of types of state information related to the health condition of a person included in each of monitoring images periodically acquired by each of a plurality of monitoring cameras. , the plurality of types of status information detected by the information extracting unit are acquired from the information extracting unit detecting from each of the monitoring images, and based on the acquired plurality of types of status information, each of the monitoring images is a determination unit that determines whether the included person is a specific person who corresponds to a prescribed symptom; and a person extracted from the surveillance images of each of the plurality of surveillance cameras based on the determination result of the determination unit. a registration unit that registers an image of the specific person corresponding to the specified symptoms among the images of the specified person as a tracking target person image, and from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras, a detection unit that detects an image of a person similar to the tracking target person image registered by the registration unit; and a tracking output screen for presenting an image of a person similar to the tracking target person image detected by the detection unit. and a tracking output screen generation unit that generates a tracking output screen.

One aspect of the person tracking system according to the present invention includes an aspect of the person tracking device according to the present invention, the plurality of surveillance cameras, the surveillance terminal operated by a supervisor, and an analysis section including the information extraction section. It is designed to include the following.

One aspect of the person tracking method according to the present invention is a person tracking method for tracking a person to be tracked from surveillance images obtained by a plurality of surveillance cameras, wherein a computer tracks a person to be tracked from surveillance images obtained by a plurality of surveillance cameras. an information extraction unit that detects a plurality of types of status information related to the health status of a person included in each of the regularly acquired monitoring images from each of the monitoring images, the plurality of types of status detected by the information extraction unit; information, and determine whether the person included in each of the monitoring images is a specific person corresponding to a prescribed symptom based on the acquired plurality of types of status information, and based on the result of the determination. Based on this, among the images of the person extracted from the monitoring images of each of the plurality of monitoring cameras, an image of the specific person corresponding to the prescribed symptoms is registered in a memory as a tracking target person image, and An image of a person similar to the registered image of the tracked person is detected from each of the monitoring images periodically acquired by each of the monitoring cameras of the camera, and an image of a person similar to the detected image of the tracked person is detected. A tracking output screen for presenting images is generated.

One aspect of the person tracking program according to the present invention is a program that causes a computer to operate as each part of one aspect of the person tracking device according to the present invention.

According to each aspect of the present invention, it becomes possible to detect and track a person who has a prescribed symptom from a surveillance image obtained by a surveillance camera.

FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of the video analysis function section shown in FIG. 1. FIG. 3 is a block diagram showing the software configuration of the Web server shown in FIG. 1 as a person tracking device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the storage contents of the past search data storage section shown in FIG. 3. FIG. 5 is a diagram showing an example of the storage contents of the detection history data storage section shown in FIG. 3. FIG. 6 is a diagram showing an example of the storage contents of the management data storage section shown in FIG. 3. FIG. 7 is a diagram showing an example of the contents written in the camera information table storage section shown in FIG. 3. FIG. 8 is a diagram showing an example of the contents written in the past search data table storage section shown in FIG. 3. FIG. 9 is a diagram showing an example of the contents written in the detection history table storage section shown in FIG. 3. FIG. 10 is a diagram showing an example of the contents written in the management table storage section shown in FIG. 3. FIG. 11 is a diagram showing an example of the contents written in the tracking table storage section shown in FIG. 3. FIG. 12 is a block diagram showing the hardware configuration of the Web server in FIG. 3. FIG. 13 is a flowchart illustrating the processing procedure of detection result determination processing by the detection/tracking result determination function unit shown in FIG. FIG. 14A is a flowchart showing the first part of the processing procedure of the detection alert output processing by the detection alert output function unit shown in FIG. 3. FIG. FIG. 14B is a flowchart showing the second part of the processing procedure of the detection alert output processing by the detection alert output function unit shown in FIG. FIG. 15 is a diagram showing an example of a detection alert screen displayed on the web browser of the monitoring terminal shown in FIG. FIG. 16 is a diagram showing another example of the detection alert screen displayed on the web browser of the monitoring terminal shown in FIG. 1. FIG. 17 is a flowchart showing the processing procedure of image registration processing by the image registration function section shown in FIG. FIG. 18 is a flowchart showing the processing procedure of tracking processing by the monitoring/tracking execution function section shown in FIG. FIG. 19 is a flowchart showing the processing procedure of the tracking result judgment process by the detection/tracking result judgment function section shown in FIG. FIG. 20A is a flowchart showing a first part of the processing procedure of tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 20B is a flowchart showing the second part of the processing procedure for tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 20C is a flowchart showing the third part of the processing procedure for tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 21 is a diagram showing an example of a tracking alert screen displayed on the web browser of the monitoring terminal shown in FIG. FIG. 22 is a diagram showing another example of the tracking alert screen displayed on the web browser of the monitoring terminal shown in FIG. FIG. 23 is a flowchart showing a part of the processing procedure of the detection alert output processing by the detection alert output function unit in the modified example of the person tracking system.

Embodiments of the present invention will be described below with reference to the drawings.

[One embodiment]
(1) Configuration (1-1) Overall Configuration FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system 1 according to an embodiment of the present invention.

The person tracking system 1 is a system that tracks a specific person in a large facility with a wide monitoring range, such as a building with multiple floors such as an office building or a department store, or a commercial facility with multiple stores.

This person tracking system 1 includes a plurality of surveillance cameras 10, a video analysis function unit 20 provided corresponding to each surveillance camera 10, and a web server 30 that functions as a person tracking device according to an embodiment of the present invention. It includes a server device SV and a monitoring terminal 40. Each surveillance camera 10, server device SV, and surveillance terminal 40 are connected via a network NET.

The network NET is a local area network, such as a wireless LAN (Local Area Network) or a wired LAN. The network NET may be a wide area network such as the Internet.

The plurality of surveillance cameras 10 are network cameras that are distributed within a large facility so that their photographic ranges do not overlap or partially overlap. The surveillance camera 10 may be a video camera that photographs a moving image within a photographing range, and can acquire each frame image forming the moving image as a monitoring image of the photographing range. Furthermore, the surveillance camera 10 may be a still camera that periodically acquires surveillance images by photographing still images at fixed time intervals.

Each of the plurality of video analysis functional units 20 arranged in the server device SV has various analysis functions for monitoring images acquired from the corresponding monitoring camera 10. Details of the analysis function provided by this video analysis function section 20 will be described later. Note that the video analysis function unit 20 is not placed inside the server device SV in this way, but is placed independently in a dedicated computing device, cloud, etc., and is connected to the corresponding surveillance camera 10 via the network NET. It is also possible to exchange data with the server device SV. Further, the video analysis function unit 20 can be placed inside the surveillance camera 10 if it has a calculation function. In this embodiment, the surveillance camera 10 and the video analysis function section 20 are configured in a one-to-one correspondence, but one video analysis function section 20 may be assigned to a plurality of surveillance cameras 10. Also good.

The Web server 30 of the server device SV is constituted by a server computer, and includes a file server 31, a database (hereinafter abbreviated as DB) server 32, a Web application 35, a search person detection result storage function section 33, and a function section. , a detection/tracking result determination function section 34. Details of each of these functional units will be described later.

The monitoring terminal 40 is configured by a computer such as a personal computer, and can execute a program of a web browser 41. Using the web browser 41, a supervisor such as a security guard who operates the monitoring terminal 40 can view the person tracking results provided by the web application 35 of the web server 30. Although FIG. 1 shows only one monitoring terminal 40, the person tracking system 1 may include a plurality of monitoring terminals 40.

Further, the server device SV may have the function of the monitoring terminal 40.

(1-2) Video analysis function section 20
FIG. 2 is a block diagram showing an example of the configuration of the video analysis function section 20 shown in FIG. 1. As shown in FIG. The video analysis function section 20 includes an image acquisition module 21, a human detection information extraction function section 22, and a monitoring/tracking execution function section 23.

The image acquisition module 21 acquires frame images transmitted from the surveillance camera 10 via the network NET according to a predetermined communication protocol such as RTSP (Real Time Streaming Protocol). The image acquisition module 21 transmits the acquired frame image to the Web server 30 via intra-server communication, Websocket, etc., and causes the file server 31 to accumulate it. Note that when the video analysis function section 20 is arranged independently from the server device SV, the image acquisition module 21 transmits the acquired frame image to the Web server 30 via the network NET. Further, the image acquisition module 21 sends the acquired frame image to the human detection information extraction function section 22.

The human detection information extraction function unit 22 extracts a full-body image of a person from the input image, extracts a whole-body feature amount and a face image of the person from this full-body image, and further extracts the features of the person from this face image. It functions as an extraction unit that extracts facial feature quantities. Specifically, the human detection information extraction function unit 22 includes a whole body detection module 221, a region tracking module 222, a whole body feature extraction module 223, a face detection module 224, a face feature extraction module 225, a body temperature detection module 226, and a pulse detection module. It includes a module 227 and a cough detection module 228.

The whole body detection module 221 is a whole body image extraction unit that detects the whole body of each person appearing in a frame image that is an input image and extracts a whole body image. The whole body detection module 221 can detect the whole body of a person, for example, based on a whole body image of the person trained in advance by machine learning such as deep learning. The whole body detection module 221 transmits the extracted whole body image to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function section 20 is arranged independently from the server device SV, the whole body detection module 221 transmits the extracted whole body image to the Web server 30 via the network NET. The whole body detection module 221 also sends the extracted whole body image to the whole body feature extraction module 223 , the face detection module 224 , and the body temperature detection module 226 . Further, the whole body detection module 221 sends the frame image and the extracted whole body image to the region tracking module 222 .

The region tracking module 222 detects the region position of the whole body image in the frame image input from the whole body detection module 221.

The whole body feature extraction module 223 is a whole body feature extraction unit that extracts whole body features from the input whole body image. The whole body feature quantity is an objective numerical representation of the body features of the person included in the whole body image. For example, the whole-body feature amount can be a numerical representation of attribute information about the person, such as body shape, height, gender, age, etc., estimated from the image. The whole body feature extraction module 223 can extract whole body features using a model learned in advance such as deep learning (for example, deep-person-reid). Further, the whole-body feature extraction module 223 can also calculate these whole-body features as local image features such as HOG (Histogram of Oriented Gradients) and SIFT (Scaled Invariance Feature Transform). The whole body feature amount extraction module 223 transmits the extracted whole body feature amount to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the whole body feature amount extraction module 223 transmits the extracted whole body feature amount to the Web server 30 via the network NET. Furthermore, the whole body feature amount extraction module 223 sends the extracted whole body feature amount to the monitoring/tracking execution function unit 23.

The face detection module 224 is a face image extraction unit that detects a person's face from the input whole body image and extracts a face image. The face detection module 224 can detect a person's face, for example, based on a person's face image trained in advance by machine learning such as deep learning. The face detection module 224 sends the extracted face image to a face feature extraction module 225 , a body temperature detection module 226 , a pulse detection module 227 , and a cough detection module 228 .

The facial feature extraction module 225 is a facial feature extraction unit that extracts facial features from the input facial image. The facial feature amount is an objective numerical representation of the facial features of a person included in the facial image. For example, similar to the whole body feature extraction module 223 that extracts whole body features, the facial feature extraction module 225 can extract facial features using a model learned in advance by deep learning or the like. Further, the face detection module 224 may calculate these facial feature amounts as local image feature amounts such as HOG or SIFT. The face detection module 224 transmits the extracted facial feature amount to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the facial feature amount extraction module 225 transmits the extracted facial feature amount to the Web server 30 via the network NET. Further, the face detection module 224 sends the extracted facial feature amount to the monitoring/tracking execution function unit 23.

The body temperature detection module 226 detects the body surface temperature of a person from the whole body image input from the whole body detection module 221 or the face image input from the face detection module 224. Various methods for detecting body surface temperature from images have been proposed in the past, and in this embodiment, the body surface temperature detection method to be used is not particularly limited. The body temperature detection module 226 transmits the detected body temperature to the web server via intra-server communication, websocket, or the like. Note that when the video analysis function section 20 is arranged independently from the server device SV, the body temperature detection module 226 transmits the detected body temperature to the Web server 30 via the network NET.

The pulse detection module 227 detects a person's pulse rate from the face image input from the face detection module 224. Techniques for detecting pulse rate from images are proposed, for example, in References 1 to 3 below.
[Reference 1] Issei Ohara, 4 others, “Pulse wave information extraction from video” [Online], [Retrieved June 19, 2020], Internet <URL: https://www.sairct.idac .tohoku.ac.jp/wp-content/uploads/2016/07/160728_c2_yoshizawa_DLpre2.pdf＞
[Reference 2] Masahide Abe and 2 others, “Measurement of pulse rate and respiration rate using non-contact sensor” [Online], [Retrieved June 19, 2020], Internet <URL: http:/ /www.mk.ecei.tohoku.ac.jp/papers/data/F05760013.pdf＞
[Reference 3] “Development of technology to measure pulse in real time from facial images” [Online], [Retrieved June 19, 2020], Internet <URL: https://pr.fujitsu.com/ jp/news/2013/03/18.html＞
In this embodiment, the pulse rate detection method used is not particularly limited. The pulse detection module 227 transmits the detected pulse rate to the Web server via intra-server communication, Websocket, or the like. Note that when the video analysis function section 20 is arranged independently from the server device SV, the pulse detection module 227 transmits the detected pulse rate to the Web server 30 via the network NET.

The cough detection module 228 detects cough symptoms of a person from the face image input from the face detection module 224. The cough detection module 228 detects whether or not the person is coughing and the degree of the cough based on, for example, the degree of blur in the facial image, changes in the face in a plurality of consecutive facial images, and the like. The cough detection module 228 can detect cough symptoms based on a person's face image trained in advance by machine learning such as deep learning, for example. The cough detection module 228 transmits cough symptoms indicating the degree of the detected cough symptoms to the web server via intra-server communication, websocket, or the like. Note that when the video analysis function section 20 is arranged independently from the server device SV, the cough detection module 228 transmits the detected cough symptoms to the Web server 30 via the network NET.

In addition, the monitoring/tracking execution function unit 23 calculates whole body image similarity, which is the similarity between the whole body features of the two input images, and facial image similarity, which is the similarity between the facial features of the two input images. functions as a similarity calculation unit. Specifically, the monitoring/tracking execution function unit 23 includes a whole body matching module 231 and a face matching module 232.

The whole body verification module 231 calculates the degree of similarity between the whole body feature input from the whole body feature extraction module 223 and the whole body feature of the person to be tracked instructed by the monitoring terminal 40 . For example, the whole body feature amount of the person to be tracked is extracted in advance by the whole body feature amount extraction module 223 and registered in the file server 31 of the Web server 30. The whole body verification module 231 transmits the calculated degree of similarity to the web server 30 via intra-server communication, websocket, or the like. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the whole body verification module 231 transmits the calculated similarity to the Web server 30 via the network NET.

The face matching module 232 calculates the degree of similarity between the facial feature amount input from the facial feature amount extraction module 225 and the facial feature amount of the person to be tracked instructed by the monitoring terminal 40 . The facial feature amount of the person to be tracked is, for example, extracted in advance by the facial feature amount extraction module 225 and registered in the file server 31 of the Web server 30. The face matching module 232 transmits the calculated similarity to the web server 30 via intra-server communication, websocket, or the like. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the face matching module 232 transmits the calculated similarity to the Web server 30 via the network NET.

(1-3) Web server 30
FIG. 3 is a block diagram showing the software configuration of the Web server 30. As described above, the Web server 30 includes a file server 31, a DB server 32, a search person detection result storage function section 33, a detection/tracking result determination function section 34, and a Web application 35 as functional sections.

The file server 31 stores various data files. The file server 31 can include a past search data storage section 311, a detection history data storage section 312, and a management data storage section 313.

The past search data storage unit 311 stores past search data that is data acquired by each of the plurality of video analysis function units 20. FIG. 4 is a diagram showing an example of past search data stored in this past search data storage section 311. The past search data can be stored in the past search data storage section 311 for each whole body image detected from the frame image by the whole body detection module 221 of each video analysis function section 20. The past search data includes a frame image 3111 acquired by the image acquisition module 21 of the video analysis function unit 20, a whole body image 3112 extracted by the whole body detection module 221, a whole body feature 3113 extracted by the whole body feature extraction module 223, and The facial feature amount 3114 extracted by the facial feature amount extraction module 225 can be included. The past search data may include body temperature 3115, pulse rate 3116, and cough symptoms 3117, which are state information detected by the whole body detection module 221 of the video analysis function unit 20. These frame images 3111, whole body image 3112, whole body features 3113, face features 3114, body temperature 3115, pulse rate 3116, and cough symptoms 3117 may be stored in the same path or have the same characters as part of the file name, for example. You can link them by adding columns.

The detection history data storage unit 312 stores detection history data that is data regarding each specific person detected as a tracking candidate who has a prescribed symptom such as influenza. FIG. 5 is a diagram showing an example of detection history data of a detected specific person stored in the detection history data storage section 312. The detection history data includes a detected face image 3121 which is a face image of a detected specific person, a facial feature amount 3122 corresponding to the detected facial image 3121, and a detected whole body image 3123 which is a whole body image of the detected specific person. , and a whole body feature amount 3124 corresponding to the detected whole body image 3123. These detected face image 3121, facial feature amount 3122, detected whole body image 3123, and whole body feature amount 3124 can be stored in the same path, or by adding the same character string to part of the file name, for example. Can be linked.

The information in the detection history data is stored in the detection history data storage unit when the web server 30 detects a tracking candidate who has a specified body temperature, pulse rate, or cough symptoms, for example, from the surveillance images of any of the surveillance cameras 10. 312. That is, the Web server 30 uses the detected whole-body image 3123, the detected whole-body feature quantity 3124, the whole-body image 3112, the whole-body feature quantity 3113, and the facial feature quantity 3114 acquired from the monitoring image and stored in the past search data storage unit 311. and can be stored in the detection history data storage unit 312 as a facial feature amount 3122. In addition, the Web server 30 transmits the whole body image 3112 acquired from the monitoring image and stored in the past search data storage section 311 to the face detection module 224 of the video analysis function section 20 via intra-server communication, Websocket, etc. By inputting , a face image can be acquired from the face detection module 224 and stored in the detection history data storage unit 312 as the detected face image 3121 . This one video analysis function unit 20 may be a video analysis function unit 20 corresponding to the surveillance camera 10 that acquired the surveillance image in which the tracking candidate was detected. If the face detection module 224 of the video analysis function unit 20 is buffering the extracted face image, the web server 30 detects the face image by requesting the face detection module 224 for the buffered face image. A face image 3121 may also be acquired.

The management data storage unit 313 stores management data that is data regarding each tracked person. FIG. 6 is a diagram showing an example of management data for each person to be tracked stored in the management data storage unit 313. Management data for each person to be tracked can be stored in the management data storage unit 313 according to a designation from the monitoring terminal 40. The management data includes a registered face image 3131 which is a face image of a designated person to be tracked, a facial feature quantity 3132 which is a feature extracted from the registered face image 3131, and a registered whole body image which is a whole body image of the person to be tracked. 3133 and a whole body feature amount 3134 that is a feature extracted from the registered whole body image 3133. These registered face image 3131, facial feature amount 3132, registered whole body image 3133, and whole body feature amount 3134 can be stored in the same path, or by adding the same character string to part of the file name, for example. Can be linked.

For example, when a tracking candidate is detected, the web server 30 stores the detected face image 3121 in the detection history data for the monitoring target stored in the detection history data storage unit 312 and the state information that was the source of the detection. is viewed by the supervisor using the web browser 41 of the monitoring terminal 40. The status information to be viewed here is the body temperature 3115, pulse rate 3116, and cough symptoms 3117 stored in the past search data storage unit 311, corrected by correction values for each surveillance camera 10 depending on the type, performance, etc. It is. Then, when the monitoring terminal 40 specifies that the person is to be tracked, the Web server 30 stores the detected face image 3121 in the detection history data as the registered face image 3131 in this management data. Furthermore, the Web server 30 can store the facial feature amount 3122 in the detection history data as the facial feature amount 3132 in this management data. Further, the web server 30 acquires the whole body image that is the source of the detected face image 3121 in the specified detection history data from the past search data stored in the past search data storage unit 311, and also acquires the whole body feature amount. are also acquired from the past search data and stored as the registered whole body image 3133 and whole body feature amount 3134 in this management data.

Alternatively, the web server 30 allows the web browser 41 of the monitoring terminal 40 to view the whole body images 3112, body temperature 3115, pulse rate 3116, and cough symptoms 3117 of a plurality of people stored in the past search data storage unit 311. , an arbitrary full-body image may be designated from the monitoring terminal 40, and that person may be designated as the designated tracking target. In this case, the results of correcting the body temperature 3115, pulse rate 3116, and cough symptoms 3117 using the corresponding correction values for the surveillance camera 10 may be viewed on the Web browser 41. Once the person to be tracked is specified, the Web server 30 inputs the specified full-body image 3112 to the face detection module 224 of one video analysis function section 20 via intra-server communication, Websocket, etc., thereby detecting the face. A face image can be acquired from the detection module 224 and stored in the management data storage unit 313 as a registered face image 3131. Further, in this case, as for the face feature amount 3132, the registered whole body image 3133, and the whole body feature amount 3134, those stored in the past search data storage section 311 may be acquired and stored.

The DB server 32 accumulates various data tables. The DB server 32 can include a camera information table storage section 321, a past search data table storage section 322, a detection history table storage section 323, a management table storage section 324, a tracking table storage section 325, and the like.

The camera information table storage unit 321 is provided corresponding to each of the plurality of surveillance cameras 10, and stores in advance a camera information table 3211 in which various information regarding the corresponding surveillance camera 10 is described. FIG. 7 is a diagram showing an example of the contents of the camera information table 3211 for each surveillance camera 10, which is stored in the camera information table storage section 321. The camera information table 3211 includes, for example, camera ID, camera name, aspect ratio, camera position Verification thresholds 1 to 3, etc. are described.

Here, the camera ID is identification information that uniquely identifies the corresponding surveillance camera 10. The camera name is the name of the corresponding surveillance camera 10. This can be a name associated with the installation position of the corresponding surveillance camera 10 in a large facility. The aspect ratio is the aspect ratio of the image acquired by the corresponding surveillance camera 10. Camera position X and camera position Y are XY coordinates within a large facility that indicate the installation position of the corresponding surveillance camera 10. The camera angle indicates the installation direction of the corresponding surveillance camera 10.

In addition, the body temperature correction coefficient, pulse rate correction coefficient, and cough symptom correction coefficient are used to correct the body temperature, pulse rate, and cough symptoms detected by the body temperature detection module 226, pulse detection module 227, and cough detection module 228 of the video analysis function unit 20. is the coefficient of These correction coefficients depend on the type, performance, installation conditions, etc. of the corresponding surveillance camera 10 itself, so they must be adjusted to appropriate values between the installation of the surveillance camera 10 and the start of actual operation. and is recorded in the camera information table 3211.

Further, the face matching thresholds 1 to 3 and the whole body matching thresholds 1 to 3 are thresholds for comparison with the degrees of similarity calculated by the whole body matching module 231 and the face matching module 232 of the video analysis function section 20. The face matching threshold 1 is a first threshold corresponding to the face image similarity for classifying whether the face image of the person extracted from the monitoring image of the corresponding surveillance camera 10 is the face image of the person to be tracked. be. If the face image similarity calculated by the face matching module 232 is greater than or equal to the first threshold, which is the face matching threshold 1, the web server 30 determines whether the face image of the person extracted from the monitoring image of the corresponding monitoring camera 10 is It can be determined that this is a face image of a person to be tracked. Face matching thresholds 2 and 3 correspond to face image similarity greater than face matching threshold 1, and when the extracted person's face image is the face image of the tracking target, the face image is divided into multiple stages. This is the second threshold for classification. That is, the magnitudes of these thresholds have a relationship of [face matching threshold 1<face matching threshold 2<face matching threshold 3]. Similarly, the whole body matching threshold 1 is the first whole body image similarity corresponding to the whole body image similarity for classifying whether the whole body image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the whole body image of the person to be tracked. is the threshold value. If the whole body image similarity calculated by the whole body matching module 231 is equal to or higher than the whole body matching threshold 1, the web server 30 determines that the whole body image of the person extracted from the monitoring image of the corresponding monitoring camera 10 is the whole body image of the tracking target person. It can be determined that Whole body matching thresholds 2 and 3 correspond to a whole body image similarity greater than whole body matching threshold 1, and when the extracted whole body image of a person is a whole body image of a person to be monitored or a person to be tracked, the whole body image is This is the second threshold for classifying into multiple stages. That is, the magnitudes of these threshold values have a relationship of [whole body verification threshold 1<whole body verification threshold 2<whole body verification threshold 3].

The value of each threshold is generally determined by the matching model used by the face matching module 232 and the matching model used by the whole body matching module 231 in the monitoring/tracking execution function section 23 of the video analysis function section 20 that calculates the degree of similarity. Each threshold value depends on the type, performance, installation conditions, etc. of the surveillance camera 10 itself, so the threshold value is adjusted to an appropriate value between the time the surveillance camera 10 is installed and the actual operation begins. It is written in the information table 3211. Note that, of course, the face matching threshold and the whole body matching threshold are not limited to three levels. That is, at least one second threshold value is sufficient.

The past search data table storage unit 322 includes a past search data table 3221 in which various data related to the past search data are described, corresponding to each of the past search data stored in the past search data storage unit 311. remember. When storing the past search data in the past search data storage section 311, the Web server 30 creates this past search data table 3221 and stores it in the past search data table storage section 322. FIG. 8 is a diagram showing an example of the contents of the past search data table 3221 stored in the past search data table storage section 322. The past search data table 3221 includes, for example, detection ID, detection date and time, detection camera ID, tracking ID, frame ID, detection coordinate information X, detection coordinate information Y, detection coordinate information Width, detection coordinate information Height, etc. be done.

Here, the detection ID is identification information assigned to each whole-body image 3112 in the frame image 3111 detected by the whole-body detection module 221 of the video analysis function section 20. The detection date and time is the date and time when the whole body detection module 221 detected the whole body image 3112. The detected camera ID is the camera ID of the surveillance camera 10 that acquired the frame image 3111. The tracking ID is identification information for linking the whole body images 3112 of the same person among multiple frame images. The frame ID is identification information for uniquely identifying the frame image 3111. Detected coordinate information For example, it shows the XY coordinates of the upper left corner in the frame image, and the image width and image height from there.

Note that the Web server 30 can assign the tracking ID of the past search data table 3221 in the following manner, for example. The web server 30 stores the region position of the whole body image provided from the human detection information extraction function unit 22 of the video analysis function unit 20 and the past information in the previous frame image of the surveillance camera 10 stored in the past search data storage unit 311. The identity of the person in the whole-body image is determined by comparing the area positions of the respective whole-body images in the search data table 3221 and taking into consideration overlapping areas. If the same person is not shown in the previous frame image, the web server 30 assigns a new tracking ID. Furthermore, if the same person appears in the previous frame image, the tracking ID is taken over from the past search data table 3221 corresponding to the whole body image 3112 of the person. Note that the Web server 30 may determine the identity of the person by comparing the whole-body feature amount instead of the position of the whole-body image, and determine whether or not to provide a new tracking ID.

The detection history table storage unit 323 stores a detection history table 3231 in which various data related to the detection history data are described, corresponding to each piece of detection history data stored in the detection history data storage unit 312. When storing detection history data in the detection history data storage section 312, the Web server 30 creates this detection history table 3231 and stores it in the detection history table storage section 323. FIG. 9 is a diagram showing an example of the contents of the detection history table 3231 stored in the detection history table storage section 323. The detection history table 3231 includes, for example, detection ID, detection date and time, detection camera ID, attention-seeking ID, corrected body temperature, corrected pulse rate, corrected cough symptoms, face left/right angle, face up/down angle, tracking ID, etc. is described.

Here, the detection ID, detection date and time, detection camera ID, and tracking ID are as described for the past search data table 3221. The attention-seeking person ID is identification information that is given when the detection history data is stored in the detection history data storage unit 312 and uniquely identifies the tracking candidate. The corrected body temperature, corrected pulse rate, and corrected cough symptoms are correction results based on correction coefficients written in the camera information table 3211 stored in the camera information table storage section 321. The face left/right angle and the face up/down angle indicate the direction of the face in the detected face image 3121 of the corresponding detection history data.

The management table storage unit 324 stores, corresponding to each piece of management data stored in the management data storage unit 313, a management table 3241 in which various data related to the management data are described. When storing management data in the management data storage section 313, the Web server creates this management table 3241 and stores it in the management table storage section 324. FIG. 10 is a diagram showing an example of the contents of the management table 3241 stored in the management table storage section 324. The management table 3241 includes, for example, attention-seeking ID, corrected body temperature, corrected pulse rate, corrected cough symptoms, query facial image path, facial feature file path, monitoring status, pinning flag, long stay flag, etc. will be described.

Here, the person requiring attention ID, corrected body temperature, corrected pulse rate, and corrected cough symptom are transferred from the detection history table 3231. The query face image path and face feature amount file path indicate the storage paths of the registered face image 3131 and face feature amount 3132 of management data in the management data storage unit 313. The monitoring status is attribute information other than the risk type that is arbitrarily set for the person to be tracked from the monitoring terminal 40. The pinning flag is a flag that is set to indicate when a person is designated as a tracking target. The long-term stay flag is a flag that is set when a person is not designated as a tracking target but stays at the facility for a specified period of time or longer. That is, when there is a person who is not a monitoring target but has been staying for a long time, the Web server 30 also stores the face image and facial feature amount of the person in the management data storage unit 313, and stores the face image and facial feature amount of the person in the management table storage unit 324. Management tables can be stored. This makes it possible to discover new suspicious persons and new customer candidates.

The tracking table storage unit 325 stores the whole body image or face image of the person to be tracked by the web server 30 based on the whole body image similarity or face image similarity calculated by the whole body matching module 231 or the face matching module 232 of the video analysis function unit 20. A tracking table is stored in which the degree of similarity of the person extracted from the monitoring image of any of the monitoring cameras 10 determined to be described is described. FIG. 11 is a diagram showing an example of the contents of the tracking table 3251 stored in the tracking table storage section 325. The tracking table 3251 includes a record consisting of a detection ID, a whole body image similarity degree, and a face image similarity degree for each person-of-care ID of the person to be tracked, and each record is a record that the web server 30 uses as a whole body image or a face image of the person to be tracked. It is added each time it is determined that it is. For the whole body image similarity degree and the face image similarity degree, the real values of the similarity calculated by the whole body matching module 231 and the face matching module 232 of the video analysis function unit 20 are stored, but as will be described later, tracking with the Web browser 41 is performed. In order to display the alert screen, it can be rewritten into a three-level whole body image similarity label ID and a three-level facial image similarity label ID.

The search person detection result storage function section 33 and the detection/tracking result determination function section 34 are provided as back-end functions of the Web server. The search person detection result storage function section 33 receives the output from each function section of the person detection information extraction function section 22 of the video analysis function section 20 and creates a search data table to be stored in the past search data table storage section 322. do. The detection/tracking result determination function unit 34 receives the output from each function unit of the human detection information extraction function unit 22 of the video analysis function unit 20, and determines whether a tracking candidate who corresponds to a prescribed symptom is in the surveillance image of the surveillance camera 10. A detection history table is created to be stored in the detection history table storage unit 323, and the detected face image 3121 and facial features of the person extracted by each functional unit of the human detection information extraction function unit 22 are detected. The amount 3122, the detected whole body image 3123, and the whole body feature amount 3124 are stored in the detection history data storage section 312. The detection/tracking result determination function unit 34 also receives the output of each functional unit of the monitoring/tracking execution function unit 23 of the video analysis function unit 20 and determines the whole body image and/or the tracking target person specified by the monitor. It is detected that the same person as the face image appears in the monitoring image of any of the monitoring cameras 10, and in response to the detection, a record is created in the tracking table of the tracking target person stored in the tracking table storage unit 325. will be added.

The web application 35 is an application program that performs various processes in response to requests from the web browser 41 of the monitoring terminal 40 and creates data for viewing indicating the processing results. The Web application 35 provides the Web server 30 with functions as a login function section 351, a monitoring function section 352, a tracking function section 353, and a past search function section 354.

The login function unit 351 receives a login from the Web browser 41, performs authentication, and permits the authorized user to use the functions provided by the Web application 35.

The monitoring function unit 352 generates an alert and presents it to the web browser 41 when the detection/tracking result determination function unit 34 detects a tracking candidate. For this purpose, the monitoring function section 352 includes a detection alert output function section 3521 that generates a detection alert screen to be presented to the web browser 41. The detection alert screen is an output screen for presenting an image of a person detected as a tracking candidate who is a specific person corresponding to a prescribed symptom, together with status information thereof.

The tracking function unit 353 tracked the person in real time across the plurality of surveillance cameras 10 using the same face image and full body image of the person as the face image of the tracking candidate detected by the detection/tracking result determination function unit 34. The tracking results are presented to the web browser 41. To this end, the tracking function section 353 includes an image registration function section 3531 and a tracking alert output function section 3532. The image registration function unit 3531 receives a designation of a face image of a person to be tracked to be tracked through a designation operation from a supervisor on the web browser 41, and stores a registered face image 3131 and facial feature amounts in the management data storage unit 313. 3132 is stored, and at the same time, a management table 3241 is created and stored in the management table storage section 324. The tracking alert output function unit 3532 outputs an image of a person detected to be similar to the tracked person when the detection/tracking result determination function unit 34 detects a person who is the same as the whole body image and/or face image of the tracked person. A tracking alert screen is presented on the Web browser 41, which is an output screen for presenting the following along with a step display indicating the caution step based on the degree of similarity.

The past search function section 354 receives a time and place specification from the Web browser 41, displays a list of people who visited the specified time and place, and accepts the specification of a person to be searched from among them. Then, the past search function unit 354 searches for a similar person stored in the past search data storage unit 311 from the face image and full body image of the specified person by the detection/tracking result determination function unit 34, and searches the result on the Web. It is presented on the browser 41.

FIG. 12 is a block diagram showing the hardware configuration of the Web server 30. The web server 30 includes, for example, a hardware processor 301A such as a CPU (Central Processing Unit), a program memory 301B, a storage device 302, and a communication interface device 303. Program memory 301B, storage device 302 and communication interface device 303 are connected to hardware processor 301A via bus 304.

The communication interface device 303 includes, for example, one or more wired or wireless communication interface units, and transmits and receives various information between the video analysis function unit 20 and the monitoring terminal 40 according to the communication protocol used in the network NET. enable.

The program memory 301B includes, as a storage medium, a non-volatile memory that can be written to and read from at any time such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a non-volatile memory such as a ROM (Read Only Memory). It stores programs necessary for executing various control processes according to an embodiment of the present invention by being executed by a hardware processor 301A such as a CPU. That is, the hardware processor 301A reads and executes the program stored in the program memory 301B, thereby functioning as the search person detection result storage function unit 33 and the detection/tracking result determination function unit 34 as shown in FIG. be able to. Note that these processing functional units may be realized by separate hardware processors. That is, the Web server 30 may include multiple hardware processors. In addition, at least a portion of these processing functional units may include various other hardware circuits, including integrated circuits such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-programmable Gate Arrays), and GPUs (Graphics Processing Units). It may be realized in the form of Furthermore, the programs stored in the program memory 301B can include a program of a web application 35 as shown in FIG.

The storage device 302 is a storage medium that uses a combination of a non-volatile memory such as an HDD or an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory). It is used to store various data acquired and created during processing. Furthermore, the storage device 302 can be configured with a file server 31 and a DB server 32 as shown in FIG.

(2) Operation Next, the operation of the person tracking system configured as above will be explained.

(2-1) Monitoring operation When the web browser 41 is started on the monitoring terminal 40 and an access operation is performed to the web server 30, the web server 30 performs an authentication operation using the login function section 351 provided by the web application 35. do. If it is confirmed that the user is an authorized user, the web server 30 uses the monitoring function unit 352 provided by the web application 35 to generate a monitoring screen for viewing on the web browser 41 and sends the screen to the web browser 41. Send. A monitoring screen can be generated as follows.

Each of the plurality of surveillance cameras 10 periodically acquires surveillance images, and inputs the acquired surveillance images to the corresponding video analysis function section 20. The image acquisition module 21 of the video analysis function section 20 acquires a frame image which is a surveillance image from the corresponding surveillance camera 10, and the human detection information extraction function section 22 acquires a whole body image of the person shown in the frame image, The region position, whole body feature amount, and facial feature amount of the whole body image are extracted. Then, the video analysis function unit 20 transmits the frame image, the whole body image, the region position of the whole body image, the whole body feature amount, the face feature amount, the body temperature, the pulse rate, and the cough symptoms to the web server 30 via intra-server communication, Websocket, etc. do.

The web server 30 stores the frame images, whole body images, whole body features, facial features, body temperature, pulse rate, and cough symptoms transmitted from each of the plurality of video analysis function units 20 as past search data on a file server. 31 is stored in the past search data storage unit 311. At this time, the search person detection result storage function section 33 of the Web server 30 creates a past search data table including the region positions of the whole body images transmitted from each of the plurality of video analysis function sections 20, and stores the data table in the DB. It is stored in the past search data table storage section 322 of the server 32. In this way, each time a surveillance image is acquired by each surveillance camera 10, past search data is accumulated in the past search data storage section 311, and linked to the past search data in the past search data table storage section 322. Past search data tables are accumulated.

The monitoring function unit 352 can generate a monitoring screen by arranging the frame images from each monitoring camera 10 accumulated in the past search data storage unit 311 on one screen. By viewing this monitoring screen displayed on the web browser 41 of the monitoring terminal 40, a supervisor such as a security guard can grasp the real-time situation of each part within the facility.

In addition, in the person tracking system, in parallel with the real-time situation monitoring of each part, the operation of detecting the arrival of a tracking candidate is performed. This pursuit candidate visit detection operation includes detection result determination processing by the detection/tracking result determination function section 34 and detection alert output processing by the detection alert output function section 3521 of the monitoring function section 352.

FIG. 13 is a flowchart showing the processing procedure of detection result determination processing by the detection/tracking result determination function unit 34 of the Web server 30. The detection/tracking result determination function unit 34 executes the process shown in this flowchart every time past search data based on a monitoring image of any surveillance camera 10 is stored in the past search data table storage unit 322. be able to. Alternatively, the detection/tracking result determination function unit 34 performs the process shown in this flowchart every time the past search data based on the monitoring images of each surveillance camera 10 is stored in the past search data table storage unit 322 for a predetermined number of times. may be implemented. In the latter case, any one of the plural past search data, for example, the latest one, can be used as the processing target.

The detection/tracking result determination function unit 34 first obtains state information such as body temperature 3115, pulse rate 3116, and cough symptoms 3117 from the past search data stored in the past search data storage unit 311 (step S101). . In addition, the detection/tracking result determination function unit 34 uses correction values of state information from the camera information table 3211 of the surveillance camera 10 from which the past search data is acquired, which is stored in the camera information table storage unit 321. A body temperature correction value, a pulse rate correction value, and a cough symptom correction value are obtained (step S102). Then, the body temperature 3115, pulse rate 3116, and cough symptom 3117 acquired above are corrected using the body temperature correction value, pulse rate correction value, and cough symptom correction value (step S103).

Thereafter, the detection/tracking result determination function unit 34 determines whether the corrected body temperature is equal to or higher than a pre-stored prescribed body temperature (step S104). When it is determined that the corrected body temperature is equal to or higher than the specified body temperature, the detection/tracking result determination function unit 34 stores the detected face image 3121, facial feature amount 3122, and detected whole body image as detection history data in the detection history data storage unit 312. 3123 and the whole body feature amount 3124 are stored, a detection history table 3231 is generated, and it is stored in the detection history table storage section 323 (step S105). In the generated detection history table 3231, the corrected body temperature, corrected pulse rate, and corrected cough symptoms, which are the corrected state information corrected in step S103, are registered. After that, the detection/tracking result determination function section 34 ends this detection/tracking result determination process. Note that it is assumed that values corresponding to prescribed symptoms such as influenza are stored in advance in the storage device 302 for the prescribed body temperature, prescribed pulse rate, and prescribed cough symptoms that will be described later.

On the other hand, if it is determined in step S104 that the corrected body temperature is not higher than the specified body temperature, the detection/tracking result determination function unit 34 determines that the corrected pulse rate is equal to or higher than the pre-stored specified pulse rate. It is determined whether or not (step S106). If it is determined that the corrected pulse rate is equal to or higher than the specified pulse rate, the detection/tracking result determination function unit 34 proceeds to the process of step S105.

Further, if it is determined that the corrected pulse rate is not equal to or higher than the prescribed pulse rate, the detection/tracking result determination function unit 34 determines whether the corrected cough symptoms are equal to or higher than the prescribed cough symptoms stored in advance. (Step S107). If it is determined that the corrected cough symptoms are equal to or higher than the prescribed cough symptoms, the detection/tracking result determination function unit 34 proceeds to the process of step S105. If it is determined that the corrected cough symptoms are not greater than the prescribed cough symptoms, the detection/tracking result determination function unit 34 ends this detection/tracking result determination process.

As described above, if any of the corrected body temperature, pulse rate, and cough symptoms is greater than or equal to the predetermined temperature, pulse rate, or cough symptom, it is determined that a tracking candidate has been detected, and the detection history data storage unit The detection history data is stored in the detection history table 312, and the corrected body temperature, pulse rate, and cough symptoms are registered in the detection history table 3231.

14A and 14B are flowcharts illustrating the processing procedure of detection alert output processing by the detection alert output function section 3521 of the monitoring function section 352. The detection alert output function unit 3521 periodically performs the process shown in this flowchart, for example, at the same processing timing as the detection result determination process by the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection alert output function unit 3521 first determines whether a narrowing designation operation has been performed on the detection alert screen displayed on the Web browser 41 by the monitor on the monitoring terminal 40 (step S201). If it is determined that the narrowing down designation operation has been performed, the detection alert output function unit 3521 proceeds to step S206, which will be described later.

On the other hand, if it is determined that the narrowing down designation operation has not been performed, the detection alert output function unit 3521 acquires information of a new detection history table 3231 from the detection history table storage unit 323 (step S202). That is, the detection alert output function unit 3521 extracts and obtains detection dates and times in the detection history table 3231 in which dates and times after the previous processing date and time are recorded.

Then, the detection alert output function unit 3521 creates a new detected person card, which is a screen image, based on the acquired information of the new detection history table 3231 (step S203). This detected person card is a screen image that includes a detected face image 3121 and a detected whole body image 3123 stored as detection history data in the detection history data storage unit 312, and the corresponding state information after correction in the detection history table 3231. be. Details of this detected person card will be described later.

The detection alert output function unit 3521 updates the detection alert screen to be displayed on the web browser 41 using the created new detected person card (step S204). Then, the detection alert output function unit 3521 transmits this updated detection alert screen to the Web browser 41 via the network NET, and displays it there (step S205). The web browser 41 displays this detection alert screen in an alert screen display area provided in a part of the monitoring screen, or by opening a window separate from the monitoring screen. Then, the detection alert output function unit 3521 ends this detection alert output processing.

On the other hand, if it is determined in step S201 that a narrowing down designation operation has been performed, the detection alert output function unit 3521 acquires the specified narrowing down level (step S206). This narrowing level is an index for narrowing down the tracking information cards to be displayed on the detection alert screen.

The detection alert output function unit 3521 also acquires information on the detection history table 3231 to be processed from the detection history table storage unit 323 (step S207). That is, the detection alert output function unit 3521 acquires information from one detection history table in order from the newest one based on the detection date and time in the detection history table 3231.

Thereafter, the detection alert output function unit 3521 determines whether the acquired corrected body temperature is higher than the specified body temperature specified as the narrowing level, that is, whether the corrected body temperature is equal to or higher than the specified body temperature ( Step S208). If it is determined that the corrected body temperature is equal to or higher than the specified body temperature, the detection alert output function unit 3521 creates a detected person card, which is a screen image, based on the information in the acquired detection history table 3231 (step S209). .

Then, the detection alert output function unit 3521 determines whether a predetermined number of detected person cards have been created (step S210). This predetermined number can depend on the number of displayable detection alert screens. Alternatively, it may be any number. If it is determined that the predetermined number of detected person cards have not been created yet, the detection alert output function unit 3521 returns to the process of step S207.

Further, if it is determined in step S208 that the corrected body temperature is not equal to or higher than the specified body temperature, the detection alert output function unit 3521 determines that the acquired corrected pulse rate is equal to or higher than the specified pulse rate specified as the narrowing level. It is determined whether or not (step S211). If it is determined that the corrected pulse rate is equal to or higher than the designated pulse rate, the detection alert output function unit 3521 proceeds to the process of step S209 and creates a detected person card.

Further, if it is determined in step S213 that the corrected pulse rate is not equal to or higher than the specified pulse rate, the detection alert output function unit 3521 determines that the obtained corrected cough symptom is the specified cough specified as the narrowing level. It is determined whether the condition is more than a symptom (step S212). If it is determined that the corrected cough symptoms are equal to or higher than the designated cough symptoms, the detection alert output function unit 3521 proceeds to the process of step S209 and creates a detected person card. If it is determined that the corrected cough symptoms are not greater than the designated cough symptoms, the detection alert output function unit 3521 proceeds to the process of step S210.

If it is determined in step S210 that a predetermined number of detected person cards have been created, the detection alert output function unit 3521 sorts the created detected person cards in order of detection date and time, and updates the detection alert screen ( Step S213). Then, the detection alert output function unit 3521 proceeds to the process of step S205, and transmits this updated detection alert screen to the Web browser 41 via the network NET to display it there.

Note that if the information in the detection history table 3231 to be processed that should be acquired in the above step S207 runs out before the predetermined number of detected person cards are created, the process may proceed to step S213.

FIG. 15 is a diagram showing an example of the detection alert screen 42 displayed on the Web browser 41. The detection alert screen 42 includes a detected person card 421 for each detected person and a narrowing down instruction area 422.

The detected person card 421 includes an alert label 4211, a person requiring attention ID 4212, a tracking candidate image 4213, a detected body temperature 4214, a detected pulse rate 4215, a detected cough symptom 4216, and a tracking start button 4217 as display contents. Can be done.

Here, the attention label 4211 is a message for calling the observer's attention, and may be an identification display of a blinking indicator light. The person requiring attention ID 4212 is identification information transcribed from the detection history table 3231. The tracking candidate image 4213 is a detected face image 3121 and a detected whole body image 3123 stored in the detection history data storage unit 312 of the person detected as the tracking candidate. The detected body temperature 4214, the detected pulse rate 4215, and the detected cough symptoms 4216 are the corrected body temperature, corrected pulse rate, and corrected cough symptoms transferred from the detection history table 3231. The tracking start button 4217 is a button that is pressed when tracking the person.

Note that each time a new specific person is detected as a tracking candidate, the old detected person card 421 is shifted downward so that it is displayed at the top of the detection alert screen 42. . A new detected person card 421 is not added for the same person.

Further, the narrowing down instruction area 422 is an area for receiving an instruction for narrowing down the detected person cards 421 to be displayed on the detection alert screen 42. This narrowing instruction area 422 includes a body temperature presentation threshold setting slider 4221, a pulse rate presentation threshold setting slider 4222, and a cough symptom threshold setting button 4223.

Here, the body temperature presentation threshold setting slider 4221 is a slider that is slid to set the presentation threshold indicating the designated body temperature used in step S208. A tracking information card 432 containing a corrected body temperature that is higher than the body temperature instructed by the supervisor's sliding operation of the slider is displayed on the detection alert screen 42. In FIG. 15, the body temperature presentation threshold setting slider 4221 shows a state where the prescribed body temperature of 37.5° C. is set.

Similarly, the pulse rate presentation threshold setting slider 4222 is a slider that is slid to set the presentation threshold indicating the designated pulse rate used in step S211. A tracking information card 432 containing a corrected pulse rate greater than or equal to the pulse rate instructed by the monitor's slide operation of the slider is displayed on the detection alert screen 42. In FIG. 15, the pulse rate presentation threshold setting slider 4222 shows a state where a prescribed pulse rate of 110 is set.

Furthermore, in this embodiment, the cough symptom threshold setting button 4223 indicates the degree of cough symptoms in three stages, so it includes three buttons corresponding to the number of stages of cough symptoms. When the observer presses these three buttons, a tracking information card 432 containing corrected cough symptoms of a specified number or more is displayed on the detection alert screen 42. The example in FIG. 15 shows a state in which the number of stages in the cough symptom threshold setting button 4223 is one, which is a prescribed cough symptom. The cough symptom threshold setting button 4223 also includes a reset button that is pressed when resetting the specified number of stages.

In the example of FIG. 15, the detection alert screen 42 includes three detected person cards 421. The detection person card 421 presented at the top has a body temperature of 37.6°C, which is a body temperature of 37.5°C or higher specified by the body temperature presentation threshold setting slider 4221, or a body temperature specified by the cough symptom threshold setting button 4223. This is a detected person card 421 that presents tracking candidates for whom cough symptoms of level 2, which is a degree of cough symptoms of level 1 or higher, have been detected. The detection person card 421 presented in the second row has a body temperature of 38.8°C, which is 37.5°C or higher, which is specified by the body temperature presentation threshold setting slider 4221, and a pulse rate presentation threshold setting slider 4222. A detection person card presenting a tracking candidate for whom a pulse rate of 115, which is a pulse rate of 110 or more, or a cough symptom of 3 levels, which is a cough symptom level of 1 level or more specified by the cough symptom threshold setting button 4223, is detected. It is 421. The detected person card 421 presented in the third row has a pulse rate of 115, which is a pulse of 110 or more specified by the pulse rate presentation threshold setting slider 4222, or 1, which is specified by the cough symptom threshold setting button 4223. This is a detected person card 421 that presents tracking candidates for whom cough symptoms of level 2, which is a degree of cough symptoms higher than level 2, have been detected.

FIG. 16 is an example of the detection alert screen 42 when 37.8° C. is specified by the body temperature presentation threshold setting slider 4221 and 2 steps are specified by the cough symptom threshold setting button 4223. In this example, as a result of the narrowing down based on the presentation thresholds for body temperature and cough symptoms, the detected person card 421 of the tracking candidate that was presented at the top in the example of FIG. 15 is no longer presented. In the example of FIG. 16, the detected person cards 421 that were presented in the second and third rows in the example of FIG. Detected person who presents a tracking candidate who has a body temperature of 7.9°C, which is a body temperature of 7.9°C, or a cough symptom of 2 levels, which is a cough symptom level of 2 levels or more specified by the cough symptom threshold setting button 4223. Card 421 is presented in the third row.

(2-2) Tracking operation When the monitor presses the tracking start button 4217 on the detection alert screen 42, the web server 30 uses the tracking function unit 353 provided by the web application 35 to perform the video analysis function unit 20 and the detection /Using the tracking result determination function section 34, the Web browser 41 is caused to present the tracking results of tracking the person. This tracking operation includes image registration processing by the image registration function unit 3531 of the tracking function unit 353, tracking processing by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, and tracking results by the detection/tracking result determination function unit 34. This includes determination processing and tracking alert output processing by the tracking alert output function section 3532 of the tracking function section 353.

FIG. 17 is a flowchart showing the processing procedure of image registration processing by the image registration function section 3531 of the tracking function section 353. The image registration function unit 3531 periodically performs the processing shown in this flowchart, for example, in synchronization with the acquisition interval of monitoring images of the monitoring camera 10.

The image registration function unit 3531 first determines whether or not the monitor presses the tracking start button 4217 on the detection alert screen 42 displayed on the Web browser 41 on the monitoring terminal 40 (step S301). If it is determined that the tracking start button 4217 has not been pressed, the image registration function unit 3531 ends this image registration process.

On the other hand, if it is determined that the tracking start button 4217 has been pressed, the image registration function unit 3531 acquires the face image of the person as a tracking face image (step S302). That is, the image registration function unit 3531 selects the detected face image 3121 from the detection history data storage unit 312 as a tracking face image based on the tracking ID of the detection history table storage unit 323 that has the person-of-care ID on the detection alert screen 42. get. For example, the image registration function unit 3531 acquires the detected face image 3121 specified by the detection ID listed in the detection history table 3231 having the tracking ID, whose detection date and time are in a specific period, for example, the latest one. be able to. The specific period is not limited to the latest, but may be the oldest, the latest or oldest within a specified period, or the season (spring, summer, fall, winter, etc.). Alternatively, the image registration function unit 3531 generates a selection screen that includes the detected face images 3121 specified by the detection IDs listed in each of the detection history tables 3231 having the tracking ID in a list format, and displays the selection screen on the web browser 41. may be displayed so that the observer can select the detected face image 3121 to be used as the tracking face image. Then, the image registration function unit 3531 stores the acquired tracking face image in the management data storage unit 313 as a registered face image 3131. The image registration function unit 3531 also creates a management table 3241 and stores it in the management table storage unit 324. The management table 3241 includes a query facial feature path indicating the storage path of the registered facial image 3131 in the management data storage unit 313.

Then, the image registration function unit 3531 obtains, as a tracking full-body image, a whole-body image corresponding to the detected face image 3121 obtained from the detection history data storage unit 312 as a tracking-use full-body image (step S303). That is, the image registration function unit 3531 acquires the detected whole body image 3123 specified by the detection ID described in the detection history table 3231 as the whole body image for tracking with respect to the detected face image 3121 acquired as the face image for tracking. Alternatively, the image registration function unit 3531 acquires the whole body image 3112 as the tracking whole body image from the past search data storage unit 311 based on the detection ID. The image registration function section 3531 stores this acquired tracking whole body image in the management data storage section 313 as a registered whole body image 3133. Further, the image registration function unit 3531 adds a query whole body feature path indicating the storage path of the registered whole body image 3133 in the management data storage unit 313 to the management table 3241 stored in the management table storage unit 324.

Furthermore, the image registration function unit 3531 acquires a facial image feature amount for tracking and a whole body image feature amount for tracking (step S304). That is, the image registration function unit 3531 acquires the face feature amount 3122 and the whole body feature amount 3124 from the detection history data storage unit 312 as the tracking face feature amount and the tracking whole body feature amount based on the detection ID. The image registration function unit 3531 stores the acquired tracking facial feature amount and tracking whole body feature amount in the management data storage unit 313 as a facial feature amount 3132 and a whole body feature amount 3134. Further, the image registration function unit 3531 adds the face feature file path and the whole body feature file path indicating the feature storage paths to the management table 3241 stored in the management table storage unit 324.

Then, the image registration function unit 3531 uses the registered face image 3131 and the registered whole body image 3133 stored in the management data storage unit 313 as tracking images, and also uses the facial feature amount 3132 and the whole body feature amount 3134 as tracking feature amounts. , and is transmitted to each of the plurality of video analysis function units 20 via intra-server communication, Websocket, etc. (step S305). Thereafter, the image registration function unit 3531 ends this image registration process.

FIG. 18 is a flowchart showing the processing procedure of the tracking process by the monitoring/tracking execution function unit 23 of the video analysis function unit 20. The monitoring/tracking execution function unit 23 performs the processing shown in this flowchart every time it receives a tracking image and a feature amount from the tracking function unit 353 and image registration function unit 3531 of the web server 30, for example, through intra-server communication, Websocket, etc. implement.

The monitoring/tracking execution function unit 23 first acquires the tracking image and feature quantity received from the image registration function unit 3531 of the Web server 30 (step S411).

Thereafter, the monitoring/tracking execution function unit 23 acquires the whole body image at time T and its feature amount from the human detection information extraction function unit 22 as the comparison whole body image and the comparison whole body feature amount (step S412). When the monitoring images of the corresponding monitoring camera 10 include full-body images of multiple people, the monitoring/tracking execution function unit 23 acquires the full-body image for comparison and the full-body feature for comparison for each person.

Then, the monitoring/tracking execution function unit 23 uses the whole body matching module 231 to determine whether each of the whole body features for matching from the human detection information extraction function unit 22 is similar to the whole body features for tracking from the image registration function unit 3531. The whole body image similarity, which is the degree of similarity, is calculated (step S413).

Further, the monitoring/tracking execution function unit 23 acquires the face image at time T and its feature amount from the human detection information extraction function unit 22 as a matching face image and matching face feature amount (step S414). When the monitoring image of the corresponding monitoring camera 10 includes facial images of a plurality of people, the monitoring/tracking execution function unit 23 acquires a matching facial image and a matching facial feature amount for each person. Note that depending on the orientation of the person, a full body image may be extracted but a face image may not be extracted. Therefore, the number of full body images for comparison acquired in step S412 and the number of facial images for comparison acquired in step S414 do not necessarily match. Furthermore, there may be a case in which no face image can be extracted and no matching face image (and matching face feature amount) can be obtained.

Then, the monitoring/tracking execution function unit 23 uses the face matching module 232 to determine whether each of the matching face features from the human detection information extraction function unit 22 is similar to the tracking face feature from the image registration function unit 3531. The face image similarity is calculated as the degree of similarity (step S415).

Thereafter, the monitoring/tracking execution function unit 23 transmits the set of similarity and matching images for the whole body image and face image to the detection/tracking result determination function unit 34 of the web server 30 via intra-server communication, Websocket, etc. (Step S416). Note that the set of similarity and matching images may include both a set of full-body image similarity and matching full-body images and a set of face image similarity and matching facial images, There may be a case where only a set of whole body image similarity and a comparison whole body image is included without a set of similarity and a comparison face image.

Then, the monitoring/tracking execution function unit 23 determines whether or not the monitor presses the tracking stop button on the tracking alert screen displayed on the Web browser 41 on the monitoring terminal 40 (step S417). Details of the tracking alert screen having a tracking stop button will be described later. If it is determined that the tracking stop button has been pressed, the monitoring/tracking execution function unit 23 ends this monitoring/tracking process.

Further, if it is determined that the tracking stop button has not been pressed, the monitoring/tracking execution function unit 23 adds the time interval t1 to the time T, that is, after updating to the next processing time ( Step S418), repeats the above processing from step S412. Here, the time interval t1 can be any arbitrary time interval, for example, several seconds. Alternatively, the time interval t1 may be a time interval that is an integral multiple of the acquisition interval of surveillance images of the surveillance camera 10.

FIG. 19 is a flowchart showing the processing procedure of the tracking result judgment process by the detection/tracking result judgment function unit 34 of the Web server 30. The detection/tracking result determination function unit 34 periodically executes the process shown in this flowchart, for example, in synchronization with the processing interval of the tracking process of the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection/tracking result determination function unit 34 first determines whether or not a set of similarity and matching images has been received from the monitoring/tracking execution function unit 23 (step S121). If it is determined that the set of similarity and matching images has not been received, the detection/tracking result determination function unit 34 ends this tracking result determination process.

On the other hand, if it is determined that the set of similarity and matching images has been received, the detection/tracking result determination function unit 34 acquires the face matching threshold 1 and the whole body matching threshold 1 (step S122). That is, the detection/tracking result determination function unit 34 determines the surveillance camera 10 corresponding to the video analysis function unit 20 that is the source of the set of similarity and matching images stored in the camera information table storage unit 321. The face matching threshold 1 and the whole body matching threshold 1 are obtained from the camera information table 3211 of .

Then, the detection/tracking result determination function unit 34 compares each of the received whole-body image similarities with the acquired whole-body matching threshold 1, and determines whether there is a whole-body image similarity that is equal to or higher than the whole-body matching threshold 1 ( Step S123). If it is determined that the whole body image similarity is equal to or greater than the whole body matching threshold value 1, the detection/tracking result determination function unit 34 determines that the person having the whole body feature amount is the person to be tracked. Therefore, in this case, the detection/tracking result determination function unit 34 stores the whole body image similarity determined to be large in a predetermined storage area of the storage device 302 together with the corresponding whole body image for comparison received in step S121. At the same time, the whole body image similarity is registered in the tracking table 3251 (step S124). When registering this whole body image similarity, if the tracking table 3251 of the relevant person ID is not yet stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is tracked. It is stored in the table storage section 325. Furthermore, if the tracking table 3251 of the relevant person ID is already stored in the tracking table storage unit 325, a record is added instead of generating a new tracking table 3251. The whole body image similarity registered in the tracking table 3251 is a whole body image similarity having a value equal to or greater than the whole body matching threshold 1.

After that, or if it is determined in step S123 that there is no whole-body image similarity equal to or higher than the whole-body matching threshold 1, the detection/tracking result determination function unit 34 sets the obtained face matching threshold 1 for each of the received face image similarities. It is determined whether the face image similarity is equal to or greater than the face matching threshold value 1 (step S125). If it is determined that the facial image similarity is equal to or greater than the face matching threshold value 1, the detection/tracking result determination function unit 34 determines that the person having the facial feature amount is the person to be tracked. Therefore, in this case, the detection/tracking result determination function unit 34 stores the facial image similarity determined to be large in a predetermined storage area of the storage device 302 together with the corresponding matching face image received in step S121. The face image similarity is stored in the tracking table 3251 (step S126). When registering this face image similarity, if the tracking table 3251 of the relevant person ID is not yet stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is tracked. It is stored in the table storage section 325. Furthermore, if the tracking table 3251 of the relevant person ID is already stored in the tracking table storage unit 325, a record is added instead of generating a new tracking table 3251. The face image similarity registered in the tracking table 3251 is a whole body image similarity with a value equal to or greater than the face matching threshold value 1.

After that, or if it is determined in step S125 that there is no face image similarity equal to or higher than the face matching threshold 1, the detection/tracking result determination function unit 34 ends this tracking result determination process.

20A to 20C are flowcharts showing the processing procedure of the tracking alert output processing by the tracking alert output function section 3532 of the tracking function section 353. The tracking alert output function unit 3532 periodically executes the process shown in this flowchart, for example, at every time interval t1 in the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The tracking alert output function unit 3532 first determines whether a narrowing designation operation has been performed on the tracking alert screen displayed on the Web browser 41 by the monitor on the monitoring terminal 40 (step S501). If it is determined that the narrowing down designation operation has been performed, the tracking alert output function unit 3532 proceeds to step S519, which will be described later.

On the other hand, if it is determined that the narrowing down specification operation has not been performed, the tracking alert output function section 3532 outputs the information from the camera information table 3211 for each surveillance camera 10 stored in the camera information table storage section 321. Face matching thresholds 1 to 3 and whole body matching thresholds 1 to 3 are acquired (step S502).

After that, the tracking alert output function unit 3532 obtains new tracking results from the tracking table 3251 stored in the tracking table storage unit 325 (step S503). That is, the tracking alert output function unit 3532 extracts and obtains the whole body image similarity and face image similarity in the tracking table 3251 in which real numbers of similarities are recorded instead of image similarity IDs.

Then, the tracking alert output function unit 3532 determines whether the face image similarity value in the tracking table 3251 is equal to or greater than the face matching threshold 3 (step S504). When it is determined that the face image similarity value is equal to or higher than the face matching threshold 3, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 into a face image from a real number. The similarity label ID, here, is rewritten to ID=3 (step S505).

On the other hand, if it is determined that the face image similarity value is not equal to or higher than the face matching threshold 3, the tracking alert output function unit 3532 further determines whether the face image similarity value is equal to or higher than the face matching threshold 2. (Step S506). When it is determined that the face image similarity value is equal to or higher than the face matching threshold 2, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the face image similarity value from the real number. The similarity label ID, here, is rewritten to ID=2 (step S507).

Furthermore, if it is determined that the face image similarity value is not greater than or equal to the face matching threshold 2, the tracking alert output function unit 3532 further determines whether or not the face image similarity value is greater than or equal to the face matching threshold 1. (Step S508). If it is determined that the face image similarity value is equal to or greater than the face matching threshold 1, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the face image from the real number. The similarity label ID, here, is rewritten to ID=1 (step S509).

In the processing related to the tracking operation of the tracking function unit 353, both the face image similarity and the whole body similarity are used, and if only one of the face image similarity and the whole body similarity is registered in the tracking table 3251. exists. In other words, when the face image similarity value is not greater than the face matching threshold 2, there is a case where a face image similarity value with a value greater than the face matching threshold 1 is registered, and a case where the face image similarity value is smaller than the face matching threshold 1. There are cases in which a value of degree of similarity is registered (or a value of degree of face image similarity is not registered). Therefore, just because the face image similarity value is not greater than or equal to the face matching threshold 2, it cannot be determined that the face image similarity value is greater than or equal to the face matching threshold 1.

If the face image similarity value in the tracking table 3251 is rewritten to the face image similarity label ID as described above, or if the face image similarity value is not greater than or equal to the face matching threshold 1 in step S508, If determined, the tracking alert output function unit 3532 moves to a whole body image similarity determination operation.

That is, the tracking alert output function unit 3532 determines whether the value of the whole body image similarity in the tracking table 3251 is equal to or greater than the whole body matching threshold 3 (step S510). If it is determined that the whole body image similarity value is equal to or higher than the whole body matching threshold 3, the tracking alert output function unit 3532 converts the whole body image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the whole body image from the real number. The similarity label ID, here, is rewritten to ID=3 (step S511).

On the other hand, if it is determined that the whole body image similarity value is not greater than the whole body matching threshold 3, the tracking alert output function unit 3532 further determines whether the whole body image similarity value is not greater than the whole body matching threshold 2. (Step S512). When it is determined that the value of the whole body image similarity is equal to or higher than the whole body matching threshold 2, the tracking alert output function unit 3532 converts the value of the whole body image similarity in the tracking table 3251 of the tracking table storage unit 325 into a whole body image from the real number. The similarity label ID, here, is rewritten to ID=2 (step S513).

Further, if it is determined that the value of the whole body image similarity is not equal to or greater than the whole body matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of the whole body image similarity is equal to or greater than the whole body matching threshold 1. (Step S514). When determining that the value of the whole body image similarity is equal to or higher than the whole body matching threshold 1, the tracking alert output function unit 3532 converts the value of the whole body image similarity in the tracking table 3251 of the tracking table storage unit 325 into a whole body image from the real number. The similarity label ID, here, is rewritten to ID=1 (step S515).

If the whole-body image similarity value in the tracking table 3251 is rewritten to the whole-body image similarity label ID as described above, or if the whole-body image similarity value is not greater than the whole-body matching threshold 1 in step S514 above, If determined, the tracking alert output function unit 3532 creates a new tracking information card that is a screen image (step S516). This tracking information card includes the full body image for comparison and the face image for comparison that were saved in the predetermined storage area of the storage device 302 in steps S124 and S126, and the corresponding whole body image similarity label ID and ID in the tracking table 3251. It is a screen image including a caution level display indicating a caution level based on a face image similarity degree label ID. Details of this tracking information card will be described later.

The tracking alert output function unit 3532 updates the tracking alert screen to be displayed on the web browser 41 using the created new tracking information card (step S517). Then, the tracking alert output function unit 3532 transmits this updated tracking alert screen to the Web browser 41 via the network NET, and displays it there (step S518). The web browser 41 displays this tracking alert screen in an alert screen display area provided in a part of the monitoring screen, or by opening a window separate from the monitoring screen. The tracking alert output function unit 3532 then ends this tracking alert output processing.

On the other hand, if it is determined in step S501 that a narrowing down designation operation has been performed, the tracking alert output function unit 3532 acquires the specified narrowing down level (step S519). This narrowing down level is an index for narrowing down the tracking information cards to be displayed on the tracking alert screen.

The tracking alert output function unit 3532 extracts the caution-required person ID according to the specified narrowing down level from the tracking table 3251 stored in the tracking table storage unit 325 (step S520).

Then, the tracking alert output function unit 3532 outputs the whole body image 3112 stored as past search data in the past search data storage unit 311 and the video analysis function unit 20 based on the detected ID of the extracted person ID. A number of tracking information cards that can be displayed on the tracking alert screen, including a face image extracted from the whole body image 3112 by the human detection information extraction function unit 22, are created (step S521).

Thereafter, the tracking alert output function unit 3532 creates a new tracking information card (step S522) in the same manner as in step S516 above, and updates the tracking alert screen with the created tracking information card in the same way as in step S517 (step S523). ). The tracking alert output function unit 3532 then proceeds to the process of step S518, and transmits the updated tracking alert screen to the Web browser 41 to display it there.

FIG. 21 is a diagram showing an example of the tracking alert screen 43 displayed on the web browser 41. The tracking alert screen 43 includes a tracking target person information area 431, a tracking information card 432, and a narrowing down instruction area 433.

The tracking target information area 431 is an area for displaying information regarding the tracking target, and is additionally displayed at the top of the tracking alert screen 43 each time a tracking target is added. The old tracked person information area 431 is sequentially shifted downward. Here, the display contents are different between the highest level, that is, the latest tracking target person information area 431 and the other tracking target person information areas 431. That is, the highest tracking target person information area 431 includes an attention-calling label 4311, a caution-required person ID 4312, a registered image 4313, tracking information 4314, and a tracking stop button 4315. On the other hand, the tracking target person information area 431 other than the top level only includes a registered image 4313 and tracking information 4314.

Here, the attention label 4311 is a message for calling the observer's attention, and may be an identification display of a blinking indicator light. The person requiring attention ID 4312 is identification information transferred from the management table 3241. The registered images 4313 are a registered face image 3131 and a registered whole body image 3133 stored in the management data storage unit 313 of the tracked person. The tracking information 4314 is information regarding the tracking target person, and the tracking information 4314 in the highest tracking target person information area 431 and the tracking information 4314 in the other tracking target person information areas 431 have different contents. That is, the tracking information 4314 in the highest tracking target person information area 431 includes the corrected body temperature, corrected pulse rate, and corrected cough symptoms, which are status information transferred from the management table 3241. The tracking information 4314 in the tracking target person information area 431 other than the highest level includes tracking status instead of these status information. The tracking stop button 4315 is a button that is pressed to end tracking of the person.

Further, the tracking information card 432 is additionally displayed at the top of the tracking alert screen 43 every time the tracking alert output function unit 3532 creates a new card. Old tracking information cards 432 are sequentially shifted downward. This tracking information card 432 includes, as its display contents, detection date and time information 4321, detected camera location information 4322, detected face image 4323, face caution stage display 4324, detected whole body image 4325, whole body caution stage display 4326, and detected background image. Contains 4327.

Here, the detection date and time information 4321 is date and time information when the tracking target person was detected from the monitoring image of any of the monitoring cameras 10. The detected camera location information 4322 is camera location information based on the detected surveillance camera ID. The detected face image 4323 is a comparison face image of the detected and saved tracking target person. The face attention level display 4324 indicates the level of similarity between the face image of the person to be tracked and the face image of the detected person. That is, the degree of similarity is shown not as a real number but as a warning level display of multiple levels, here three levels, based on the face image similarity label ID in the tracking table 3251. In this face attention level display 4324, the number of identified squares represented by hatching in FIG. 21 corresponds to the face image similarity label ID. For example, in the top tracking information card 432, the face image similarity label ID in the tracking table 3251 is "2", so in the face attention stage display 4324, two rectangles are identified and displayed, and two squares are displayed from the top. In the th tracking information card 432, the face image similarity label ID in the tracking table 3251 is "3", so three rectangles are identified and displayed in the face attention stage display 4324.

Similarly, the detected full-body image 4325 is a detected and saved full-body image of the tracked person for comparison. The whole body caution level display 4326 indicates the level of similarity between the whole body image of the tracked person and the whole body image of the detected person. That is, the similarity is shown not as a real number but as a warning level display of multiple levels, here three levels, based on the whole body image similarity label ID in the tracking table 3251. Similar to the face caution stage display 4324, this whole body caution level display 4326 also has the number of identified rectangles represented by hatching in FIG. 21 corresponding to the whole body image similarity degree label ID. For example, in the top tracking information card 432, the whole body image similarity label ID in the tracking table 3251 is "3", so in the whole body caution stage display 4326, three rectangles are identified and displayed, and two squares from the top are displayed. In the th tracking information card 432, the whole body image similarity label ID in the tracking table 3251 is "2", so two squares are identified and displayed in the whole body caution stage display 4326.

The detected background image 4327 is a frame image that is the original monitoring image from which the face image for comparison and/or the whole body image for comparison are extracted. As this frame image, a frame image 3111 stored in the past search data storage unit 311 is used.

Note that the tracking information card 432 does not necessarily include both the detected face image 4323 and the face caution stage display 4324, and the detected whole body image 4325 and the whole body caution stage display 4326, and may include only one of them. Possible.

The narrowing down instruction area 433 is an area for accepting instructions for narrowing down the tracking information cards 432 to be displayed on the tracking alert screen 43. This narrowing down instruction area 433 includes a face image similarity selection button 4331, a face image similarity selection reset button 4332, a whole body image similarity selection button 4333, and a whole body image similarity selection reset button 4334.

Here, the face image similarity selection buttons 4331 include three buttons corresponding to the number of stages of the face attention level display 4324. Only the tracking information cards 432 that include the face caution level display 4324 of a specified number of levels or more in accordance with the pushing operation of the face image similarity selection button 4331 by the observer are displayed on the tracking alert screen 43. In the example of FIG. 21, the number of stages is not specified with the face image similarity selection button 4331, so the tracking alert screen 43 displays a tracking information card 432 that includes a face attention stage display 4324 of one or more stages. . The face image similarity selection reset button 4332 is a button that is pressed when resetting the number of stages specified by the face image similarity selection button 4331.

Similarly, the whole body image similarity selection button 4333 includes three buttons corresponding to the number of stages of the whole body caution level display 4326. Only the tracking information cards 432 that include the whole body caution level display 4326 of a specified number or more in accordance with the press operation of the whole body image similarity selection button 4333 by the observer are displayed on the tracking alert screen 43. In the example of FIG. 21, since the number of stages is not specified with the whole body image similarity selection button 4333, the tracking information card 432 including one or more stages of whole body caution stage display 4326 is displayed on the tracking alert screen 43. . The whole body image similarity selection reset button 4334 is a button that is pressed when resetting the number of stages specified by the whole body image similarity selection button 4333.

FIG. 22 is a diagram showing another example of the tracking alert screen 43 displayed on the web browser 41. This example shows a case where three stages are designated by the face image similarity selection button 4331 and one stage is designated by the whole body image similarity selection button 4333. Here, the pressed button is identified and displayed as indicated by hatching in FIG. 21. The specification by the face image similarity selection button 4331 and the specification by the whole body image similarity selection button 4333 become an AND condition, so that only the tracking information cards 432 that match the number of stages specified in both are displayed on the tracking alert screen 43. become.

(2-3) Past Search Operation The tracking operation by the tracking function unit 353 is a function of tracking a person to be tracked in real time. On the other hand, there are cases where it is desired to investigate whether a specific person has visited the facility in the past.

Therefore, the past search function unit 354 receives the time and place specification from the web browser 41 and searches the past search data based on the past search data table 3221 stored in the past search data table storage unit 322. A full-body image 3112 of the person who visited the specified time and place is extracted from the storage unit 311. Then, a selection screen displaying a list of the extracted whole-body images 3112 is created and displayed on the Web browser 41.

Upon receiving the selection and designation of the person to be searched from the web browser 41, the past search function unit 354 uses the selected and designated full-body image 3112 as the full-body image for tracking, and also replaces it with the monitoring image from the real-time monitoring camera 10. The same operation as the tracking function section 353 is performed for the frame image 3111 stored in the past search data storage section 311 for the specified time and place.

Thereby, the past search function unit 354 searches for a person similar to the selected person from among the past visitors stored in the past search data storage unit 311, and displays the results on the tracking alert screen 43. It can be displayed on the web browser 41 as .

(3) Effects As described in detail above, the web server 30 as a person tracking device according to an embodiment can monitor the health condition of a person included in each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10. The plurality of types of status information detected by the human detection information extraction function unit 22 are acquired from the human detection information extraction function unit 22 as an information extraction unit that detects the plurality of related status information from each of the monitoring images, and A detection/tracking result determination function unit 34 as a determination unit that determines whether or not a person included in each of the monitoring images is a specific person corresponding to a prescribed symptom based on the plurality of types of acquired status information; /Based on the determination result of the tracking result determination function unit 34, among the images of the person extracted from the surveillance images of each of the plurality of surveillance cameras 10, the image of a specific person corresponding to prescribed symptoms such as influenza is tracked. The image registration function unit 3531 serves as a registration unit that registers as a target person image, and the tracking target person image registered by the image registration function unit 3531 is obtained from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10. A detection/tracking result determination function unit 34 as a detection unit that detects an image of a similar person, and a tracking output for presenting an image of a person similar to the tracking target person image detected by the detection/tracking result determination function unit 34. A tracking alert output function unit 3532 as a tracking output screen generation unit that generates a tracking alert screen 43 as a screen is provided. This makes it possible to detect and track a person who has a prescribed symptom from the surveillance image acquired by the surveillance camera 10.

Note that, based on the determination result of the detection/tracking result determination function unit 34, the web server 30 selects a specific person who corresponds to a prescribed symptom from among the images of the person extracted from the surveillance images of each of the plurality of surveillance cameras 10. The image registration function section 3531 further includes a detection alert output function section 3521 as a detection output screen generation section that generates a detection alert screen 42 as a detection output screen for presenting the image of , together with status information. From the monitoring terminal 40 that displays the detection alert screen 42 generated by the detection alert output function unit 3521, the designation of the image of the person to be tracked to be tracked is obtained from among the images of the person presented on the detection alert screen 42. , the acquired specified image is registered as a tracking target person image. Therefore, by presenting a plurality of tracking candidates together with their status information on the detection alert screen 42, the monitor can easily recognize the tracking target who must be tracked.

In addition, when any one of the acquired multiple types of status information is equal to or higher than a predetermined value, the detection/tracking result determination function unit 34 detects the person extracted from the surveillance images of each of the multiple surveillance cameras 10. The image is determined to be of a specific person who has a specified symptom. Therefore, it is possible to reliably pick up a person who has even the slightest possibility of falling under the prescribed symptoms.

In addition, the detection alert output function unit 3521 acquires presentation thresholds specified for each of the plurality of types of status information, and based on the acquired presentation thresholds, the detection/tracking result determination function unit 34 determines the specified symptoms. The images to be presented on the detection alert screen 42 are narrowed down from among the images of the specific person determined to be applicable to the above. Therefore, by specifying the presentation threshold, the monitor can narrow down the people who may fall under the prescribed symptoms presented on the detection alert screen 42, and the monitor can identify the persons to be tracked who must be tracked. become easily recognizable.

In this case, the detection alert output function unit 3521 selects an image of a specific person for which status information equal to or higher than any one of the acquired presentation thresholds for each type is detected as an image to be presented on the detection alert screen 42. I am trying to choose as such. Therefore, there is no risk that a person who has even the slightest possibility of falling under the prescribed symptoms will be excluded from the narrowing down.

The web server 30 further includes a camera information table storage unit 321 as a camera information storage unit that stores correction coefficients for correcting status information for each of the plurality of surveillance cameras 10, and has a detection/tracking result determination function. The unit 34 corrects the status information acquired from the human detection information extraction function unit 22 using the correction coefficient stored in the camera information table storage unit 321, and based on the corrected status information, the image of the person is adjusted to the specified value. It is determined whether the image is of a specific person who has the symptoms. Therefore, variations in status information depending on the type of surveillance camera 10 can be reduced.

Note that the status information can include at least one of body temperature, pulse rate, and cough symptoms, which are each extracted from the monitoring image.

Further, the person tracking system according to one embodiment includes a web server 30 as a person tracking device according to one embodiment, a plurality of surveillance cameras 10, a surveillance terminal 40 operated by a supervisor, and a person detection information extraction function. A video analysis function section 20 as an analysis section including a section 22 can be provided.

Further, the person tracking method according to one embodiment is a person tracking method for tracking a person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10, and the Web server 30, which is a computer, A human detection information extraction function unit serving as an information extraction unit that detects, from each of the monitoring images, multiple types of status information related to the health status of the person included in each of the monitoring images periodically acquired by each of the monitoring cameras 10. 22, multiple types of status information detected by the person detection information extraction function unit 22 are acquired, and based on the acquired multiple types of status information, the person included in each of the monitoring images corresponds to a prescribed symptom. It is determined whether the person is a specific person or not, and based on the result of the determination, an image of a specific person corresponding to a prescribed symptom is selected from among the images of the person extracted from the surveillance images of each of the plurality of surveillance cameras 10. An image of a person similar to the registered image of a person to be tracked is detected from each of the surveillance images that are registered as images of the person to be tracked in the file server 31, which is a memory, and periodically acquired by each of the plurality of surveillance cameras 10. , a tracking alert screen 43 is generated to present images of people similar to the detected images of the tracked person. This makes it possible to detect and track a person who has a prescribed symptom from the surveillance image acquired by the surveillance camera 10.

[Other embodiments]
In the embodiment described above, the detected person card 421 presented on the detection alert screen 42 is of a person whose body temperature, pulse rate, or cough symptom is higher than any of the presented thresholds, but if the detected person card 421 is higher than any of the presented thresholds It may be conditional on certain things. In this case, the processing procedure of the detection alert output function unit 3521 may be changed to the procedure shown in the flowchart shown in FIG. 23 instead of the procedure shown in the flowchart shown in FIG. 14B. That is, if it is determined in step S208 that the corrected body temperature is not equal to or higher than the specified body temperature, or if it is determined in step S211 that the corrected pulse rate is not equal to or greater than the specified pulse rate, or if the corrected cough symptom is determined to be equal to or higher than the specified cough symptom in step S212. If it is determined that this is not the case, the detection alert output function unit 3521 proceeds to the determination process of step S210 without creating a detected person card. Then, in step S208, it is determined that the corrected body temperature is equal to or higher than the specified body temperature, in step S211, it is determined that the corrected pulse rate is equal to or higher than the specified pulse rate, and in step S212, the corrected cough symptom is determined to be equal to or higher than the specified cough symptom. Only when it is determined that this is the case, the detection alert output function unit 3521 proceeds to step S209 and creates a detected person card.

In this way, the detection alert output function unit 3521 selects an image of a specific person for which status information equal to or higher than the presentation threshold is detected for all of the obtained presentation thresholds for each type as an image to be presented on the detection alert screen 42. do. Therefore, it is possible to easily narrow down people who are likely to have the specified symptoms.

Further, in the above embodiment, the past search data storage unit 311 of the file server 31 stores a frame image 3111, a whole body image 3112, a whole body feature amount 3113, and a face feature amount 3114 as past search data. However, the facial image from which the facial feature amount 3114 is extracted may also be stored.

Further, the order of the processing steps shown in the flowcharts of FIGS. 13 to 14B, FIGS. 17 to 20C, and FIG. 23 is an example, and the order is not limited to this order. For example, the processing in step S101 and the processing in step S102 in FIG. 13 may be performed in reverse order or may be performed in parallel. Further, in FIG. 18, the processing in steps S412 and S413 and the processing in steps S414 and S415 may be performed in reverse order or may be performed in parallel. Furthermore, the processing in steps S504 to S509 in FIG. 20A and the processing in steps S510 to S515 in FIG. 20B may be performed in reverse order or in parallel. In this way, the processing order and the like of each processing step may be changed as long as there is no discrepancy with the preceding or subsequent processing steps.

Further, the detection alert screen 42 shown in FIG. 15 (and FIG. 16) and the tracking alert screen 43 shown in FIG. 21 (and FIG. 22) are not limited to the layout and display content shown in the figures. For example, the detected cough symptoms 4216, the face caution stage display 4324, and the whole body caution stage display 4326 are expressed by the number of rectangles that are identified and displayed, but the color of one square is changed, for example, yellow, orange, or red. The caution level may be indicated by an identification display. Alternatively, instead of the number or color of rectangles, the caution levels may be expressed in letters, such as A to C or 1 to 3.

Further, in one embodiment, a full body image of a person is extracted from a surveillance image acquired by the surveillance camera 10, and a face image of a person is extracted from the extracted whole body image, but the present invention is not limited to this. . If a person's face is included in the monitoring image, the face image may be extracted without extracting the whole body image.

Further, in one embodiment, the attribute information of the person itself is digitized as the whole body image feature amount and the face image feature amount, but the present invention is not limited to this. The whole body feature amount may include, for example, a numerical representation of incidental information associated with a person, such as clothing, belongings (type and color of bag, stroller, etc.), accessories (sunglasses, mask, etc.). Further, the whole body feature amount may include numerical information such as fever information, pulse rate, etc., which is obtained from an image of the person or from other sensors. Further, the facial feature amount may include numerical values of incidental information attached to the face, such as accessories (glasses, hats, etc.).

In addition, the whole body image feature amount and face image feature amount are only those that quantify the attribute information of the person itself, and those that quantify the accompanying information and the person's internal information are treated as other feature quantities. Another feature amount may be used as one of the detection criteria similar to the whole body image feature amount and the face image feature amount.

In short, the present invention is not limited to the above-described embodiments as they are, but can be implemented by modifying the constituent elements within the scope of the invention at the implementation stage. Moreover, various inventions can be formed by appropriately combining the plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be combined as appropriate.

1... Person tracking system 10... Surveillance camera 20... Video analysis function unit 21... Image acquisition module 22... Person detection information extraction function unit 23... Monitoring/tracking execution function unit 30... Web server 31... File server 32... Database (DB) Server 33...Search human detection result storage function unit 34...Detection/tracking result determination function unit 35...Web application 40...Monitoring terminal 41...Web browser 42...Detection alert screen 43...Tracking alert screen 221...Full body detection module 222...Area tracking Module 223...Whole body feature extraction module 224...Face detection module 225...Face feature extraction module 226...Body temperature detection module 227...Pulse detection module 228...Cough detection module 231...Whole body matching module 232...Face matching module 251...Tracking table 301A ...Hardware processor 301B...Program memory 302...Storage device 303...Communication interface device 304...Bus 311...Data storage section for past search 312...Detection history data storage section 313...Management data storage section 321...Camera information table storage section 322... Past search data table storage section 323... Detection history table storage section 324... Management table storage section 325... Tracking table storage section 351... Login function section 352... Monitoring function section 353... Tracking function section 354... Past search function section 421... Detection Person card 422, 433... Refinement instruction area 431... Tracking target person information area 432... Tracking information card 3111... Frame image 3112... Whole body image 3113, 3124, 3134... Whole body feature amount 3114, 3122, 3132... Face feature amount 3115... Body temperature 3116...Pulse rate 3117...Cough symptoms 3121...Detected face image 3123...Detected whole body image 3131...Registered face image 3133...Registered whole body image 3211...Camera information table 3221...Data table for past search 3231...Detection history table 3241...Management table 3251 ...Tracking table 3521...Detection alert output function section 3531...Image registration function section 3532...Tracking alert output function section 4211, 4311...Caution label 4212, 4312...Attention required person ID
4213... Tracking candidate image 4214... Detected body temperature 4215... Detected pulse rate 4216... Detected cough symptoms 4217... Tracking start button 4221... Body temperature presentation threshold setting slider 4222... Pulse rate presentation threshold setting slider 4223... Cough symptom threshold setting button 4313... Registration Image 4314...Tracking information 4315...Tracking stop button 4321...Detection date and time information 4322...Detected camera location information 4323...Detected face image 4324...Face caution stage display 4325...Detected whole body image 4326...Whole body caution stage display 4327...Detected background image 4331... Face image similarity selection button 4332...Face image similarity selection reset button 4333...Whole body image similarity selection button 4334...Whole body image similarity selection reset button NET...Network SV...Server device

Claims (11)

The information extraction section detects from each of the surveillance images multiple types of status information related to the health condition of the person included in each of the surveillance images periodically acquired by each of the multiple surveillance cameras. Obtaining the plurality of types of detected status information, and determining whether the person included in each of the monitoring images is a specific person corresponding to a prescribed symptom based on the acquired plurality of types of status information. a discriminating unit to
Based on the discrimination result of the discrimination unit, among the images of the person extracted from the surveillance images of each of the plurality of surveillance cameras, an image of the specific person corresponding to the prescribed symptoms is selected as a tracking target person image. A registration department to register,
a detection unit that detects an image of a person similar to the tracking target person image registered by the registration unit from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras;
a tracking output screen generation unit that generates a tracking output screen for presenting an image of a person similar to the tracking target person image detected by the detection unit;
A person tracking device equipped with Based on the discrimination result of the discrimination unit, an image of the specific person corresponding to the prescribed symptom is presented together with the condition information among the images of the person extracted from the surveillance images of each of the plurality of surveillance cameras. further comprising a detection output screen generation unit that generates a detection output screen for
The registration unit specifies an image of a person to be tracked to be tracked from among the images of the person presented on the detection output screen from a monitoring terminal that displays the detection output screen generated by the detection output screen generation unit. The person tracking device according to claim 1, wherein the acquired specified image is registered as a tracking target person image. When any one of the acquired plurality of types of status information is equal to or greater than a predetermined value, the determination unit determines that the image of the person extracted from the monitoring images of each of the plurality of monitoring cameras is The person tracking device according to claim 2, wherein the person tracking device determines that the image is of the specific person who corresponds to a prescribed symptom. The detection output screen generation unit acquires a presentation threshold specified for each of the plurality of types of status information, and based on the acquired presentation threshold, the discrimination unit determines that the specified symptom corresponds to the specified symptom. The person tracking device according to claim 2 or 3, wherein images to be presented on the detection output screen are narrowed down from among the determined images of the specific person. The detection output screen generation unit presents, on the detection output screen, an image of the specific person whose state information is detected to be equal to or higher than any one of the acquired presentation thresholds for each of the types. The person tracking device according to claim 4, which is selected as the target image. The detection output screen generation unit is configured to select, as an image to be presented on the detection output screen, an image of the specific person for which state information greater than or equal to the presentation threshold has been detected for all of the acquired presentation thresholds for each of the types. The person tracking device according to claim 4, which selects the person. Further comprising a camera information storage unit storing correction coefficients for correcting the state information for each of the plurality of surveillance cameras,
The determining unit corrects the status information acquired from the information extracting unit using the correction coefficient stored in the camera information storage unit, and based on the corrected status information, the image of the person is determined to be within the specified range. The person tracking device according to any one of claims 1 to 6, which determines whether or not the image is of the specific person corresponding to a symptom. 8. The person tracking device according to claim 1, wherein the state information includes at least one of body temperature, pulse rate, and cough symptoms, each extracted from the monitoring image. A person tracking device according to any one of claims 1 to 8,
the plurality of surveillance cameras;
A monitoring terminal operated by a supervisor,
an analysis section including the information extraction section;
A person tracking system equipped with A person tracking method for tracking a person to be tracked from surveillance images obtained by multiple surveillance cameras, the method comprising:
The computer is
The information extraction section detects from each of the surveillance images multiple types of status information related to the health condition of the person included in each of the surveillance images periodically acquired by each of the multiple surveillance cameras. Obtaining the plurality of types of detected status information, and determining whether the person included in each of the monitoring images is a specific person corresponding to a prescribed symptom based on the acquired plurality of types of status information. death,
Based on the result of the discrimination, among the images of the person extracted from the monitoring images of each of the plurality of monitoring cameras, an image of the specific person corresponding to the prescribed symptoms is stored in a memory as a tracking target person image. Register and
detecting an image of a person similar to the registered tracking target person image from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras;
generating a tracking output screen for presenting an image of a person similar to the detected image of the tracked person;
How to track people. A person tracking program that causes a computer to operate as each part of the person tracking device according to any one of claims 1 to 8.