JP7642882B2 - SPECIFIC PERSON DETECTION DEVICE, SPECIFIC PERSON DETECTION SYSTEM, SPECIFIC PERSON DETECTION METHOD, AND SPECIFIC PERSON DETECTION PROGRAM - Google Patents

Description

One aspect of the present invention relates to a specified person detection device, a specified person detection system, a specified person detection method, and a specified person detection program.

Various technologies have been proposed to extract a person's face from a surveillance image captured by a surveillance camera and identify the person based on the features of that face. Furthermore, in recent years, a technology has been proposed that improves the accuracy of person identification by combining the features of the face and the features of the entire body to identify the person (see, for example, Patent Document 1).

JP 2019-023785 A

Conventional person identification technology is extremely effective for detecting the presence of suspicious individuals or regular customers when facial images of visitors can be reliably captured, such as by surveillance cameras installed at the entrances of buildings, stores, etc.

On the other hand, in recent years, there has been a strong demand for measures against infectious diseases such as influenza and COVID-19 (novel coronavirus disease). Therefore, there is a demand for technology that can detect the arrival of people suspected of having such an infectious disease, and furthermore, track the movements of such people within a facility.

This invention was made with the above in mind, and its purpose is to make it possible to detect and track people who exhibit prescribed symptoms corresponding to infectious diseases, etc. from surveillance images captured by surveillance cameras.

In order to achieve the above-mentioned object, one aspect of the specified person detection device of the present invention includes an information extraction unit that detects multiple types of status information related to the health condition of a person contained in each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, and a discrimination unit that acquires the multiple types of status information detected by the information extraction unit from each of the surveillance images, and determines whether the person contained in each of the surveillance images is a specified person corresponding to a specified symptom based on the acquired multiple types of status information, wherein the discrimination unit is configured to determine that the image of the person extracted from the surveillance images of each of the plurality of surveillance cameras is an image of the specified person corresponding to the specified symptom when any one of the acquired multiple types of status information is equal to or greater than a predetermined value .

One aspect of the specified person detection system of the present invention includes one aspect of the specified person detection device of the present invention, the multiple surveillance cameras, and an analysis unit that includes the information extraction unit.

One aspect of the specific person detection method of the present invention is a specific person detection method for detecting a person to be tracked from surveillance images acquired by a plurality of surveillance cameras, in which a computer acquires multiple types of status information related to the health condition of a person contained in each of the surveillance images acquired periodically by each of the plurality of surveillance cameras from an information extraction unit that detects the multiple types of status information from each of the surveillance images, and when any one of the acquired multiple types of status information is equal to or greater than a predetermined value, the computer determines that the image of the person extracted from the surveillance images of each of the plurality of surveillance cameras is an image of a specific person corresponding to a specified symptom.

One aspect of the specified person detection program of the present invention is a program that causes a computer to operate as each part of one aspect of the specified person detection device of the present invention.

According to each aspect of the present invention, it becomes possible to detect and track people who meet certain symptoms from surveillance images captured by a surveillance camera.

FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of the video analysis function unit shown in FIG. FIG. 3 is a block diagram showing the software configuration of the Web server shown in FIG. 1 as the person tracking device according to one embodiment of the present invention. FIG. 4 is a diagram showing an example of the contents stored in the past search data storage unit shown in FIG. FIG. 5 is a diagram illustrating an example of the contents stored in the detection history data storage unit illustrated in FIG. 3. As illustrated in FIG. FIG. 6 is a diagram illustrating an example of the contents stored in the management data storage unit illustrated in FIG. FIG. 7 is a diagram showing an example of the contents of the camera information table storage unit shown in FIG. FIG. 8 is a diagram showing an example of the contents of the past search data table storage unit shown in FIG. FIG. 9 is a diagram showing an example of the contents of the detection history table storage unit shown in FIG. 3. As shown in FIG. FIG. 10 is a diagram showing an example of the contents of the management table storage unit shown in FIG. 3. As shown in FIG. FIG. 11 is a diagram showing an example of the contents of the tracing table storage unit shown in FIG. 3. As shown in FIG. FIG. 12 is a block diagram showing the hardware configuration of the Web server of FIG. FIG. 13 is a flowchart showing a processing procedure of the detection result determination process performed by the detection/tracking result determination function unit shown in FIG. FIG. 14A is a flowchart showing a first portion of the processing procedure of the detection alert output processing by the detection alert output function unit shown in FIG. 3. FIG. 14B is a flowchart showing a second part of the processing procedure of the detection alert output processing by the detection alert output function unit shown in FIG. 3. FIG. 15 is a diagram showing an example of a detection alert screen displayed on the Web browser of the monitoring terminal shown in FIG. FIG. 16 is a diagram showing another example of the detection alert screen displayed on the Web browser of the monitoring terminal shown in FIG. FIG. 17 is a flowchart showing a processing procedure of the image registration process by the image registration function unit shown in FIG. FIG. 18 is a flowchart showing a procedure of the tracing process performed by the monitoring/tracing execution function unit shown in FIG. FIG. 19 is a flowchart showing a processing procedure of the tracking result determination process performed by the detection/tracking result determination function unit shown in FIG. FIG. 20A is a flowchart showing a first part of the processing procedure of the tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. 20B is a flowchart showing a second part of the processing procedure of the tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. 20C is a flowchart showing a third part of the processing procedure of the tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. 21 is a diagram showing an example of a tracking alert screen displayed on the Web browser of the monitoring terminal shown in FIG. FIG. 22 is a diagram showing another example of the tracking alert screen displayed on the Web browser of the monitoring terminal shown in FIG. FIG. 23 is a flowchart showing a part of a processing procedure of a detection alert output process by a detection alert output function unit in the modified example of the person tracking system.

The following describes an embodiment of the present invention with reference to the drawings.

[One embodiment]
(1) Configuration (1-1) Overall Configuration FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system 1 according to an embodiment of the present invention.

The person tracking system 1 is a system that tracks a specific person in a large facility with a wide surveillance area, such as an office building, a department store, or other building with multiple floors, or a commercial facility with multiple stores.

This person tracking system 1 includes a plurality of surveillance cameras 10, a server device SV equipped with a video analysis function unit 20 provided corresponding to each surveillance camera 10 and a web server 30 that functions as a person tracking device according to an embodiment of the present invention, and a surveillance terminal 40. Each surveillance camera 10, the server device SV, and the surveillance terminal 40 are connected via a network NET.

The network NET is an in-house network, for example a wireless LAN (Local Area Network) or a wired LAN. The network NET may also be a wide area network such as the Internet.

The multiple surveillance cameras 10 are network cameras that are distributed throughout a large facility so that their shooting ranges do not overlap or overlap partially. The surveillance cameras 10 may be video cameras that capture moving images within their shooting ranges, and each frame image that constitutes the moving image can be acquired as a surveillance image of the shooting range. The surveillance cameras 10 may also be still cameras that capture still images at regular time intervals to periodically acquire surveillance images.

Each of the multiple video analysis function units 20 arranged in the server device SV has various analysis functions for the surveillance images acquired from the corresponding surveillance camera 10. The analysis functions provided by the video analysis function unit 20 will be described in detail later. Note that the video analysis function unit 20 is not arranged inside the server device SV in this way, but can be arranged independently on a dedicated computing device or cloud, and data can be exchanged between the corresponding surveillance camera 10 and the server device SV via the network NET. Also, the video analysis function unit 20 can be arranged inside the surveillance camera 10 if the surveillance camera 10 has a computing function. Note that in this embodiment, the surveillance camera 10 and the video analysis function unit 20 are configured in one-to-one correspondence, but one video analysis function unit 20 may be assigned to multiple surveillance cameras 10.

The web server 30 of the server device SV is configured by a server computer, and includes as its functional units a file server 31, a database (hereinafter abbreviated as DB) server 32, a web application 35, a search-use person detection result storage function unit 33, and a detection/tracking result determination function unit 34. Details of each of these functional units will be described later.

The monitoring terminal 40 is configured by a computer such as a personal computer, and can execute a program of a web browser 41. The web browser 41 allows a supervisor such as a security guard who operates the monitoring terminal 40 to view person tracking results provided by a web application 35 of the web server 30. Note that although only one monitoring terminal 40 is shown in FIG. 1, the person tracking system 1 may include multiple monitoring terminals 40.

The server device SV may also have the functionality of the monitoring terminal 40.

(1-2) Video analysis function unit 20
Fig. 2 is a block diagram showing an example of the configuration of the video analysis function unit 20 shown in Fig. 1. The video analysis function unit 20 includes an image acquisition module 21, a human detection information extraction function unit 22, and a monitoring/tracking execution function unit 23.

The image acquisition module 21 acquires frame images transmitted from the surveillance camera 10 via the network NET in accordance with a predetermined communication protocol such as RTSP (Real Time Streaming Protocol). The image acquisition module 21 transmits the acquired frame images to the Web server 30 via intra-server communication or WebSocket, etc., and stores them in the file server 31. Note that if the video analysis function unit 20 is disposed independently of the server device SV, the image acquisition module 21 transmits the acquired frame images to the Web server 30 via the network NET. The image acquisition module 21 also sends the acquired frame images to the human detection information extraction function unit 22.

The human detection information extraction function unit 22 functions as an extraction unit that extracts a whole-body image of a person from an input image, extracts whole-body features and a facial image of the person from the whole-body image, and further extracts facial features from the facial image. Specifically, the human detection information extraction function unit 22 includes a whole-body detection module 221, an area tracking module 222, a whole-body feature extraction module 223, a facial detection module 224, a facial feature extraction module 225, a body temperature detection module 226, a pulse detection module 227, and a cough detection module 228.

The whole body detection module 221 is a whole body image extraction unit that detects the whole body of each person appearing in the frame image, which is an input image, and extracts a whole body image. The whole body detection module 221 can detect the whole body of a person based on a whole body image of the person that has been learned in advance by machine learning such as deep learning. The whole body detection module 221 transmits the extracted whole body image to the Web server 30 by intra-server communication or WebSocket, etc., and accumulates it in the file server 31. Note that, when the video analysis function unit 20 is arranged independently of the server device SV, the whole body detection module 221 transmits the extracted whole body image to the Web server 30 via the network NET. The whole body detection module 221 also transmits the extracted whole body image to the whole body feature extraction module 223, the face detection module 224, and the body temperature detection module 226. Furthermore, the whole body detection module 221 transmits the frame image and the extracted whole body image to the area tracking module 222.

The area tracking module 222 detects the area position of the whole body image in the frame image input from the whole body detection module 221.

The whole-body feature extraction module 223 is a whole-body feature extraction unit that extracts whole-body features from the input whole-body image. The whole-body feature is an objective quantification of the body features of a person contained in the whole-body image. For example, the whole-body feature can be a quantification of attribute information of the person himself, such as body type, height, sex, age, etc., estimated from the image. The whole-body feature extraction module 223 can extract whole-body features using a pre-trained model such as deep learning (e.g., Deep-person-reid, etc.). The whole-body feature extraction module 223 can also calculate these whole-body features as local image features such as HOG (Histogram of Oriented Gradients) and SIFT (Scaled Invariance Feature Transform). The whole-body feature extraction module 223 transmits the extracted whole-body features to the Web server 30 via intra-server communication or Websocket, etc., and accumulates them in the file server 31. If the video analysis function unit 20 is disposed independently of the server device SV, the whole-body feature extraction module 223 transmits the extracted whole-body feature to the web server 30 via the network NET. Furthermore, the whole-body feature extraction module 223 transmits the extracted whole-body feature to the monitoring/tracking execution function unit 23.

The face detection module 224 is a face image extraction unit that detects a person's face from an input whole-body image and extracts a face image. The face detection module 224 can detect a person's face based on a face image of a person that has been learned in advance by machine learning such as deep learning. The face detection module 224 sends the extracted face image to the face feature extraction module 225, the body temperature detection module 226, the pulse detection module 227, and the cough detection module 228.

The facial feature extraction module 225 is a facial feature extraction unit that extracts facial features from an input facial image. The facial feature is an objective numerical representation of the facial features of a person included in a facial image. The facial feature extraction module 225 can extract facial features using a model previously trained by deep learning, for example, in the same way as the whole-body feature extraction module 223 that extracts whole-body features. The face detection module 224 may also calculate these facial features as local image features such as HOG and SIFT. The face detection module 224 transmits the extracted facial features to the Web server 30 via intra-server communication or Websocket, and accumulates them in the file server 31. In addition, when the video analysis function unit 20 is arranged independently of the server device SV, the facial feature extraction module 225 transmits the extracted facial features to the Web server 30 via the network NET. Furthermore, the face detection module 224 sends the extracted face features to the monitoring/tracking execution function unit 23.

The body temperature detection module 226 detects the body surface temperature of a person from the whole body image input from the whole body detection module 221 or the face image input from the face detection module 224. Various methods for detecting body surface temperature from an image have been proposed in the past, and the body surface temperature detection method used in this embodiment is not particularly limited. The body temperature detection module 226 transmits the detected body temperature to a web server via intra-server communication, web socket, or the like. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the body temperature detection module 226 transmits the detected body temperature to the web server 30 via the network NET.

The pulse detection module 227 detects the pulse rate of a person from the face image input from the face detection module 224. Techniques for detecting the pulse rate from an image are proposed in, for example, the following reference documents 1 to 3.
[Reference 1] Issei Obara and 4 others, "Extraction of pulse wave information from video" [Online], [Retrieved June 19, 2020], Internet <URL: https://www.sairct.idac.tohoku.ac.jp/wp-content/uploads/2016/07/160728_c2_yoshizawa_DLpre2.pdf>
[Reference 2] Masahide Abe and two others, "Measurement of pulse rate and respiratory rate using non-contact sensors" [Online], [Retrieved June 19, 2020], Internet <URL: http://www.mk.ecei.tohoku.ac.jp/papers/data/F05760013.pdf>
[Reference 3] “Technology developed to measure pulse rate in real time from facial images” [Online], [Retrieved June 19, 2020], Internet <URL: https://pr.fujitsu.com/jp/news/2013/03/18.html>
In this embodiment, the pulse rate detection method used is not particularly limited. The pulse rate detection module 227 transmits the detected pulse rate to the Web server by intra-server communication, WebSocket, etc. If the video analysis function unit 20 is arranged independently of the server device SV, the pulse rate detection module 227 transmits the detected pulse rate to the Web server 30 via the network NET.

The cough detection module 228 detects a person's cough symptoms from the face image input from the face detection module 224. The cough detection module 228 detects whether or not a person is coughing and the severity of the cough from, for example, the degree of blurring of the face image or facial changes in multiple consecutive face images. The cough detection module 228 can detect cough symptoms based on, for example, face images of a person that have been trained in advance by machine learning such as deep learning. The cough detection module 228 transmits the detected cough symptoms indicating the severity of the cough symptoms to the web server via intra-server communication, web socket, or the like. Note that when the video analysis function unit 20 is arranged independently of the server device SV, the cough detection module 228 transmits the detected cough symptoms to the web server 30 via the network NET.

The monitoring/tracking execution function unit 23 also functions as a similarity calculation unit that calculates a whole-body image similarity, which is the similarity of the whole-body features of the two input images, and a face image similarity, which is the similarity of the face features of the two input images. Specifically, the monitoring/tracking execution function unit 23 includes a whole-body matching module 231 and a face matching module 232.

The whole-body matching module 231 calculates the similarity between the whole-body features input from the whole-body feature extraction module 223 and the whole-body features of the person to be tracked instructed from the monitoring terminal 40. The whole-body features of the person to be tracked are, for example, extracted in advance by the whole-body feature extraction module 223 and registered in the file server 31 of the Web server 30. The whole-body matching module 231 transmits the calculated similarity to the Web server 30 by intra-server communication, WebSocket, or the like. Note that when the video analysis function unit 20 is arranged independently of the server device SV, the whole-body matching module 231 transmits the calculated similarity to the Web server 30 via the network NET.

The face matching module 232 calculates the similarity between the facial features input from the facial feature extraction module 225 and the facial features of the person to be tracked instructed by the monitoring terminal 40. The facial features of the person to be tracked are, for example, extracted in advance by the facial feature extraction module 225 and registered in the file server 31 of the Web server 30. The face matching module 232 transmits the calculated similarity to the Web server 30 by intra-server communication, WebSocket, or the like. Note that when the video analysis function unit 20 is arranged independently of the server device SV, the face matching module 232 transmits the calculated similarity to the Web server 30 via the network NET.

(1-3) Web server 30
3 is a block diagram showing the software configuration of the web server 30. As described above, the web server 30 has, as functional units, a file server 31, a DB server 32, a search-purpose human detection result storage function unit 33, a detection/tracking result determination function unit 34, and a web application 35.

The file server 31 accumulates various data files. The file server 31 can include a past search data storage unit 311, a detection history data storage unit 312, and a management data storage unit 313.

The past search data storage unit 311 stores past search data, which is data acquired by each of the multiple video analysis function units 20. FIG. 4 is a diagram showing an example of past search data stored in the past search data storage unit 311. The past search data can be stored in the past search data storage unit 311 for each whole-body image detected from a frame image by the whole-body detection module 221 of each video analysis function unit 20. The past search data can include a frame image 3111 acquired by the image acquisition module 21 of the video analysis function unit 20, a whole-body image 3112 extracted by the whole-body detection module 221, a whole-body feature 3113 extracted by the whole-body feature extraction module 223, and a face feature 3114 extracted by the face feature extraction module 225. The past search data can include a body temperature 3115, a pulse rate 3116, and a cough symptom 3117, which are status information detected by the whole-body detection module 221 of the video analysis function unit 20. The frame image 3111, whole-body image 3112, whole-body features 3113, facial features 3114, body temperature 3115, pulse rate 3116, and cough symptoms 3117 can be linked together, for example, by storing them in the same path or by adding the same character string to part of the file name.

The detection history data storage unit 312 stores detection history data, which is data related to each specific person detected as a tracking candidate who corresponds to a specified symptom such as influenza. FIG. 5 is a diagram showing an example of detection history data of a detected specific person stored in the detection history data storage unit 312. The detection history data can include a detected face image 3121 which is a face image of the detected specific person, a face feature amount 3122 corresponding to the detected face image 3121, a detected whole body image 3123 which is a whole body image of the detected specific person, and a whole body feature amount 3124 corresponding to the detected whole body image 3123. The detected face image 3121, the face feature amount 3122, the detected whole body image 3123, and the whole body feature amount 3124 can be linked by, for example, storing them in the same path or adding the same character string to part of the file name.

The information in the detection history data can be stored in the detection history data storage unit 312 when the Web server 30 detects, for example, a tracking candidate who meets a specified body temperature, pulse rate, or coughing symptoms from a surveillance image of any of the surveillance cameras 10. That is, the Web server 30 can store the whole-body image 3112, whole-body feature amount 3113, and facial feature amount 3114 acquired from the surveillance image and stored in the past search data storage unit 311 in the detection history data storage unit 312 as the detected whole-body image 3123, whole-body feature amount 3124, and facial feature amount 3122. In addition, the Web server 30 can input the whole body image 3112 acquired from the surveillance image and stored in the past search data storage unit 311 to the face detection module 224 of one of the video analysis function units 20 via intra-server communication, WebSocket, or the like, to acquire a face image from the face detection module 224 and store it in the detection history data storage unit 312 as a detected face image 3121. This one video analysis function unit 20 may be the video analysis function unit 20 corresponding to the surveillance camera 10 that acquired the surveillance image in which the tracking candidate was detected. If the face detection module 224 of the video analysis function unit 20 buffers the extracted face image, the Web server 30 may acquire the detected face image 3121 by requesting the buffered face image from the face detection module 224.

The management data storage unit 313 stores management data related to each tracking target. FIG. 6 is a diagram showing an example of management data for each tracking target stored in the management data storage unit 313. The management data for each tracking target can be stored in the management data storage unit 313 in response to a designation from the monitoring terminal 40. The management data can include a registered face image 3131 which is a face image of the designated tracking target, a facial feature amount 3132 which is a feature extracted from the registered face image 3131, a registered whole-body image 3133 which is a whole-body image of the tracking target, and a whole-body feature amount 3134 which is a feature extracted from the registered whole-body image 3133. The registered face image 3131, the facial feature amount 3132, the registered whole-body image 3133, and the whole-body feature amount 3134 can be linked together, for example, by storing them in the same path or adding the same character string to part of the file name.

For example, when a tracking candidate is detected, the Web server 30 allows the monitor to view the detected face image 3121 in the detection history data for the monitoring target stored in the detection history data storage unit 312 and the status information that was the source of the detection on the Web browser 41 of the monitoring terminal 40. The status information to be viewed here is the body temperature 3115, pulse rate 3116, and cough symptoms 3117 stored in the past search data storage unit 311, corrected by a correction value for each monitoring camera 10 that depends on the type and performance, etc. Then, when the person is designated as a tracking target from the monitoring terminal 40, the Web server 30 stores the detected face image 3121 in the detection history data as the registered face image 3131 in this management data. Furthermore, the Web server 30 can store the facial feature amount 3122 in the detection history data as the facial feature amount 3132 in this management data. In addition, the Web server 30 acquires the whole-body image that is the source of the detected face image 3121 in the specified detection history data from the past search data stored in the past search data storage unit 311, and also acquires the whole-body features from the past search data, and stores them as the registered whole-body image 3133 and whole-body features 3134 in this management data.

Alternatively, the Web server 30 may allow the Web browser 41 of the monitoring terminal 40 to view the whole-body images 3112, body temperature 3115, pulse rate 3116, and cough symptoms 3117 of multiple people stored in the past search data storage unit 311, and may receive a designation of an arbitrary whole-body image from the monitoring terminal 40 to designate that person as a designated tracking target. In this case, the Web browser 41 may allow the results of correcting the body temperature 3115, pulse rate 3116, and cough symptoms 3117 using a correction value for the corresponding monitoring camera 10 to be viewed. If a tracking target is designated, the Web server 30 may input the designated whole-body image 3112 to the face detection module 224 of one video analysis function unit 20 via intra-server communication or WebSocket, etc., to obtain a face image from the face detection module 224 and store it in the management data storage unit 313 as a registered face image 3131. In this case, the facial feature amount 3132, the registered whole-body image 3133, and the whole-body feature amount 3134 may be obtained and stored from the past search data storage unit 311.

The DB server 32 accumulates various data tables. The DB server 32 may include a camera information table storage unit 321, a past search data table storage unit 322, a detection history table storage unit 323, a management table storage unit 324, a tracking table storage unit 325, etc.

The camera information table storage unit 321 is provided corresponding to each of the multiple surveillance cameras 10, and pre-stores a camera information table 3211 in which various information related to the corresponding surveillance camera 10 is described. FIG. 7 is a diagram showing an example of the contents of the camera information table 3211 for each surveillance camera 10 stored in the camera information table storage unit 321. The camera information table 3211 contains, for example, the camera ID, camera name, aspect ratio, camera position X, camera position Y, camera angle, body temperature correction coefficient, pulse correction coefficient, cough symptom correction coefficient, face matching thresholds 1 to 3, whole body matching thresholds 1 to 3, etc.

Here, the camera ID is identification information that uniquely identifies the corresponding surveillance camera 10. The camera name is the name of the corresponding surveillance camera 10. This can be a name associated with the installation position of the corresponding surveillance camera 10 in a large facility. The aspect ratio is the aspect ratio of the image captured by the corresponding surveillance camera 10. The camera position X and the camera position Y are XY coordinates within the large facility that indicate the installation position of the corresponding surveillance camera 10. The camera angle indicates the installation direction of the corresponding surveillance camera 10.

The body temperature correction coefficient, pulse correction coefficient, and cough symptom correction coefficient are coefficients for correcting the body temperature, pulse rate, and cough symptoms detected by the body temperature detection module 226, pulse detection module 227, and cough detection module 228 of the video analysis function unit 20. These correction coefficients depend on the type, performance, installation conditions, etc. of the corresponding surveillance camera 10 itself, so they are adjusted to appropriate values between the installation of the surveillance camera 10 and the start of actual operation, and are recorded in the camera information table 3211.

Facial matching thresholds 1-3 and whole body matching thresholds 1-3 are thresholds for comparison with the similarity calculated by the whole body matching module 231 and the face matching module 232 of the video analysis function unit 20. The face matching threshold 1 is a first threshold corresponding to a facial image similarity for classifying whether or not the facial image of a person extracted from the surveillance image of the corresponding surveillance camera 10 is the facial image of the tracking target. If the facial image similarity calculated by the face matching module 232 is equal to or greater than the first threshold, the face matching threshold 1, the Web server 30 can determine that the facial image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the facial image of the tracking target. The face matching thresholds 2 and 3 correspond to a facial image similarity greater than the face matching threshold 1, and are second thresholds for classifying the facial image into multiple stages when the extracted facial image of the person is the facial image of the tracking target. In other words, the magnitudes of these thresholds have the relationship of [face matching threshold 1 < face matching threshold 2 < face matching threshold 3]. Similarly, the whole-body matching threshold 1 is a first threshold corresponding to the whole-body image similarity for classifying whether or not the whole-body image of a person extracted from the surveillance image of the corresponding surveillance camera 10 is the whole-body image of the tracking target. If the whole-body image similarity calculated by the whole-body matching module 231 is equal to or greater than the whole-body matching threshold 1, the Web server 30 can determine that the whole-body image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the whole-body image of the tracking target. The whole-body matching thresholds 2 and 3 correspond to whole-body image similarities greater than the whole-body matching threshold 1, and are second thresholds for classifying the whole-body image of the extracted person into a plurality of stages when the whole-body image is the whole-body image of the surveillance target or the tracking target. In other words, the magnitudes of these thresholds have the relationship of [whole-body matching threshold 1 < whole-body matching threshold 2 < whole-body matching threshold 3].

The value of each threshold is generally determined by the matching model used by the face matching module 232 and the whole body matching module 231 in the monitoring/tracking execution function unit 23 of the video analysis function unit 20 that calculates the similarity. Since each threshold depends on the type, performance, installation conditions, etc. of the surveillance camera 10 itself, it is adjusted to an appropriate value between the installation of the surveillance camera 10 and the start of actual operation, and is recorded in the camera information table 3211. It should be noted that the face matching threshold and the whole body matching threshold are not limited to three levels each. In other words, it is sufficient that there is at least one second threshold.

The past search data table storage unit 322 stores a past search data table 3221 in which various data related to the past search data is described, corresponding to each past search data stored in the past search data storage unit 311. When storing past search data in the past search data storage unit 311, the Web server 30 creates this past search data table 3221 and stores it in the past search data table storage unit 322. FIG. 8 is a diagram showing an example of the contents of the past search data table 3221 stored in the past search data table storage unit 322. The past search data table 3221 contains, for example, a detection ID, a detection date and time, a detection camera ID, a tracking ID, a frame ID, detection coordinate information X, detection coordinate information Y, detection coordinate information Width, detection coordinate information Height, and the like.

Here, the detection ID is identification information assigned to each full-body image 3112 in the frame image 3111 detected by the full-body detection module 221 of the video analysis function unit 20. The detection date and time is the date and time when the full-body detection module 221 detected the full-body image 3112. The detection camera ID is the camera ID of the surveillance camera 10 that acquired the frame image 3111. The tracking ID is identification information for linking the full-body images 3112 of the same person between multiple frame images. The frame ID is identification information for uniquely identifying the frame image 3111. The detection coordinate information X, detection coordinate information Y, detection coordinate information Width, and detection coordinate information Height are information indicating the area position of the full-body image 3112 in the frame image 3111 detected by the area tracking module 222, and indicate, for example, the XY coordinates in the frame image of the upper left corner of the full-body image 3112, and the image width and image height from there.

The Web server 30 can assign the tracking ID of the past search data table 3221, for example, as follows. The Web server 30 compares the area position of the whole body image provided by the human detection information extraction function unit 22 of the video analysis function unit 20 with the area position of each whole body image in the past search data table 3221 in the previous frame image of the surveillance camera 10 stored in the past search data storage unit 311, and determines the identity of the person in the whole body image by considering overlapping areas, etc. If the same person is not in the previous frame image, the Web server 30 assigns a new tracking ID. Also, if the same person is in the previous frame image, the tracking ID is taken over from the past search data table 3221 corresponding to the whole body image 3112 of the person. The Web server 30 may determine the identity of the person by comparing the whole body features instead of the position of the whole body image, and determine whether or not to assign a new tracking ID.

The detection history table storage unit 323 stores a detection history table 3231 in which various data related to the detection history data is described, corresponding to each of the detection history data stored in the detection history data storage unit 312. When storing detection history data in the detection history data storage unit 312, the Web server 30 creates this detection history table 3231 and stores it in the detection history table storage unit 323. FIG. 9 is a diagram showing an example of the contents of the detection history table 3231 stored in the detection history table storage unit 323. The detection history table 3231 describes, for example, the detection ID, detection date and time, detection camera ID, suspicious person ID, corrected body temperature, corrected pulse rate, corrected cough symptoms, face left-right angle, face up-down angle, tracking ID, etc.

Here, the detection ID, detection date and time, detection camera ID, and tracking ID are as explained for the past search data table 3221. The suspect ID is identification information that uniquely identifies the tracking candidate and is assigned when the detection history data is stored in the detection history data storage unit 312. The corrected body temperature, corrected pulse rate, and corrected cough symptoms are the results of correction using the correction coefficients written in the camera information table 3211 stored in the camera information table storage unit 321. The face left-right angle and face up-down angle indicate the direction of the face in the detected face image 3121 of the corresponding detection history data.

The management table storage unit 324 stores a management table 3241 in which various data related to each piece of management data stored in the management data storage unit 313 is described. When storing management data in the management data storage unit 313, the web server creates this management table 3241 and stores it in the management table storage unit 324. FIG. 10 is a diagram showing an example of the contents of the management table 3241 stored in the management table storage unit 324. The management table 3241 describes, for example, a person to watch out for ID, corrected body temperature, corrected pulse rate, corrected cough symptoms, query face image path, face feature file path, monitoring status, pinning flag, long-term stay flag, and the like.

Here, the suspicious person ID, corrected body temperature, corrected pulse rate, and corrected cough symptoms are transcribed from the detection history table 3231. The query face image path and face feature file path indicate the storage paths of the registered face image 3131 and face feature 3132 of the management data in the management data storage unit 313. The monitoring status is attribute information set arbitrarily by the monitoring terminal 40 for the tracking target person other than the danger level type. The pinning flag is a flag that is set to indicate that the person is designated as a tracking target. The long-term stay flag is a flag that is set when the person is not designated as a tracking target but has stayed in the facility for a specified time or longer. In other words, when a person who is not a monitoring target but has been staying for a long time exists, the Web server 30 can store the face image and face feature of the person in the management data storage unit 313 and store the management table in the management table storage unit 324. This makes it possible to discover new suspicious persons and candidates for new customers.

The tracking table storage unit 325 stores a tracking table that describes the similarity of a person extracted from a surveillance image of any of the surveillance cameras 10, which is determined by the Web server 30 to be a whole-body image or a face image of a tracking target from the whole-body image similarity or face image similarity calculated by the whole-body matching module 231 or face matching module 232 of the video analysis function unit 20. FIG. 11 is a diagram showing an example of the contents of the tracking table 3251 stored in the tracking table storage unit 325. The tracking table 3251 includes a record consisting of a detection ID, a whole-body image similarity, and a face image similarity for each suspicious person ID of the tracking target, and each record is added each time the Web server 30 determines that the image is a whole-body image or a face image of the tracking target. The whole body image similarity and face image similarity are stored as real value values calculated by the whole body matching module 231 and face matching module 232 of the video analysis function unit 20, but as described below, they can be rewritten into a three-level whole body image similarity label ID and face image similarity label ID for displaying a tracking alert screen on the web browser 41.

The search person detection result storage function unit 33 and the detection/tracking result determination function unit 34 are provided as back-end functions of the Web server. The search person detection result storage function unit 33 receives the output of each functional unit of the human detection information extraction function unit 22 of the video analysis function unit 20 and creates a search data table to be stored in the past search data table storage unit 322. The detection/tracking result determination function unit 34 receives the output of each functional unit of the human detection information extraction function unit 22 of the video analysis function unit 20, detects that a tracking candidate corresponding to a specified symptom has appeared in the surveillance image of the surveillance camera 10, creates a detection history table to be stored in the detection history table storage unit 323, and stores the detected face image 3121, face feature amount 3122, detected whole body image 3123, and whole body feature amount 3124 of the person extracted by each functional unit of the human detection information extraction function unit 22 in the detection history data storage unit 312. The detection/tracking result determination function unit 34 also receives the output of each function unit of the monitoring/tracking execution function unit 23 of the video analysis function unit 20, detects that a person identical to the whole-body image and/or face image of the tracking target designated by the monitor has appeared in the monitoring image of one of the monitoring cameras 10, and in response to this detection, adds a record to the tracking table of the tracking target stored in the tracking table storage unit 325.

The web application 35 is an application program that performs various processes in response to requests from the web browser 41 of the monitoring terminal 40 and creates data for viewing that shows the results of the processes. The web application 35 provides the web server 30 with the functions of a login function unit 351, a monitoring function unit 352, a tracking function unit 353, and a past search function unit 354.

The login function unit 351 accepts logins from the web browser 41, performs authentication, and allows the authorized user to use the functions provided by the web application 35.

When the detection/tracking result determination function unit 34 detects a tracking candidate, the monitoring function unit 352 generates an alert and presents it on the web browser 41. To this end, the monitoring function unit 352 includes a detection alert output function unit 3521 that generates a detection alert screen to be presented on the web browser 41. The detection alert screen is an output screen for presenting images of the tracking candidate, who is a specific person who meets the specified symptoms, and the detected person, together with their status information.

The tracking function unit 353 presents the tracking results of tracking the relevant person in real time across multiple surveillance cameras 10 with the facial image and full-body image of the same person as the facial image of the tracking candidate detected by the detection/tracking result determination function unit 34 to the Web browser 41. For this purpose, the tracking function unit 353 includes an image registration function unit 3531 and a tracking alert output function unit 3532. The image registration function unit 3531 receives a facial image of the tracking target person to be tracked by a designation operation from the monitor on the Web browser 41, and stores the registered facial image 3131 and facial feature amount 3132 in the management data storage unit 313, and creates a management table 3241 and stores it in the management table storage unit 324. When the detection/tracking result determination function unit 34 detects a person identical to the whole-body image and/or face image of the tracking target, the tracking alert output function unit 3532 presents, to the web browser 41, a tracking alert screen, which is an output screen for presenting an image of the person detected as similar to the tracking target together with a stage display showing a warning level based on the degree of similarity.

The past search function unit 354 receives a time and place specification from the web browser 41, displays a list of people who visited the specified time and place, and accepts the specification of a person to search for from among them. The past search function unit 354 then presents to the web browser 41 the results of the detection/tracking result determination function unit 34 searching for similar people stored in the past search data storage unit 311 from the face image and full-body image of the specified person.

Figure 12 is a block diagram showing the hardware configuration of the Web server 30. The Web server 30 includes a hardware processor 301A, such as a CPU (Central Processing Unit), a program memory 301B, a storage device 302, and a communication interface device 303. The program memory 301B, the storage device 302, and the communication interface device 303 are connected to the hardware processor 301A via a bus 304.

The communication interface device 303 includes, for example, one or more wired or wireless communication interface units, and enables the transmission and reception of various information between the video analysis function unit 20 and the monitoring terminal 40 according to the communication protocol used in the network NET.

The program memory 301B is a storage medium that combines a non-volatile memory that can be written and read at any time, such as a hard disk drive (HDD) or a solid state drive (SSD), with a non-volatile memory such as a read only memory (ROM), and stores programs necessary for executing various control processes according to one embodiment of the present invention by being executed by a hardware processor 301A such as a CPU. That is, the hardware processor 301A can function as a search person detection result storage function unit 33 and a detection/tracking result determination function unit 34 as shown in FIG. 3 by reading and executing the programs stored in the program memory 301B. Note that these processing function units may be realized by separate hardware processors. That is, the Web server 30 may be equipped with multiple hardware processors. In addition, at least some of these processing function units may be realized in the form of various other hardware circuits, including integrated circuits such as an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a graphics processing unit (GPU). Additionally, the programs stored in the program memory 301B can include a program for a web application 35 as shown in FIG. 3.

The storage device 302 is a storage medium that combines a non-volatile memory such as an HDD or SSD that can be written to and read from at any time with a volatile memory such as a RAM (Random Access Memory), and is used to store various data acquired and created in the course of person tracking processing. In addition, the storage device 302 can be configured with a file server 31 and a DB server 32 as shown in FIG. 3.

(2) Operation Next, the operation of the person tracking system configured as above will be described.

(2-1) Monitoring Operation When the Web browser 41 is started on the monitoring terminal 40 and an operation to access the Web server 30 is performed, the Web server 30 performs an authentication operation using the login function unit 351 provided by the Web application 35. Then, if it is confirmed that the user is an authorized user, the Web server 30 generates a monitoring screen for viewing on the Web browser 41 using the monitoring function unit 352 provided by the Web application 35, and transmits it to the Web browser 41. The monitoring screen can be generated as follows.

Each of the multiple surveillance cameras 10 periodically acquires surveillance images and inputs the acquired surveillance images to the corresponding video analysis function unit 20. The image acquisition module 21 of the video analysis function unit 20 acquires frame images, which are surveillance images from the corresponding surveillance camera 10, and the human detection information extraction function unit 22 extracts the whole-body image, the area position of the whole-body image, whole-body features, and facial features of the person appearing in the frame image. The video analysis function unit 20 then transmits the frame image, whole-body image, area position of the whole-body image, whole-body features, facial features, body temperature, pulse rate, and cough symptoms to the Web server 30 via intra-server communication, Websocket, etc.

The Web server 30 accumulates the frame images, whole-body images, whole-body features, facial features, body temperature, pulse rate, and cough symptoms transmitted from each of the multiple video analysis function units 20 as past search data in the past search data storage unit 311 of the file server 31. At this time, the search-use human detection result storage function unit 33 of the Web server 30 creates a past search data table including the area positions of the whole-body images transmitted from each of the multiple video analysis function units 20, and stores it in the past search data table storage unit 322 of the DB server 32. In this way, each time a surveillance image is acquired by each surveillance camera 10, past search data is accumulated in the past search data storage unit 311, and past search data tables linked to the past search data are accumulated in the past search data table storage unit 322.

The monitoring function unit 352 can generate a monitoring screen by arranging the frame images from each monitoring camera 10 that are being accumulated in the past search data storage unit 311 on one screen. By viewing this monitoring screen displayed on the web browser 41 of the monitoring terminal 40, a security guard or other monitor can grasp the real-time situation of each part of the facility.

In addition, in the person tracking system, in parallel with the real-time situation monitoring of each part, an operation to detect the arrival of a candidate to be tracked is performed. This operation to detect the arrival of a candidate to be tracked includes a detection result determination process by the detection/tracking result determination function unit 34 and a detection alert output process by the detection alert output function unit 3521 of the monitoring function unit 352.

Figure 13 is a flowchart showing the processing procedure of the detection result determination process by the detection/tracking result determination function unit 34 of the Web server 30. The detection/tracking result determination function unit 34 can perform the processing shown in this flowchart each time past search data based on the surveillance images of any of the surveillance cameras 10 is stored in the past search data table storage unit 322. Alternatively, the detection/tracking result determination function unit 34 may perform the processing shown in this flowchart each time a predetermined number of pieces of past search data based on the surveillance images of each of the surveillance cameras 10 are stored in the past search data table storage unit 322. In the latter case, any one of the multiple pieces of past search data, for example the most recent one, can be used as the processing target.

The detection/tracking result determination function unit 34 first acquires the body temperature 3115, pulse rate 3116, and cough symptoms 3117, which are status information, from the past search data stored in the past search data storage unit 311 (step S101). The detection/tracking result determination function unit 34 also acquires the body temperature correction value, pulse rate correction value, and cough symptom correction value, which are correction values for status information, from the camera information table 3211 of the surveillance camera 10 from which the past search data was acquired and which is stored in the camera information table storage unit 321 (step S102). The body temperature 3115, pulse rate 3116, and cough symptoms 3117 acquired above are then corrected using the body temperature correction value, pulse rate correction value, and cough symptom correction value (step S103).

Then, the detection/tracking result determination function unit 34 judges whether the corrected body temperature is equal to or higher than the pre-stored prescribed body temperature (step S104). If it is judged that the corrected body temperature is equal to or higher than the prescribed body temperature, the detection/tracking result determination function unit 34 stores the detected face image 3121, the facial feature amount 3122, the detected whole body image 3123, and the whole body feature amount 3124 as detection history data in the detection history data storage unit 312, and generates a detection history table 3231 and stores it in the detection history table storage unit 323 (step S105). The corrected body temperature, corrected pulse rate, and corrected cough symptoms, which are the state information corrected in step S103, are registered in the generated detection history table 3231. Then, the detection/tracking result determination function unit 34 ends this detection/tracking result determination process. Note that the prescribed body temperature and the prescribed pulse rate and prescribed cough symptoms to be described later, which are the objects of comparison, are values corresponding to prescribed symptoms such as influenza, which are stored in the storage device 302 in advance.

In contrast, if it is determined in step S104 that the corrected body temperature is not equal to or higher than the specified body temperature, the detection/tracking result determination function unit 34 determines whether the corrected pulse rate is equal to or higher than the specified pulse rate stored in advance (step S106). If it is determined that the corrected pulse rate is equal to or higher than the specified pulse rate, the detection/tracking result determination function unit 34 proceeds to the processing of step S105.

If it is determined that the corrected pulse rate is not equal to or greater than the specified pulse rate, the detection/tracking result determination function unit 34 determines whether the corrected cough symptoms are equal to or greater than the specified cough symptoms stored in advance (step S107). If it is determined that the corrected cough symptoms are equal to or greater than the specified cough symptoms, the detection/tracking result determination function unit 34 proceeds to the processing of step S105 above. If it is determined that the corrected cough symptoms are not equal to or greater than the specified cough symptoms, the detection/tracking result determination function unit 34 ends this detection/tracking result determination processing.

In this way, if any of the corrected body temperature, pulse rate, and cough symptoms are equal to or higher than the predetermined body temperature, pulse rate, or cough symptoms, it is determined that a tracking candidate has been detected, and the detection history data is stored in the detection history data storage unit 312, and the corrected body temperature, pulse rate, and cough symptoms are registered in the detection history table 3231.

Figures 14A and 14B are flowcharts showing the processing steps of the detection alert output process by the detection alert output function unit 3521 of the monitoring function unit 352. The detection alert output function unit 3521 periodically performs the processing shown in this flowchart, for example, at the same processing timing as the detection result determination process by the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection alert output function unit 3521 first determines whether or not a narrowing down operation has been performed by the monitor on the detection alert screen displayed on the web browser 41 of the monitoring terminal 40 (step S201). If it is determined that a narrowing down operation has been performed, the detection alert output function unit 3521 proceeds to step S206, which will be described later.

In contrast, if it is determined that a narrowing down operation has not been performed, the detection alert output function unit 3521 acquires information of a new detection history table 3231 from the detection history table storage unit 323 (step S202). That is, the detection alert output function unit 3521 extracts and acquires the detection date and time in the detection history table 3231 that is recorded as a date and time after the previous processing date and time.

Then, the detection alert output function unit 3521 creates a new detected person card, which is a screen image, based on the acquired information of the new detection history table 3231 (step S203). This detected person card is a screen image that includes the detected face image 3121 and the detected whole body image 3123 stored as detection history data in the detection history data storage unit 312, and the corresponding corrected status information in the detection history table 3231. The detected person card will be described in detail later.

The detection alert output function unit 3521 updates the detection alert screen to be displayed on the Web browser 41 with this newly created detected person card (step S204). The detection alert output function unit 3521 then transmits this updated detection alert screen to the Web browser 41 via the network NET and displays it there (step S205). The Web browser 41 displays this detection alert screen in an alert screen display area provided as part of the monitoring screen, or by opening a window separate from the monitoring screen. The detection alert output function unit 3521 then ends this detection alert output process.

On the other hand, if it is determined in step S201 that a narrowing down operation has been performed, the detection alert output function unit 3521 acquires the specified narrowing down level (step S206). This narrowing down level is an index for narrowing down the tracking information cards to be displayed on the detection alert screen.

The detection alert output function unit 3521 also acquires information on the detection history table 3231 to be processed from the detection history table storage unit 323 (step S207). That is, the detection alert output function unit 3521 acquires information on one detection history table in order from the most recent one based on the detection date and time in the detection history table 3231.

Then, the detection alert output function unit 3521 determines whether the acquired corrected body temperature is higher than the designated body temperature specified as the narrowing level, i.e., whether the corrected body temperature is equal to or higher than the designated body temperature (step S208). If it is determined that the corrected body temperature is equal to or higher than the designated body temperature, the detection alert output function unit 3521 creates a detected person card, which is a screen image, based on the acquired information from the detection history table 3231 (step S209).

Then, the detection alert output function unit 3521 determines whether or not a predetermined number of detected person cards have been created (step S210). This predetermined number can depend on the number of items that can be displayed on the detection alert screen. Alternatively, it may be an arbitrary number. If it is determined that the predetermined number of detected person cards have not yet been created, the detection alert output function unit 3521 returns to the processing of step S207 above.

If it is determined in step S208 that the corrected body temperature is not equal to or higher than the designated body temperature, the detection alert output function unit 3521 determines whether the corrected pulse rate obtained is equal to or higher than the designated pulse rate designated as the narrowing level (step S211). If it is determined that the corrected pulse rate is equal to or higher than the designated pulse rate, the detection alert output function unit 3521 proceeds to the processing of step S209 and creates a detected person card.

If it is determined in step S213 that the corrected pulse rate is not equal to or greater than the designated pulse rate, the detection alert output function unit 3521 determines whether the corrected cough symptoms acquired are equal to or greater than the designated cough symptoms designated as the narrowing level (step S212). If it is determined that the corrected cough symptoms are equal to or greater than the designated cough symptoms, the detection alert output function unit 3521 proceeds to the processing of step S209 and creates a detected person card. If it is determined that the corrected cough symptoms are not equal to or greater than the designated cough symptoms, the detection alert output function unit 3521 proceeds to the processing of step S210.

If it is determined in step S210 that a predetermined number of detection person cards have been created, the detection alert output function unit 3521 rearranges the created detection person cards in order of detection date and time and updates the detection alert screen (step S213). The detection alert output function unit 3521 then proceeds to the processing of step S205, and transmits this updated detection alert screen to the web browser 41 via the network NET, and displays it there.

Note that if there is no more information in the detection history table 3231 to be processed in step S207 before a predetermined number of detected person cards are created, the process may proceed to step S213.

Figure 15 is a diagram showing an example of a detection alert screen 42 displayed on the web browser 41. The detection alert screen 42 includes a detected person card 421 for each detected person and a narrowing down instruction area 422.

The detected person card 421 may include, as its display contents, a warning label 4211, a suspicious person ID 4212, a tracking candidate image 4213, a detected body temperature 4214, a detected pulse rate 4215, a detected cough symptom 4216, and a tracking start button 4217.

The warning label 4211 is a message to alert the observer, and may be displayed as a flashing indicator light. The suspect ID 4212 is identification information transcribed from the detection history table 3231. The tracking candidate image 4213 is the detected face image 3121 and detected whole body image 3123 stored in the detection history data storage unit 312 of the person detected as the tracking candidate. The detected body temperature 4214, detected pulse rate 4215, and detected coughing symptoms 4216 are the corrected body temperature, corrected pulse rate, and corrected coughing symptoms transcribed from the detection history table 3231. The start tracking button 4217 is a button pressed when tracking the person.

Note that each time a new specific person is detected as a tracking candidate, the old detected person card 421 is shifted downward so that it is displayed at the top of the detection alert screen 42. A new detected person card 421 is not added for the same person.

The narrowing down instruction area 422 is an area for receiving instructions to narrow down the detected person cards 421 to be displayed on the detection alert screen 42. This narrowing down instruction area 422 includes a body temperature presentation threshold setting slider 4221, a pulse rate presentation threshold setting slider 4222, and a cough symptom threshold setting button 4223.

The body temperature presentation threshold setting slider 4221 is a slider that is slid to set the presentation threshold indicating the designated body temperature used in step S208 above. A tracking information card 432 including a corrected body temperature equal to or higher than the body temperature indicated by the monitor sliding this slider is displayed on the detection alert screen 42. In FIG. 15, the body temperature presentation threshold setting slider 4221 shows a state in which the designated body temperature of 37.5°C is set.

Similarly, the pulse rate presentation threshold setting slider 4222 is a slider that is slid to set a presentation threshold indicating a specified pulse rate used in step S211 above. A tracking information card 432 including a corrected pulse rate equal to or greater than the pulse rate indicated by the monitor's sliding operation of this slider is displayed on the detection alert screen 42. In FIG. 15, the pulse rate presentation threshold setting slider 4222 shows a state in which the specified pulse rate of 110 is set.

In addition, the cough symptom threshold setting button 4223 includes three buttons corresponding to the levels of cough symptoms, since in this embodiment the severity of cough symptoms is indicated in three stages. When the monitor presses these three buttons, a tracking information card 432 including corrected cough symptoms of a specified number or more stages is displayed on the detection alert screen 42. The example in FIG. 15 shows a state in which the default cough symptom, which is stage 1, is set as the stage in the cough symptom threshold setting button 4223. The cough symptom threshold setting button 4223 also includes a reset button that is pressed to reset the specified number of stages.

In the example of FIG. 15, the detection alert screen 42 includes three detection person cards 421. The detection person card 421 presented at the top is a detection person card 421 presenting a tracking candidate who has been detected with a body temperature of 37.6° C., which is a body temperature of 37.5° C. or higher, specified by the body temperature presentation threshold setting slider 4221, or a second-stage cough symptom, which is a first-stage or higher cough symptom degree, specified by the cough symptom threshold setting button 4223. The detection person card 421 presented in the second stage is a detection person card 421 presenting a tracking candidate who has been detected with a body temperature of 38.8° C., which is a body temperature of 37.5° C. or higher, specified by the body temperature presentation threshold setting slider 4221, a pulse rate of 115, which is a pulse rate of 110 or higher, specified by the pulse rate presentation threshold setting slider 4222, or a third-stage cough symptom, which is a first-stage or higher cough symptom degree, specified by the cough symptom threshold setting button 4223. The detected person card 421 presented in the third row is a detected person card 421 that presents a tracking candidate who has been detected with a pulse rate of 115, which is a pulse rate of 110 or more as specified by the pulse rate presentation threshold setting slider 4222, or a cough symptom of 2 levels, which is a cough symptom degree of 1 level or more as specified by the cough symptom threshold setting button 4223.

Figure 16 is an example of the detection alert screen 42 when 37.8°C is specified by the body temperature presentation threshold setting slider 4221 and two levels are specified by the cough symptom threshold setting button 4223. In this example, as a result of narrowing down by the body temperature and cough symptom presentation thresholds, the detected person card 421 of the tracking candidate that was presented at the top in the example of Figure 15 is no longer presented. In the example of Figure 16, the detected person cards 421 presented in the second and third rows in the example of Figure 15 are shifted upward, and a new detected person card 421 is presented in the third row, presenting a tracking candidate who has a body temperature of 7.9°C, which is a body temperature of 37.8°C or higher specified by the body temperature presentation threshold setting slider 4221, or a cough symptom of two levels or higher, which is a cough symptom degree of two levels or higher specified by the cough symptom threshold setting button 4223.

(2-2) Tracking Operation When the monitor presses the tracking start button 4217 on the detection alert screen 42, the Web server 30 uses the tracking function unit 353 provided by the Web application 35 to present the tracking results of tracking the person on the Web browser 41 using the video analysis function unit 20 and the detection/tracking result determination function unit 34. This tracking operation includes image registration processing by the image registration function unit 3531 of the tracking function unit 353, tracking processing by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, tracking result determination processing by the detection/tracking result determination function unit 34, and tracking alert output processing by the tracking alert output function unit 3532 of the tracking function unit 353.

Figure 17 is a flowchart showing the processing procedure of the image registration processing by the image registration function unit 3531 of the tracking function unit 353. The image registration function unit 3531 periodically performs the processing shown in this flowchart, for example, in synchronization with the acquisition interval of the surveillance images of the surveillance camera 10.

The image registration function unit 3531 first determines whether or not the monitor has pressed the tracking start button 4217 on the detection alert screen 42 displayed on the web browser 41 of the monitoring terminal 40 (step S301). If it is determined that the tracking start button 4217 has not been pressed, the image registration function unit 3531 ends this image registration process.

On the other hand, if it is determined that the tracking start button 4217 has been pressed, the image registration function unit 3531 acquires the face image of the person as a face image for tracking (step S302). That is, the image registration function unit 3531 acquires the detection face image 3121 as a face image for tracking from the detection history data storage unit 312 based on the tracking ID of the detection history table storage unit 323 having the suspicious person ID of the detection alert screen 42. For example, the image registration function unit 3531 can acquire the detection face image 3121 specified by the detection ID written in the detection history table 3231 having the tracking ID, which has a detection date and time of a specific period, for example, the latest one. The specific period is not limited to the latest, but may be the oldest, the latest or oldest during a specified period, or a season (spring, summer, autumn, winter, etc.). Alternatively, the image registration function unit 3531 may generate a selection screen including in a list format the detected face images 3121 identified by the detection IDs written in the detection history tables 3231 having the tracking IDs, and display the selection screen on the Web browser 41 to allow the monitor to select the detected face image 3121 to be used as the tracking face image. Then, the image registration function unit 3531 stores the acquired tracking face image in the management data storage unit 313 as the registered face image 3131. The image registration function unit 3531 also creates a management table 3241 and stores it in the management table storage unit 324. The management table 3241 includes a query face feature path indicating the storage path of the registered face image 3131 in the management data storage unit 313.

Then, the image registration function unit 3531 acquires a whole-body image corresponding to the detected face image 3121 acquired from the detection history data storage unit 312 as a tracking face image (step S303). That is, the image registration function unit 3531 acquires the detected whole-body image 3123 identified by the detection ID described in the detection history table 3231 for the detected face image 3121 acquired as the tracking face image as a tracking whole-body image. Alternatively, the image registration function unit 3531 acquires the whole-body image 3112 from the past search data storage unit 311 as a tracking whole-body image based on the detection ID. The image registration function unit 3531 stores the acquired tracking whole-body image in the management data storage unit 313 as a registered whole-body image 3133. In addition, the image registration function unit 3531 adds a query whole-body feature path indicating the storage path of the registered whole-body image 3133 in the management data storage unit 313 to the management table 3241 stored in the management table storage unit 324.

Furthermore, the image registration function unit 3531 acquires facial image features for tracking and whole-body image features for tracking (step S304). That is, the image registration function unit 3531 acquires facial features 3122 and whole-body features 3124 from the detection history data storage unit 312 as facial features for tracking and whole-body features for tracking based on the above detection ID. The image registration function unit 3531 stores these acquired facial features for tracking and whole-body features for tracking in the management data storage unit 313 as facial features 3132 and whole-body features 3134. The image registration function unit 3531 also adds the facial feature file path and whole-body feature file path indicating the feature storage path to the management table 3241 stored in the management table storage unit 324.

Then, the image registration function unit 3531 transmits the registered face image 3131 and registered whole-body image 3133 stored in the management data storage unit 313 as tracking images, and the facial feature amount 3132 and whole-body feature amount 3134 as tracking features to each of the multiple video analysis function units 20 via intra-server communication, WebSocket, or the like (step S305). After that, the image registration function unit 3531 ends this image registration process.

Figure 18 is a flowchart showing the processing procedure of the tracking process by the monitoring/tracking execution function unit 23 of the video analysis function unit 20. The monitoring/tracking execution function unit 23 performs the process shown in this flowchart each time it receives a tracking image and feature amount from the image registration function unit 3531 of the tracking function unit 353 of the Web server 30, for example, via intra-server communication, WebSocket, etc.

The monitoring/tracking execution function unit 23 first acquires the tracking image and features received from the image registration function unit 3531 of the Web server 30 (step S411).

Then, the monitoring/tracking execution function unit 23 acquires the whole-body image and its features at time T from the human detection information extraction function unit 22 as a whole-body image for matching and whole-body features for matching (step S412). If the surveillance image of the corresponding surveillance camera 10 includes whole-body images of multiple people, the monitoring/tracking execution function unit 23 acquires a whole-body image for matching and whole-body features for matching for each person.

Then, the monitoring/tracking execution function unit 23 calculates, by the whole-body matching module 231, a whole-body image similarity, which is the degree of similarity between each of the whole-body features for matching from the human detection information extraction function unit 22 and the whole-body features for tracking from the image registration function unit 3531 (step S413).

The monitoring/tracking execution function unit 23 also acquires the face image and its features at time T from the human detection information extraction function unit 22 as a matching face image and matching face features (step S414). If the surveillance image of the corresponding surveillance camera 10 contains face images of multiple people, the monitoring/tracking execution function unit 23 acquires a matching face image and matching face features for each person. Depending on the orientation of the person, there are cases where a whole-body image can be extracted but a face image cannot be extracted. Therefore, the number of matching whole-body images acquired in step S412 above and the number of matching face images acquired in step S414 do not necessarily match. There may also be cases where no face image can be extracted and no matching face image (and matching face features) is acquired.

Then, the monitoring/tracking execution function unit 23 calculates, by the face matching module 232, a face image similarity, which is the degree of similarity between each of the face features for matching from the human detection information extraction function unit 22 and the face features for tracking from the image registration function unit 3531 (step S415).

Then, the monitoring/tracking execution function unit 23 transmits the sets of similarities and matching images for the whole-body image and face image to the detection/tracking result determination function unit 34 of the Web server 30 via intra-server communication, WebSocket, or the like (step S416). Note that the set of similarities and matching images may include both a set of whole-body image similarity and whole-body image for matching and a set of face image similarity and face image for matching, or may include only a set of whole-body image similarity and whole-body image for matching without a set of face image similarity and face image for matching.

Then, the monitoring/tracking execution function unit 23 determines whether or not the monitor has pressed the tracking stop button on the tracking alert screen displayed on the web browser 41 at the monitoring terminal 40 (step S417). Details of the tracking alert screen having the tracking stop button will be described later. If it is determined that the tracking stop button has been pressed, the monitoring/tracking execution function unit 23 ends this monitoring/tracking process.

If it is determined that the tracking stop button has not been pressed, the monitoring/tracking execution function unit 23 adds a time interval t1 to the time T, i.e., updates it to the next processing time (step S418), and then repeats the above processing from step S412. Here, the time interval t1 can be any time interval, such as a few seconds. Alternatively, the time interval t1 may be an integer multiple of the interval at which the monitoring camera 10 captures the monitoring images.

Figure 19 is a flowchart showing the processing procedure of the tracking result determination process by the detection/tracking result determination function unit 34 of the web server 30. The detection/tracking result determination function unit 34 periodically performs the processing shown in this flowchart, for example, in synchronization with the processing interval of the tracking process of the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection/tracking result determination function unit 34 first determines whether or not a set of similarity and a matching image has been received from the monitoring/tracking execution function unit 23 (step S121). If it is determined that a set of similarity and a matching image has not been received, the detection/tracking result determination function unit 34 ends this tracking result determination process.

In contrast, if it is determined that a set of similarity and matching image has been received, the detection/tracking result determination function unit 34 acquires face matching threshold 1 and whole body matching threshold 1 (step S122). That is, the detection/tracking result determination function unit 34 acquires face matching threshold 1 and whole body matching threshold 1 from the camera information table 3211 for the surveillance camera 10 corresponding to the video analysis function unit 20 that is the sender of the set of similarity and matching image, which is stored in the camera information table storage unit 321.

Then, the detection/tracking result determination function unit 34 compares each of the received whole-body image similarities with the acquired whole-body matching threshold 1 and determines whether or not there is a whole-body image similarity equal to or greater than the whole-body matching threshold 1 (step S123). If it is determined that there is a whole-body image similarity equal to or greater than the whole-body matching threshold 1, the detection/tracking result determination function unit 34 determines that the person having the whole-body feature is a tracking target. Therefore, in this case, the detection/tracking result determination function unit 34 stores the whole-body image similarity determined to be large in a predetermined storage area of the storage device 302 together with the corresponding matching whole-body image received in the above step S121, and registers the whole-body image similarity in the tracking table 3251 (step S124). When registering this whole-body image similarity, if the tracking table 3251 for the corresponding suspicious person ID has not yet been stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is stored in the tracking table storage unit 325. Furthermore, if a tracking table 3251 for the suspected person ID is already stored in the tracking table storage unit 325, a record is added rather than a new tracking table 3251 being generated. The whole-body image similarity registered in the tracking table 3251 is a whole-body image similarity that is equal to or greater than the whole-body matching threshold value 1.

After that, or if it is determined in step S123 that there is no whole body image similarity equal to or greater than the whole body matching threshold 1, the detection/tracking result determination function unit 34 compares each of the received face image similarities with the acquired face matching threshold 1 and determines whether or not there is a face image similarity equal to or greater than the face matching threshold 1 (step S125). If it is determined that there is a face image similarity equal to or greater than the face matching threshold 1, the detection/tracking result determination function unit 34 determines that the person having that face feature is a tracking target. Therefore, in this case, the detection/tracking result determination function unit 34 stores the face image similarity determined to be large together with the corresponding matching face image received in step S121 in a predetermined storage area of the storage device 302, and registers the face image similarity in the tracking table 3251 (step S126). When registering this facial image similarity, if the tracking table 3251 for the relevant suspect ID is not yet stored in the tracking table storage unit 325, a tracking table 3251 is generated, and the generated tracking table 251 is stored in the tracking table storage unit 325. Also, if the tracking table 3251 for the relevant suspect ID is already stored in the tracking table storage unit 325, a record is added rather than a new tracking table 3251 being generated. The facial image similarity registered in the tracking table 3251 is a whole-body image similarity with a value equal to or greater than the face matching threshold 1.

After that, or if it is determined in step S125 above that there is no facial image similarity that is equal to or greater than the facial matching threshold 1, the detection/tracking result determination function unit 34 ends this tracking result determination process.

Figures 20A to 20C are flowcharts showing the processing steps of the tracking alert output process by the tracking alert output function unit 3532 of the tracking function unit 353. The tracking alert output function unit 3532 periodically performs the process shown in this flowchart, for example, at time intervals t1 in the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The tracking alert output function unit 3532 first determines whether or not a narrowing down designation operation has been performed on the tracking alert screen displayed on the web browser 41 by the monitor at the monitoring terminal 40 (step S501). If it is determined that a narrowing down designation operation has been performed, the tracking alert output function unit 3532 proceeds to step S519, which will be described later.

On the other hand, if it is determined that no narrowing down operation has been performed, the tracking alert output function unit 3532 obtains face matching thresholds 1 to 3 and whole body matching thresholds 1 to 3 from the camera information table 3211 for each surveillance camera 10 stored in the camera information table storage unit 321 (step S502).

Then, the tracking alert output function unit 3532 acquires new tracking results from the tracking table 3251 stored in the tracking table storage unit 325 (step S503). That is, the tracking alert output function unit 3532 extracts and acquires the whole-body image similarity and face image similarity in the tracking table 3251 in which the actual number of similarities is recorded instead of the image similarity ID.

Then, the tracking alert output function unit 3532 determines whether the facial image similarity value in the tracking table 3251 is equal to or greater than the facial matching threshold 3 (step S504). If it is determined that the facial image similarity value is equal to or greater than the facial matching threshold 3, the tracking alert output function unit 3532 rewrites the facial image similarity value in the tracking table 3251 in the tracking table storage unit 325 from a real number to a facial image similarity label ID, here ID=3 (step S505).

On the other hand, if it is determined that the facial image similarity value is not equal to or greater than the facial matching threshold 3, the tracking alert output function unit 3532 further determines whether the facial image similarity value is equal to or greater than the facial matching threshold 2 (step S506). If it is determined that the facial image similarity value is equal to or greater than the facial matching threshold 2, the tracking alert output function unit 3532 rewrites the facial image similarity value in the tracking table 3251 in the tracking table storage unit 325 from a real number to a facial image similarity label ID, here ID=2 (step S507).

If it is determined that the facial image similarity value is not equal to or greater than the facial matching threshold 2, the tracking alert output function unit 3532 further determines whether the facial image similarity value is equal to or greater than the facial matching threshold 1 (step S508). If it is determined that the facial image similarity value is equal to or greater than the facial matching threshold 1, the tracking alert output function unit 3532 rewrites the facial image similarity value in the tracking table 3251 in the tracking table storage unit 325 from a real number to a facial image similarity label ID, here ID=1 (step S509).

In the process related to the tracking operation of the tracking function unit 353, both facial image similarity and whole body similarity are used, and there are cases where only one of the facial image similarity and the whole body similarity is registered in the tracking table 3251. In other words, when the facial image similarity value is not equal to or greater than the facial matching threshold 2, there are cases where a facial image similarity value that is equal to or greater than the facial matching threshold 1 is registered, and cases where a facial image similarity value that is smaller than the facial matching threshold 1 is registered (or no facial image similarity value is registered). Therefore, just because the facial image similarity value is not equal to or greater than the facial matching threshold 2, it cannot be determined that the facial image similarity value is equal to or greater than the facial matching threshold 1.

Once the facial image similarity value in the tracking table 3251 has been rewritten to the facial image similarity label ID in the above manner, or if it is determined in step S508 above that the facial image similarity value is not greater than or equal to the facial matching threshold 1, the tracking alert output function unit 3532 proceeds to the whole-body image similarity determination operation.

That is, the tracking alert output function unit 3532 determines whether the value of the whole-body image similarity in the tracking table 3251 is equal to or greater than the whole-body matching threshold 3 (step S510). If it is determined that the value of the whole-body image similarity is equal to or greater than the whole-body matching threshold 3, the tracking alert output function unit 3532 rewrites the value of the whole-body image similarity in the tracking table 3251 in the tracking table storage unit 325 from a real number to a whole-body image similarity label ID, here ID=3 (step S511).

On the other hand, if it is determined that the value of the whole-body image similarity is not equal to or greater than the whole-body matching threshold 3, the tracking alert output function unit 3532 further determines whether the value of the whole-body image similarity is equal to or greater than the whole-body matching threshold 2 (step S512). If it is determined that the value of the whole-body image similarity is equal to or greater than the whole-body matching threshold 2, the tracking alert output function unit 3532 rewrites the value of the whole-body image similarity in the tracking table 3251 in the tracking table storage unit 325 from a real number to a whole-body image similarity label ID, here ID=2 (step S513).

If it is determined that the value of the whole-body image similarity is not equal to or greater than the whole-body matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of the whole-body image similarity is equal to or greater than the whole-body matching threshold 1 (step S514). If it is determined that the value of the whole-body image similarity is equal to or greater than the whole-body matching threshold 1, the tracking alert output function unit 3532 rewrites the value of the whole-body image similarity in the tracking table 3251 in the tracking table storage unit 325 from a real number to a whole-body image similarity label ID, here ID=1 (step S515).

When the value of the whole body image similarity in the tracking table 3251 is rewritten to the whole body image similarity label ID in the above manner, or when it is determined in step S514 that the value of the whole body image similarity is not equal to or greater than the whole body matching threshold 1, the tracking alert output function unit 3532 creates a new tracking information card, which is a screen image (step S516). This tracking information card is a screen image that includes the matching whole body image and matching face image stored in a predetermined memory area of the storage device 302 in steps S124 and S126, and a warning level display that indicates a warning level based on the corresponding whole body image similarity label ID and face image similarity label ID in the tracking table 3251. Details of this tracking information card will be described later.

The tracking alert output function unit 3532 updates the tracking alert screen to be displayed on the Web browser 41 with the newly created tracking information card (step S517). The tracking alert output function unit 3532 then transmits the updated tracking alert screen to the Web browser 41 via the network NET and displays it there (step S518). The Web browser 41 displays the tracking alert screen in an alert screen display area provided as part of the monitoring screen, or by opening a window separate from the monitoring screen. The tracking alert output function unit 3532 then ends the tracking alert output process.

On the other hand, if it is determined in step S501 that a narrowing down operation has been performed, the tracking alert output function unit 3532 acquires the specified narrowing down level (step S519). This narrowing down level is an index for narrowing down the tracking information cards to be displayed on the tracking alert screen.

The tracking alert output function unit 3532 extracts suspicious person IDs according to the specified filtering level from the tracking table 3251 stored in the tracking table storage unit 325 (step S520).

Then, based on the detection IDs of the extracted suspect IDs, the tracking alert output function unit 3532 creates tracking information cards, the number of which can be displayed on the tracking alert screen, including the full-body image 3112 stored as past search data in the past search data storage unit 311 and the face image extracted from the full-body image 3112 by the human detection information extraction function unit 22 of the video analysis function unit 20 (step S521).

Then, the tracking alert output function unit 3532 creates a new tracking information card (step S522), similar to step S516 above, and updates the tracking alert screen with the created tracking information card (step S523), similar to step S517 above. The tracking alert output function unit 3532 then proceeds to the processing of step S518 above, and transmits the updated tracking alert screen to the web browser 41, causing it to be displayed there.

Figure 21 is a diagram showing an example of a tracking alert screen 43 displayed on the web browser 41. The tracking alert screen 43 includes a tracking target information area 431, a tracking information card 432, and a narrowing down instruction area 433.

The tracking target information area 431 is an area that displays information about tracking targets, and each time a tracking target is added, it is added and displayed at the top of the tracking alert screen 43. Older tracking target information areas 431 are successively shifted downward. Here, the display contents of the top, i.e., the latest tracking target information area 431, and the other tracking target information areas 431 are different. That is, the top tracking target information area 431 includes a warning label 4311, a suspicious person ID 4312, a registered image 4313, tracking information 4314, and a tracking stop button 4315. In contrast, tracking target information areas 431 other than the top include only the registered image 4313 and tracking information 4314.

Here, the attention label 4311 is a message to alert the observer, and may be displayed as a flashing indicator light. The suspicious person ID 4312 is identification information transferred from the management table 3241. The registered image 4313 is the registered face image 3131 and the registered whole body image 3133 stored in the management data storage unit 313 of the tracking target. The tracking information 4314 is information about the tracking target, and the tracking information 4314 in the top tracking target information area 431 and the tracking information 4314 in the other tracking target information areas 431 have different contents. That is, the tracking information 4314 in the top tracking target information area 431 includes corrected body temperature, corrected pulse rate, and corrected cough symptoms, which are status information transferred from the management table 3241. The tracking information 4314 in the tracking target information areas 431 other than the top includes the tracking status instead of these status information. The stop tracking button 4315 is a button that is pressed to end tracking of the person.

In addition, each time the tracking alert output function unit 3532 creates a new card, a tracking information card 432 is added and displayed at the top of the tracking alert screen 43. Older tracking information cards 432 are sequentially shifted downward. This tracking information card 432 includes, as its display contents, detection date and time information 4321, detection camera location information 4322, detected face image 4323, face warning stage display 4324, detected whole body image 4325, whole body warning stage display 4326, and detected background image 4327.

Here, the detection date and time information 4321 is the date and time information when the tracking target is detected from the monitoring image of one of the monitoring cameras 10. The detection camera location information 4322 is the camera location information based on the monitoring camera ID of the detection. The detection face image 4323 is a face image for matching of the tracking target that has been detected and stored. The face attention stage display 4324 indicates the stage of similarity between the face image of the tracking target and the face image of the detected person. In other words, the similarity is not shown as a real number, but as a multiple-stage attention stage display based on the face image similarity label ID in the tracking table 3251, here three stages. In this face attention stage display 4324, the number of identified rectangles shown by hatching in FIG. 21 corresponds to the face image similarity label ID. For example, in the top tracking information card 432, the facial image similarity label ID in the tracking table 3251 is "2", so two rectangles are distinguished and displayed in the face attention stage display 4324, and in the second tracking information card 432 from the top, the facial image similarity label ID in the tracking table 3251 is "3", so three rectangles are distinguished and displayed in the face attention stage display 4324.

Similarly, the detected whole-body image 4325 is a whole-body image of the tracked person detected and stored for comparison. The whole-body attention stage display 4326 indicates the stage of similarity between the whole-body image of the tracked person and the whole-body image of the detected person. That is, the similarity is not a real number, but is shown as a multiple-stage, three-stage attention stage display based on the whole-body image similarity label ID in the tracking table 3251. As with the face attention stage display 4324, the number of squares identified and displayed by hatching in FIG. 21 corresponds to the whole-body image similarity label ID in the top tracking information card 432, so three squares are identified and displayed in the whole-body attention stage display 4326, and the second tracking information card 432 from the top, so the whole-body image similarity label ID in the tracking table 3251 is "2", so two squares are identified and displayed in the whole-body attention stage display 4326.

The detection background image 4327 is a frame image, which is the original surveillance image from which the matching face image and/or matching whole-body image was extracted. This frame image is the frame image 3111 stored in the past search data storage unit 311.

Note that the tracking information card 432 does not necessarily include both the detected face image 4323 and face warning stage display 4324, and the detected whole-body image 4325 and whole-body warning stage display 4326, and may only include one of them.

The narrowing down instruction area 433 is an area for receiving instructions to narrow down the tracking information cards 432 to be displayed on the tracking alert screen 43. This narrowing down instruction area 433 includes a facial image similarity selection button 4331, a facial image similarity selection reset button 4332, a full-body image similarity selection button 4333, and a full-body image similarity selection reset button 4334.

Here, the facial image similarity selection button 4331 includes three buttons corresponding to the number of stages of the facial attention stage display 4324. Only tracking information cards 432 including facial attention stage displays 4324 with a specified number of stages or more in response to the monitor's pressing of the facial image similarity selection button 4331 will be displayed on the tracking alert screen 43. In the example of FIG. 21, since the number of stages is not specified on the facial image similarity selection button 4331, tracking information cards 432 including one or more stages of the facial attention stage display 4324 are displayed on the tracking alert screen 43. The facial image similarity selection reset button 4332 is a button that is pressed when resetting the number of stages specified on the facial image similarity selection button 4331.

Similarly, the whole body image similarity selection button 4333 includes three buttons corresponding to the number of stages of the whole body warning stage display 4326. When the observer presses the whole body image similarity selection button 4333, only the tracking information cards 432 including the whole body warning stage display 4326 with a number of stages equal to or greater than the designated number are displayed on the tracking alert screen 43. In the example of FIG. 21, since the number of stages is not specified on the whole body image similarity selection button 4333, the tracking information cards 432 including one or more whole body warning stage displays 4326 are displayed on the tracking alert screen 43. The whole body image similarity selection reset button 4334 is a button that is pressed to reset the number of stages specified on the whole body image similarity selection button 4333.

Figure 22 is a diagram showing another example of the tracking alert screen 43 displayed on the web browser 41. This example shows a case where three levels are specified with the facial image similarity selection button 4331 and one level is specified with the full-body image similarity selection button 4333. Here, the pressed button is identifiably displayed as shown by hatching in Figure 21. The designation by the facial image similarity selection button 4331 and the designation by the full-body image similarity selection button 4333 are AND conditions, and only tracking information cards 432 that match the number of levels specified by both are displayed on the tracking alert screen 43.

(2-3) Past Search Operation The tracing operation by the tracing function unit 353 is a function for tracing a tracing target person in real time. On the other hand, there are cases where it is desired to investigate whether a specific person has visited the facility in the past.

The past search function unit 354 receives a time and place specification from the web browser 41, and extracts from the past search data storage unit 311 full-body images 3112 of people who visited at the specified time and place based on the past search data table 3221 stored in the past search data table storage unit 322. Then, it creates a selection screen that displays a list of the extracted full-body images 3112, and displays it on the web browser 41.

When the web browser 41 selects and specifies the person to be searched, the past search function unit 354 uses the selected full-body image 3112 as the full-body image for tracking, and performs the same operation as the tracking function unit 353, targeting the frame image 3111 stored in the past search data storage unit 311 for the specified time and place instead of the surveillance image from the real-time surveillance camera 10.

This enables the past search function unit 354 to search for people similar to the selected person from among past visitors stored in the past search data storage unit 311, and display the results on the web browser 41 as a tracking alert screen 43.

(3) Effects As described above in detail, the Web server 30 as a person tracking device according to one embodiment acquires, from the human detection information extraction function unit 22 as an information extraction unit that detects from each of the surveillance images, multiple types of status information related to the health status of a person included in each of the surveillance images periodically acquired by each of the multiple surveillance cameras 10, multiple types of status information detected by the human detection information extraction function unit 22, and based on the multiple types of acquired status information, the detection/tracking result determination function unit 34 as a discrimination unit that determines whether or not a person included in each of the surveillance images is a specific person corresponding to a prescribed symptom, and based on the discrimination result of the detection/tracking result determination function unit 34, The system is provided with an image registration function unit 3531 as a registration unit that registers, as a tracking target image, an image of a specific person corresponding to a prescribed symptom such as influenza among images of people extracted from the surveillance images, a detection/tracking result determination function unit 34 as a detection unit that detects an image of a person similar to the tracking target image registered by the image registration function unit 3531 from each of the surveillance images periodically acquired by each of the multiple surveillance cameras 10, and a tracking alert output function unit 3532 as a tracking output screen generation unit that generates a tracking alert screen 43 as a tracking output screen for presenting an image of a person similar to the tracking target image detected by the detection/tracking result determination function unit 34. This makes it possible to detect and track a person corresponding to a prescribed symptom from the surveillance images acquired by the surveillance cameras 10.

The Web server 30 further includes a detection alert output function unit 3521 as a detection output screen generating unit that generates a detection alert screen 42 as a detection output screen for presenting images of specific persons corresponding to specified symptoms among images of persons extracted from the surveillance images of each of the multiple surveillance cameras 10 based on the judgment result of the detection/tracking result judgment function unit 34, together with status information. The image registration function unit 3531 acquires, from the surveillance terminal 40 that displays the detection alert screen 42 generated by the detection alert output function unit 3521, a designation of an image of a tracking target to be tracked from among the images of persons presented on the detection alert screen 42, and registers the acquired designated image as a tracking target image. Thus, by presenting multiple tracking candidates on the detection alert screen 42 together with their status information, the surveillance person can easily recognize the tracking target to be tracked.

In addition, the detection/tracking result determination function unit 34 is configured to determine that the image of a person extracted from the surveillance images of each of the multiple surveillance cameras 10 is an image of a specific person who corresponds to a specified symptom when any one of the multiple types of acquired status information is equal to or greater than a predetermined value. This makes it possible to reliably pick out people who are even slightly likely to correspond to a specified symptom.

The detection alert output function unit 3521 also acquires a presentation threshold value specified for each type of multiple types of status information, and narrows down the images to be presented on the detection alert screen 42 from among images of specific persons that the detection/tracking result determination function unit 34 has determined to correspond to specified symptoms based on the acquired presentation threshold value. Therefore, by the monitor specifying the presentation threshold value, it is possible to narrow down the persons who may correspond to the specified symptoms presented on the detection alert screen 42, and the monitor can easily recognize tracking targets who need to be tracked.

In this case, the detection alert output function unit 3521 selects an image of a specific person for which condition information equal to or exceeds any one of the presentation thresholds for each acquired type as an image to be presented on the detection alert screen 42. Therefore, there is no risk of excluding a person who may even slightly meet the specified symptoms from the narrowing down.

The web server 30 further includes a camera information table storage unit 321 as a camera information storage unit that stores correction coefficients for correcting status information for each of the multiple surveillance cameras 10, and the detection/tracking result determination function unit 34 corrects the status information acquired from the human detection information extraction function unit 22 with the correction coefficients stored in this camera information table storage unit 321, and determines whether or not the image of a person is an image of a specific person corresponding to a prescribed symptom based on the corrected status information. This makes it possible to reduce variation between status information that depends on differences in the type of surveillance camera 10, etc.

The condition information may include at least one of body temperature, pulse rate, and cough symptoms, each of which is extracted from the monitoring image.

The person tracking system according to one embodiment can include a web server 30 as a person tracking device according to one embodiment, a plurality of surveillance cameras 10, a surveillance terminal 40 operated by a surveillance person, and a video analysis function unit 20 as an analysis unit having a human detection information extraction function unit 22.

In addition, a person tracking method according to one embodiment is a person tracking method for tracking a tracking target person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10, in which a Web server 30, which is a computer, acquires a plurality of types of status information detected by a human detection information extraction function unit 22 as an information extraction unit that detects, from each of the surveillance images, a plurality of types of status information related to the health status of a person included in each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10, and determines, based on the acquired plurality of types of status information, whether or not a person included in each of the surveillance images is a specific person corresponding to a prescribed symptom, and, based on the result of the determination, registers, as a tracking target image, an image of a specific person corresponding to a prescribed symptom from among the images of people extracted from the surveillance images of each of the plurality of surveillance cameras 10, in a file server 31, which is a memory, and detects an image of a person similar to the registered tracking target image from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10, and generates a tracking alert screen 43 for presenting an image of a person similar to the detected tracking target image. This makes it possible to detect and track people who meet certain symptoms from surveillance images captured by the surveillance camera 10.

[Other embodiments]
In the embodiment, the detected person card 421 presented on the detection alert screen 42 is for a person whose body temperature, pulse rate, or cough symptom is equal to or higher than any of the presentation thresholds, but the condition may be that all of the presentation thresholds are equal to or higher. In this case, the processing procedure of the detection alert output function unit 3521 may be the procedure shown in the flowchart of FIG. 23 instead of the procedure shown in the flowchart of FIG. 14B. That is, if it is determined in step S208 that the corrected body temperature is not equal to or higher than the designated body temperature, if it is determined in step S211 that the corrected pulse rate is not equal to or higher than the designated pulse rate, or if it is determined in step S212 that the corrected cough symptom is not equal to or higher than the designated cough symptom, the detection alert output function unit 3521 does not create a detected person card and proceeds to the determination process of step S210. Then, only if it is determined in step S208 that the corrected body temperature is equal to or higher than the designated body temperature, if it is determined in step S211 that the corrected pulse rate is equal to or higher than the designated pulse rate, and if it is determined in step S212 that the corrected cough symptoms are equal to or higher than the designated cough symptoms, the detection alert output function unit 3521 proceeds to step S209 to create a detected person card.

In this way, the detection alert output function unit 3521 selects, for all of the presentation thresholds for each acquired type, images of specific persons for which condition information equal to or exceeds the presentation threshold as images to be presented on the detection alert screen 42. This makes it easy to narrow down the persons who are likely to have the specified symptoms.

In addition, in the above embodiment, the frame image 3111, the whole-body image 3112, the whole-body features 3113, and the facial features 3114 are stored as past search data in the past search data storage unit 311 of the file server 31, but the facial image from which the facial features 3114 are extracted may also be stored.

The order of the processing steps shown in the flowcharts of Figures 13 to 14B, 17 to 20C, and 23 is merely an example, and is not limited to this order. For example, the processing of step S101 and the processing of step S102 in Figure 13 may be performed in the opposite order, or in parallel. In Figure 18, the processing of steps S412 and S413 and the processing of steps S414 and S415 may be performed in the opposite order, or in parallel. In addition, the processing of steps S504 to S509 in Figure 20A and the processing of steps S510 to S515 in Figure 20B may be performed in the opposite order, or in parallel. In this way, the processing order of each processing step may be changed as long as there is no discrepancy with the preceding or succeeding processing step.

Furthermore, the detection alert screen 42 shown in FIG. 15 (and FIG. 16) and the tracking alert screen 43 shown in FIG. 21 (and FIG. 22) are not limited to the layout and display contents shown in the figures. For example, the detected cough symptom 4216, the face attention stage display 4324, and the whole body attention stage display 4326 are represented by the number of multiple rectangles that are distinguished, but the attention stage may be represented by a distinguished display that changes the color of a single rectangle, such as yellow, orange, or red. Alternatively, the attention stage may be represented by letters, such as A-C or 1-3, instead of the number or color of the rectangles.

In addition, in one embodiment, a full-body image of a person is extracted from a surveillance image acquired by surveillance camera 10, and a facial image of the person is extracted from the extracted full-body image, but this is not limited to this. If a person's face is captured in the surveillance image, the facial image may be extracted without extracting a full-body image.

In one embodiment, the whole-body image features and face image features are quantified attribute information of the person themselves, but this is not limited to this. The whole-body features may include, for example, quantified information associated with a person, such as clothing, belongings (type or color of bag, stroller, etc.), accessories (sunglasses, mask, etc.). Furthermore, the whole-body features may include quantified internal information of a person obtained from an image or from another sensor, such as fever information, pulse rate, etc. Furthermore, the face features may include quantified information associated with the face, such as accessories (glasses, hat, etc.).

In addition, the whole-body image features and face image features may only be quantified attribute information of the person itself, and the quantified associated information or quantified internal information of the person may be used as separate features, and these separate features may be used as one of the detection criteria similar to the whole-body image features and face image features.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

1... Person tracking system 10... Surveillance camera 20... Video analysis function unit 21... Image acquisition module 22... Person detection information extraction function unit 23... Surveillance/tracking execution function unit 30... Web server 31... File server 32... Database (DB) server 33... Person detection result storage function unit for search 34... Detection/tracking result determination function unit 35... Web application 40... Surveillance terminal 41... Web browser 42... Detection alert screen 43... Tracking alert screen 221... Whole body detection module 222... Area tracking module 223... Whole body feature extraction module 224... Face detection module 225... Face feature extraction module 226... Body temperature detection module 227... Pulse detection module 228... Cough detection module 231... Whole body matching module 232... Face matching module 251... Tracking table 301A... Hardware processor 301B... Program memory 302... Storage device 303: Communication interface device 304: Bus 311: Past search data storage unit 312: Detection history data storage unit 313: Management data storage unit 321: Camera information table storage unit 322: Past search data table storage unit 323: Detection history table storage unit 324: Management table storage unit 325: Tracking table storage unit 351: Login function unit 352: Monitoring function unit 353: Tracking function unit 354: Past search function unit 421: Detected person card 422, 433: Narrowing instruction area 431: Tracking target person information area 432: Tracking information card 3111: Frame image 3112: Whole body image 3113, 3124, 3134: Whole body feature amount 3114, 3122, 3132: Facial feature amount 3115: Body temperature 3116: Pulse rate 3117: Coughing symptoms 3121... Detected face image 3123... Detected whole body image 3131... Registered face image 3133... Registered whole body image 3211... Camera information table 3221... Past search data table 3231... Detection history table 3241... Management table 3251... Tracking table 3521... Detection alert output function section 3531... Image registration function section 3532... Tracking alert output function section 4211, 4311... Warning label 4212, 4312... Suspect ID
4213...Tracking candidate image 4214...Detected body temperature 4215...Detected pulse rate 4216...Detected cough symptoms 4217...Tracking start button 4221...Body temperature presentation threshold setting slider 4222...Pulse rate presentation threshold setting slider 4223...Cough symptoms threshold setting button 4313...Registered image 4314...Tracking information 4315...Tracking stop button 4321...Detection date and time information 4322...Detected camera location information 4323...Detected face image 4324...Face warning stage display 4325...Detected whole body image 4326...Whole body warning stage display 4327...Detected background image 4331...Face image similarity selection button 4332...Face image similarity selection reset button 4333...Whole body image similarity selection button 4334...Whole body image similarity selection reset button NET...Network SV...Server device

Claims (10)

an information extraction unit that detects multiple types of status information related to the health status of a person included in each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, from the surveillance images, and acquires the multiple types of status information detected by the information extraction unit, and determines whether or not the person included in each of the surveillance images is a specific person corresponding to a prescribed symptom based on the multiple types of status information acquired;
Equipped with
The discrimination unit is a specific person detection device that discriminates that the image of the person extracted from the surveillance images of each of the multiple surveillance cameras is an image of the specific person corresponding to the specified symptoms when any one of the multiple types of status information acquired is equal to or above a predetermined value . The specified person detection device according to claim 1 further comprises a detection output screen generation unit that generates a detection output screen for presenting, based on the discrimination result of the discrimination unit, images of the specified person corresponding to the prescribed symptoms among images of people extracted from the surveillance images of each of the multiple surveillance cameras, together with the condition information. 3. The specified person detection device of claim 2, wherein the detection output screen generation unit acquires a presentation threshold value specified for each type of the multiple types of status information, and narrows down the images to be presented on the detection output screen from among the images of the specific person that the discrimination unit has determined to correspond to the specified symptoms based on the acquired presentation threshold value . The specified person detection device according to claim 3, wherein the detection output screen generation unit selects an image of the specified person for which status information equal to or above any one of the presentation thresholds for each of the acquired types is detected as an image to be presented on the detection output screen. The specified person detection device according to claim 3, wherein the detection output screen generation unit selects, for all of the presentation thresholds for each of the acquired types, an image of the specified person for which status information equal to or exceeds the presentation threshold as an image to be presented on the detection output screen. a camera information storage unit that stores a correction coefficient for correcting the status information for each of the plurality of surveillance cameras;
6. The specified person detection device according to claim 1, wherein the discrimination unit corrects the status information acquired from the information extraction unit using the correction coefficient stored in the camera information storage unit, and determines whether or not the image of the person is an image of the specified person corresponding to the specified symptom based on the corrected status information. 7. The specified person detection device according to claim 1 , wherein the condition information includes at least one of body temperature, pulse rate, and coughing symptoms, each of which is extracted from the monitoring image. A specified person detection device according to any one of claims 1 to 7 ,
The plurality of surveillance cameras;
an analysis unit including the information extraction unit;
A specific person detection system comprising: A method for detecting a specific person to be tracked from surveillance images captured by a plurality of surveillance cameras, comprising:
The computer
an information extraction unit that detects multiple types of status information related to the health status of a person included in each of the surveillance images periodically acquired by each of the multiple surveillance cameras, the multiple types of status information being detected by the information extraction unit;
When any one of the acquired multiple types of condition information is equal to or greater than a predetermined value, the image of the person extracted from the surveillance images of each of the multiple surveillance cameras is determined to be an image of a specific person corresponding to a prescribed symptom.
Specific person detection method. A specified person detection program for causing a computer to operate as each unit of the specified person detection device according to any one of claims 1 to 7 .