JP7612356B2 - Monitoring information processing device, monitoring information processing method, and monitoring information processing program - Google Patents

Description

Embodiments of the present invention relate to a surveillance information processing device, a surveillance information processing method, and a surveillance information processing program that are used in a system in which, for example, multiple surveillance cameras are distributed across a surveillance area.

For example, in large stores such as shopping malls, office buildings, train stations, and other facilities used by many people, a system is known in which multiple surveillance cameras are distributed throughout the surveillance area and suspicious individuals are detected based on the video information obtained by these surveillance cameras.

In such a system, information about a detected person is displayed on a screen so that it can be seen by a supervisor or the like. As a technology for displaying information about a detected person, a technology for displaying images and various information received from an authentication device has been proposed (see, for example, Patent Document 1). In addition to displaying information, Patent Document 1 proposes a technology for outputting a predetermined voice guidance or buzzer sound based on the control of the authentication device.

Patent No. 6151582

However, the technology described in Patent Document 1 merely outputs a predetermined voice guidance or a buzzer sound. Therefore, it is difficult for a monitor to understand the contents of the detection just by listening to the predetermined voice guidance or the buzzer sound. Furthermore, as the number of detections increases, it becomes more difficult to understand the contents of the detection.

This invention was made with the above in mind, and aims to provide a technology that controls notifications according to the person being detected.

In order to solve the above problem, one aspect of the present invention is an apparatus or method for performing information processing related to surveillance based on image data acquired by each of a plurality of distributed sensor devices, which detects a person based on the image data, determines label information corresponding to the characteristics of the detected person based on label data including one or more labels corresponding to the characteristics of the person stored in a memory unit, determines whether the detected person is a notification target based on notification target data including the characteristics of the plurality of people who are notification targets stored in the memory unit and the determined label information, and controls notification in a notification mode corresponding to the determined label information in response to the detection of the detected person being determined to be a notification target.

According to one aspect of the invention, it is possible to control notifications according to the person being detected.

FIG. 1 is a diagram showing an example of the configuration of a monitoring system including a monitoring information processing device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of a Web server device used as a monitoring information processing device according to an embodiment of the present invention. FIG. 3 is a block diagram showing an example of the software configuration of a Web server device used as a monitoring information processing device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the label table. FIG. 5 is a flowchart showing the procedure and contents of the notification process executed by the Web server device. FIG. 6 is a diagram showing an example of characteristics of a person who is a notification target.

The following describes an embodiment of the present invention with reference to the drawings.

[One embodiment]
(Configuration example)
(1) System
FIG. 1 is a diagram showing the overall configuration of a system including a monitoring information processing device according to an embodiment of the present invention.
For example, multiple surveillance cameras C1-Cn are distributed in the corridors and sales floors of large stores such as shopping malls and department stores. The surveillance cameras are an example of a sensor device. The surveillance cameras C1-Cn are attached to, for example, the ceiling or a wall, and output image data captured in their respective imaging ranges. Hereinafter, image data captured by each of the multiple surveillance cameras C1-Cn that are distributed in the same manner is also referred to as captured image data. An area such as a large store where multiple surveillance cameras C1-Cn are distributed in the same manner is an example of a monitored area.

The surveillance cameras C1 to Cn are respectively provided with video analysis engines VE1 to VEn. The video analysis engines VE1 to VEn extract image data of a person having image features similar to those of the person being monitored from the captured image data acquired by the corresponding surveillance cameras C1 to Cn. The person being monitored is an example of a person. Hereinafter, the image data of a person extracted from the captured image data by each of the video analysis engines VE1 to VEn is also referred to as person image data. The person image data includes at least one of face image data and whole body image data. The person image data is also an example of captured image data. For example, the image features of the person being monitored are obtained from a registered image based on the registered image data of the person being monitored. The registered image data includes at least one of face image data and whole body image data.

The registered image data of the monitored person is image data used for comparison with human image data to detect the monitored person. The registered image data of the monitored person is image data showing the monitored person that has been registered in advance. The registered image data of the monitored person may be image data showing the monitored person that is different from the captured image data that is previously imported from a PC or the like by the monitor or the facility manager. The monitor or the facility manager is an example of a user. The registered image data of the monitored person may be image data showing the monitored person that has been selected in advance by the monitor or the manager from among past captured image data. The registered image data of the monitored person may be image data showing the monitored person that has been selected in advance from among captured image data acquired in real time by the monitor or the manager. Each of the video analysis engines VE1 to VEn determines the similarity of the human image based on the human image data to the registered image based on the registered image data based on the image feature amount of the monitored person.

Each of the video analysis engines VE1 to VEn generates a video analysis result. The video analysis result includes person image data. The video analysis result includes a person ID that identifies a monitored person having image features similar to those of a person shown in the person image data. The person ID is an ID given to each person. The person ID may be set as appropriate. The video analysis result includes a camera ID that identifies the surveillance camera that acquired the person image data from among the surveillance cameras C1 to Cn. The video analysis result includes a similarity of the person image based on the person image data to a registered image based on the registered image data.

In addition, each of the video analysis engines VE1 to VEn can also extract people with various characteristics from the captured image data, in addition to the person being monitored, by analyzing the captured image data.

The various characteristics include, but are not limited to, characteristics based on appearance, behavior, and symptoms. The various characteristics include the state of a physically disabled person. For example, the state of a physically disabled person is, but is not limited to, a visually impaired person who uses a cane, a visually impaired person who is accompanied by a guide dog, a hearing impaired person who is accompanied by a hearing dog, a person in a wheelchair, and a person in a wheelchair who is accompanied by a service dog. The various characteristics include the state of a person who is in poor health. For example, the state of a person who is in poor health is, but is not limited to, a person who has fallen, a person who is crouching, and a person who is walking unsteadily. The various characteristics include a state of being lost. A state of being lost is, but is not limited to, a child alone. The various characteristics include a state of symptoms. For example, the state of symptoms is, but is not limited to, a fever, a cough, a sweat, and a heart rate. Fever, cough, sweat, and heart rate are intended to be states that are higher or higher than the standard. Each state included in the various characteristics corresponds to the contents of a label managed in the management table 33 described later. A label is information that indicates a person's characteristics.

In this example, the surveillance cameras C1 to Cn may include various sensors such as a thermo camera. Each of the video analysis engines VE1 to VEn may use sensing data from various sensors such as a thermo camera when extracting a person who is mainly in a symptomatic state. In this example, each of the video analysis engines VE1 to VEn does not require registered image data. In this example, the video analysis result includes person image data related to a person with various characteristics. The video analysis result includes a person ID that identifies a person appearing in the person image data. The person ID may be set arbitrarily. Here, an example is described in which a person appearing in the person image data is tracked using a person ID, but the video analysis result may include an arbitrary ID that tracks a person appearing in the person image data instead of the person ID. The video analysis result includes a camera ID that identifies a surveillance camera among the surveillance cameras C1 to Cn that has acquired the person image data. The video analysis result includes information on characteristics related to a person appearing in the person image data. The information on characteristics is information on the content of one or more labels corresponding to one or more conditions such as a visually impaired person using a cane, a person who has fallen, one child, and a fever.

In addition, the video analysis engines VE1 to VEn do not have to be arranged one-to-one with the surveillance cameras C1 to Cn. Instead, a smaller number of video analysis engines may be arranged for multiple cameras, and the video data from multiple surveillance cameras may be processed collectively by the smaller number of video analysis engines.

The system of one embodiment includes a Web server device SV used as a surveillance information processing device. The Web server device SV is a device that performs surveillance-related information processing based on human image data acquired by each of the distributed surveillance cameras C1 to Cn. The video analysis engines VE1 to VEn are capable of data communication with the Web server device SV via a network NW. The video analysis engines VE1 to VEn transmit video analysis results to the Web server device SV via the network NW. The network NW is, for example, a wired LAN (Local Area Network) or a wireless LAN, but may be any other network.

The Web server device SV may be equipped with video analysis engines VE1 to VEn or one video analysis engine, and the video analysis engines VE1 to VEn or one video analysis engine of the Web server device SV may receive video data from the surveillance cameras C1 to Cn via the network NW and analyze the received video data.

The system of one embodiment includes an output device MT and an administrator terminal OT. The output device MT is connected to a Web server device SV. The output device MT is a device that has a function of displaying information output from the Web server device SV. The output device MT is also a device that outputs information output from the Web server device SV as audio. The output device MT may be a wearable device or smart glasses, for example. The output device MT is an example of a display unit, audio output unit, or output unit. The monitor checks the information output by the output device MT. The administrator terminal OT is connected to the Web server device SV. The administrator terminal OT is a terminal into which the administrator inputs various settings for the Web server device SV.

(2) Web server device SV
FIG. 2 is a block diagram showing an example of a hardware configuration of the Web server SV.
The Web server device SV includes a control unit 1, a program storage unit 2, a data storage unit 3, an input/output I/F (interface) 4, and a communication I/F 5. The elements constituting the Web server device SV are connected to each other via a bus 6.

The control unit 1 corresponds to the central part of the Web server device SV. The control unit 1 has a hardware processor such as a central processing unit (CPU). The control unit 1 has a ROM (Read Only Memory) as a non-volatile memory area. The control unit 1 has a RAM (Random Access Memory) as a volatile memory area.

The program storage unit 2 is configured with a non-volatile memory that can be written to and read from at any time, such as a hard disk drive (HDD) or solid state drive (SSD) as a storage medium. The program storage unit 2 stores programs necessary to execute various control processes. For example, the program storage unit 2 stores a monitoring information processing program.

The data storage unit 3 is configured with a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time. The data storage unit 3 stores management data 31, detection history data 32, a management table 33, a detection history table 34, a camera information table 35, a label table 36, and a notification target table 37.

The management data 31 includes registered image data for each person. The registered image data can be used not only by the web server device SV but also by the video analysis engines VE1 to VEn. The registered image data is associated with the person ID of the person being monitored. The management data 31 can be updated based on the addition or deletion of registered image data.

The detection history data 32 includes detection image data of each person. The detection image data is person image data of a person detected by the control unit 1 as described below. The detection image data is associated with a person ID. The detection history data 32 can be updated based on the addition or deletion of detection image data.

The management table 33 manages records for each person ID in which the person ID and the person name are associated with each other. The person name is a name such as the person's name. The person name can be set appropriately. The label is information that indicates the characteristics of the person. The management table 33 can be updated based on updates to the data that constitutes the record. The management table 33 can be updated based on the addition or deletion of a record.

The detection history table 34 manages records for each detection ID that associate the detection ID, person ID, detection camera ID, detection date and time, etc. with each other. The detection ID is an ID given to each detection image data. The detection ID can be set as appropriate. The detection camera ID is an ID that identifies the monitoring camera that acquired the detection image data. The detection camera ID corresponds to the camera ID. The camera ID is an ID given to each of the monitoring cameras C1 to Cn. The camera ID can be set as appropriate. The detection date and time is the date and time when a person is detected based on the person image data by the control unit 1 as described below. The date and time when a person is detected based on the person image data by the control unit 1 corresponds to the date and time when the person image data is captured by any of the monitoring cameras C1 to Cn. The detection date and time is the date and time for each detection. The detection history table 34 can be updated based on updates to the data that constitutes the record. The detection history table 34 can be updated based on addition or deletion of a record.

The camera information table 35 manages records for each camera ID that associate the camera ID, camera name, restricted area information, etc. with each other. The camera name is a name related to the surveillance camera. For example, the camera name is the name of the installation location of the surveillance camera, etc. The camera name can be set as appropriate. The restricted area information is information indicating that the surveillance camera is installed in a restricted area. The restricted area information can be set as appropriate by the administrator. The camera information table 35 can be updated based on updates to the data that constitutes the record. The camera information table 35 can be updated based on the addition or deletion of a record.

The label table 36 includes one or more labels according to the characteristics of a person. The label table 36 includes one or more labels in each of one or more label groups. The label groups are groups for each item according to type. The label group includes one label according to the characteristics of a person according to the item, or two or more labels according to different characteristics of a person according to the item. By including one or more labels in each of one or more label groups according to type in the label table 36, it becomes possible to determine the characteristics of a person detected by the control unit 1 described later from various perspectives.

The label table 36 includes at least one of one or more label groups including one or more labels assigned to each person and one or more label groups including one or more labels according to the state of the person. Hereinafter, the label group including one or more labels assigned to each person is also referred to as the first label group. The label group including one or more labels according to the state of the person is also referred to as the second label group. When the label table 36 includes the first label group, it becomes possible to appropriately determine the characteristics of the person detected by the control unit 1 described later according to the characteristics assigned to each person in advance. When the label table 36 includes the second label group, it becomes possible to dynamically determine the characteristics of the person detected by the control unit 1 described later according to the state of the person in real time.

For example, the label table 36 includes one or more label groups according to the type in each of one or more categories. The categories correspond to the purpose of monitoring. For example, the categories include, but are not limited to, crime prevention, marketing, heartfulness, and infection control. The label table 36 may include only one category, or may include multiple categories. The label table 36 may include only one label group in each category, or may include multiple label groups. The label table 36 may include only one or more first label groups in each category. Alternatively, the label table 36 may include only one or more second label groups in each category. Alternatively, the label table 36 may include one or more first label groups and one or more second label groups in each category.

The label table 36 includes, for each of one or more first label groups, a person ID associated with each of one or more labels included in the first label group. Each of the one or more labels included in the first label group is associated with one or more person IDs. The one or more person IDs associated with each of the one or more labels included in the first label group may be set appropriately by an administrator. Here, an example is described in which the label table 36 manages information associating each of the one or more labels included in the first label group with one or more person IDs, but this is not limiting. The management table 33 may manage information associating each of the one or more labels included in the first label group with one or more person IDs.

The categories, label groups, and labels can be set appropriately by an administrator. The label table 36 can be updated based on the settings of the categories, label groups, and labels. The label table 36 is an example of label data.

The notification target table 37 includes the characteristics of multiple notification target persons. A notification target person is a person who is a notification target in response to detection. Notification is to notify a monitor. Notification manner includes a notification method and a notification destination. For example, the notification method includes at least one of visual notification, auditory notification, and tactile notification, but is not limited to these. The notification method may be a notification method that can be perceived by a person. The visual notification is various displays and messages such as icons of browser tabs on the display screen of the output device MT, but is not limited to these. The visual notification may include notifications by email, chat, etc. The auditory notification is audio output such as sounds and messages from the output device MT, but is not limited to these. The tactile notification is output of tactile stimuli such as vibrations from a sensor or a contact device equipped in the output device MT. The notification destination is a destination of email, a chat ID, etc.

The characteristics of a person to be notified are set based on one or more labels included in label table 36. In one example, the characteristics of a person to be notified are set based on one label in one of one or more label groups included in label table 36. The one label group may be the first label group or the second label group. By setting the characteristics of a person to be notified based on one label, it is possible to set a wide range of people to be notified. By setting a wide range of people to be notified, it is possible to increase the number of people determined to be people to be notified.

In another example, the characteristics of a person to be notified are set based on a combination of one or more labels in at least two label groups among one or more label groups included in label table 36. The at least two label groups may be in the same category, or may span two or more different categories. The at least two label groups may be composed of only the first label group, or may be composed of only the second label group. The at least two label groups may be composed of the first label group and the second label group. By setting the characteristics of a person to be notified based on a combination of multiple labels, it is possible to set a narrow range of notification targets. By setting a narrow range of notification targets, it is possible to realize notification limited to necessary people.

The characteristics of a person to be notified include characteristics associated with different durations. The duration is the time during which the same person continues to have the same characteristic. The duration may be a point in time or a predetermined time. The length of the predetermined time may be set appropriately. The characteristics of a person to be notified may include different characteristics depending on the duration. For example, a characteristic of a person to be notified that is set based on one or more labels may be the target for a duration of a point in time but not the target for a duration of a predetermined time. The same is true vice versa.

The characteristics of a person to be notified include characteristics associated with the duration of stay in the monitoring area. In this example, the characteristics of a person to be notified in the first detection of a person by the control unit 1 described later differ from the characteristics of a person to be notified in the long-term detection of a person by the control unit 1. The first detection is the first detection in a day in the monitoring area. The first detection is an example of a detection in a state where the duration of stay in the monitoring area is short. The first detection is an example of a current detection. The characteristics of a person to be notified in the first detection are an example of a characteristic of a person to be notified at a single point in time whose duration of stay is long. The long-term detection is a detection due to a stay in the monitoring area that is equal to or longer than the long-term determination reference time in a day. The long-term determination reference time is a reference time for determining a long stay. For example, the long-term determination reference time is 20 minutes, but is not limited to this. The long-term detection is an example of a detection in a state where the stay in the monitoring area is long.

The characteristics of a person to be notified may include different characteristics depending on the length of time the person stays in the monitored area. For example, the characteristics of a person to be notified that are set based on one or more labels will be the target for initial detection but not the target for long-term detection. The same is true vice versa. The characteristics of a person that need to be notified on initial detection and the characteristics of a person that need to be notified on long-term detection differ, but the characteristics of a person to be notified can be set appropriately depending on the length of time the person stays in the monitored area.

The notification target table 37 includes notification modes associated with each of the characteristics of multiple people who are notification targets. The notification mode may differ for each person's characteristics, such as a different display or audio output for each person's characteristics. For example, a different display may be a different blinking pattern of an icon or different message content. A different audio output may be a different sound pattern or different message content. This is to enable the monitor to easily identify what characteristics of a person have been detected. The monitor or administrator can arbitrarily set an appropriate notification mode according to the characteristics of the person who is the notification target, such as setting a more distinctive notification mode for the characteristics of a specific person who is the notification target.

The characteristics of the people who are the notification targets and the notification mode can be set appropriately by the administrator. The notification target table 37 can be updated based on the settings of the characteristics of the people who are the notification targets and the notification mode. The notification target table 37 is an example of notification target data.

The input/output I/F 4 is configured with an interface that connects to the output device MT and the administrator terminal OT.
The communication I/F 5 transmits data between the video analysis engines VE1 to VEn using a communication protocol defined by the network NW under the control of the control unit 1. For example, the communication I/F 5 is configured with an interface compatible with a wired LAN or a wireless LAN.

FIG. 3 is a block diagram showing an example of the software configuration of the Web server SV.
The control unit 1 includes a detection unit 11, an output unit 12, a determination unit 13, a decision unit 14, a judgment unit 15, and a notification control unit 16. Each unit is realized by causing a hardware processor of the control unit 1 to execute a monitoring information processing program stored in the program storage unit 2. Each unit can also be said to be included in the hardware processor. Each unit can also be said to be each function. Note that each unit described below can be read as the control unit 1 or the hardware processor.

The detection unit 11 detects a person based on the person image data. For example, the detection unit 11 acquires a video analysis result from each of the video analysis engines VE1 to VEn. The detection unit 11 detects a person appearing in the person image data based on the person image data included in the video analysis result. If the video analysis result includes a similarity related to a person to be monitored, the detection unit 11 compares the similarity included in the video analysis result with a threshold value. The threshold value is a reference value for determining whether or not a person appearing in the person image data is to be regarded as a person to be monitored. The threshold value can be set appropriately. If the similarity is equal to or greater than the threshold value, the detection unit 11 detects the person appearing in the person image data as a person to be monitored. The detection unit 11 may detect the same person only once. The detection unit 11 may detect the same person multiple times at different times. The detection unit 11 stores the person image data showing the detected person as detected image data in the detection history data 32 in association with a person ID included in the video analysis result.

The detection unit 11 acquires the detection date and time based on the detection. Hereinafter, the date and time when the detection unit 11 first detects a person is also referred to as the first detection date and time or the first detection date and time. The first detection corresponds to the first detection. The date and time when the detection unit 11 last detected the same person is also referred to as the latest detection date and time. The detection unit 11 adds a record to the detection history table 34 in which the detection ID, person ID, detection camera ID, and detection date and time are associated with each other based on the detection of the person. The detection unit 11 uses the person ID included in the video analysis result. The detection unit 11 uses the camera ID included in the video analysis result as the detection camera ID. The detection unit 11 uses the acquired detection date and time. The detection unit 11 adds a record to the detection history table 34 every time it detects a person.

The detection unit 11 may send an instruction to output the detection result to the output unit 12 every time a person is detected.
Each time the detection unit 11 detects the same person for the second or subsequent times, it obtains the first detection date and time associated with the person ID of the person from the detection history table 34. Each time the detection unit 11 detects the same person for the second or subsequent times, it may send the person ID, the first detection date and time, and the latest detection date and time to the determination unit 13.
The detection unit 11 may send the person ID, the camera ID, and the feature information included in the video analysis result for each detection to the determination unit 14. Note that the feature information may be generated by the detection unit 11 based on the person image data instead of the video analysis engines VE1 to VEn.

The detection unit 11 may also detect, based on the video analysis results, that the detected person has the same characteristic for a predetermined period of time. The detection unit 11 may detect, based on the video analysis results for one surveillance camera, that the detected person has the same characteristic for a predetermined period of time. The detection unit 11 may detect, based on the video analysis results for multiple surveillance cameras, that the detected person has the same characteristic for a predetermined period of time.

The output unit 12 outputs the detection results for each of the multiple people detected by the detection unit 11 to a detection alert output area included in the display screen of the output device MT. The detection results are information related to the detection. Based on the first detection of a person, the output unit 12 outputs a new detection result related to this person to the detection alert output area. The output unit 12 outputs the detection results for each of the multiple people to the detection alert output area, arranging them in order of the first detection date and time of each of the multiple people.

An example of information included in the detection result will be described.
The detection result may include a person ID. The output unit 12 may acquire the person ID from the detection unit 11. The detection result may include a person name. The output unit 12 may acquire the person name from the management table 33 based on the person ID. The detection result may include a registered image based on registered image data. The output unit 12 may acquire the registered image data from the management data 31 based on the person ID. The detection result may include a detection image based on detection image data related to the detection at the latest detection date and time. The output unit 12 may acquire the detection image data related to the detection at the latest detection date and time from the detection unit 11. The detection result may include a similarity. The output unit 12 may acquire the similarity from the detection unit 11. The detection result may include information on the history of the detection. The information on the history of the detection may include information on the detection at the first detection date and time. The information on the detection at the first detection date and time includes the first detection date and time and the camera name of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time. The output unit 12 may acquire the first detection date and time from the detection history table 34 based on the person ID. The output unit 12 may acquire, based on the person ID, the detection camera ID of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time from the detection history table 34. The output unit 12 may acquire, based on the detection camera ID, the camera name of the surveillance camera that acquired the detection image data related to the detection at the first detection date and time from the camera information table 35. The information on the detection history may include information on the detection at the latest detection date and time. The information on the detection at the latest detection date and time includes the latest detection date and time and the camera name of the surveillance camera that acquired the detection image data related to the detection at the latest detection date and time. The output unit 12 may acquire the latest detection date and time from the detection unit 11. The output unit 12 acquires the camera ID included in the video analysis result from the detection unit 11. The output unit 12 may acquire, based on the camera ID, the camera name of the surveillance camera that acquired the detection image data related to the detection at the latest detection date and time from the camera information table 35. The information on the detection history may include information on a part or all of the detection dates and times between the first detection date and time and the latest detection date and time. The detection result may include the number of detections. The output unit 12 can obtain the number of detections based on the number of records associated with the same person ID in the detection history table 34.

The output unit 12 outputs the detection result for the person determined to be a long-time resident to a long-time resident alert output area included in the display screen of the output device MT based on the long-time resident determination result by the determination unit 13 described below. The detection result output to the long-time resident alert output area is the same as the detection result output to the detection alert output area.

The determination unit 13 determines whether the interval from the first detection date and time of each of the multiple people to the second and subsequent detection dates and times is equal to or greater than the long-term determination standard time. The determination unit 13 determines that a person detected at an interval equal to or greater than the long-term determination standard time is a long-term resident in the monitoring area. The determination unit 13 sends the determination result of the long-term resident to the output unit 12.

The determination unit 14 determines label information according to the characteristics of the person detected by the detection unit 11 based on the label table 36. The label information is information indicating one or more labels. An example of the processing by the determination unit 14 will be described later.

The determination unit 15 determines whether or not the person detected by the detection unit 11 is a notification target based on the notification target table 37 and the label information determined by the determination unit 14. An example of the processing by the determination unit 15 will be described later.

In response to the person detected by the detection unit 11 being determined by the determination unit 15 to be a notification target, the notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14. An example of the processing by the notification control unit 16 will be described later.

An example of the configuration of the label table 36 will be described.
FIG. 4 is a diagram showing an example of the label table 36. As shown in FIG.
The label table 36 includes categories relating to crime prevention, marketing, compassion, and infection control.

The label table 36 includes a label group related to crime prevention history in the crime prevention category. The label group related to crime prevention history includes multiple labels according to the risk of a person. The label group related to crime prevention history includes the labels danger, caution, and caution. Each of the labels danger, caution, and caution is associated with one or more person IDs by the administrator. The label group related to crime prevention history is an example of a first label group. Note that information associating each of the labels danger, caution, and caution with one or more person IDs may be managed in the management table 33.

The label table 36 includes a label group related to restricted area management in the crime prevention category. The label group related to restricted area management includes multiple labels according to whether or not the relevant person is allowed to enter the restricted area. The label group related to restricted area management includes each label for relevant persons and non-related persons. Each label for relevant persons and non-related persons is associated with one or more person IDs by the administrator. The label group related to restricted area management is an example of a first label group. Note that information associating each label for relevant persons and non-related persons with one or more person IDs may be managed in the management table 33.

The label table 36 includes a label group related to purchase amount in the marketing category. The label group related to purchase amount includes multiple labels corresponding to past purchase amounts. The label group related to purchase amount includes each of the labels VVIP, VIP, diamond, gold, silver, and bronze. Each of the labels VVIP, VIP, diamond, gold, silver, and bronze is associated with one or more person IDs by the administrator. The label group related to purchase amount is an example of a first label group. Note that information associating each of the labels VVIP, VIP, diamond, gold, silver, and bronze with one or more person IDs may be managed in the management table 33.

The label table 36 includes a label group related to visit frequency in the marketing category. The label group related to visit frequency includes multiple labels according to past visit frequency. The label group related to visit frequency includes the labels 30 times or more, 20 times or more, 10 times or more, 1 time or more, and 0 times (newcomer). Each of the labels 30 times or more, 20 times or more, 10 times or more, and 1 time or more is associated with one or more person IDs by the administrator. The label 0 times or more is not associated with a person ID, but is associated with a person ID other than the multiple person IDs associated with the labels 30 times or more, 20 times or more, 10 times or more, and 1 time or more. The label group related to purchase amount is an example of a first label group. Note that information associating each of the labels 30 times or more, 20 times or more, 10 times or more, and 1 time or more with one or more person IDs may be managed in the management table 33.

The label table 36 includes a label group related to complaint frequency in the marketing category. The label group related to complaint frequency includes multiple labels according to past complaint frequencies. The label group related to complaint frequency includes the labels 10 or more, 5 or more, and 0. Each of the labels 10 or more, 5 or more, and 0 is associated with one or more person IDs by the administrator. The label group related to complaint frequency is an example of a first label group. Note that information associating each of the labels 10 or more, 5 or more, and 0 with one or more person IDs may be managed in the management table 33.

The label table 36 includes a label group related to the number of visitors in the marketing category. The label group related to the number of visitors includes multiple labels corresponding to the current number of visitors. The label group related to the number of visitors includes the labels 1 person, 2 people, 3 people, and 4 people or more. The multiple labels included in the label group related to the number of visitors correspond to the state of the person to be detected, and therefore are not associated with one or more person IDs. The label group related to the number of visitors is an example of a second label group.

The label table 36 includes a label group for physically disabled people in the heartfelt category. The label group for physically disabled people includes multiple labels according to the current status of the visitor. The label group for physically disabled people includes the labels of a visually impaired person who uses a cane, a visually impaired person accompanied by a guide dog, a hearing impaired person accompanied by a hearing dog, a person in a wheelchair, and a person in a wheelchair accompanied by a service dog. The multiple labels included in the label group for physically disabled people correspond to the status of the person to be detected, and are therefore not associated with one or more person IDs. The label group for physically disabled people is an example of a second label group.

The label table 36 includes a label group for people who are unwell in the heartfelt category. The label group for people who are unwell includes multiple labels according to the current state of the visitor. The label group for people who are unwell includes the labels of a person who has fallen, a person who is crouching, and a person who is staggering. The multiple labels included in the label group for people who are unwell correspond to the state of the person being detected, and are therefore not associated with one or more person IDs. The label group for people who are unwell is an example of a second label group.

The label table 36 includes a label group for lost children in the heartfelt category. The label group for lost children includes one label corresponding to the current state of the visitor. The label group for lost children includes a label for one child. The labels included in the label group for lost children correspond to the state of the person being detected, and are therefore not associated with one or more person IDs. The label group for people who are unwell is an example of a second label group.

The label table 36 includes a label group related to symptoms in the infection control category. The label group related to symptoms includes multiple labels corresponding to the current condition of the visitor. The label group related to symptoms includes the labels fever, cough, sweat, and heart rate. The multiple labels included in the label group related to symptoms correspond to the condition of the person being detected, and are therefore not associated with one or more person IDs. The label group related to people who are feeling unwell is an example of a second label group.

(Example of operation)
Next, an example of the operation of the Web server device SV configured as above will be described.
Fig. 5 is a flowchart showing the procedure and contents of the notification process executed by the Web server device SV. For the sake of simplicity, the flowchart in Fig. 5 shows the notification process based on one person detection, but in reality, the process shown in Fig. 5 is repeated every time a person is detected.

The detection unit 11 detects a person based on the person image data (step S1). In step S1, for example, the detection unit 11 detects a person appearing in the person image data based on the person image data as described above. The detection unit 11 can send the person ID, camera ID, and feature information to the determination unit 14 for each detection.

The determination unit 14 determines label information according to the characteristics of the person detected by the detection unit 11 based on the label table 36 (step S2). In step S2, for example, the determination unit 14 receives information on the person ID, camera ID, and characteristics from the detection unit 11 each time the detection unit 11 detects a person.

The determination unit 14 refers to the person ID and acquires one or more labels corresponding to the person ID based on the label table 36 or the management table 33. The one or more labels acquired by the determination unit 14 with reference to the person ID correspond to the labels included in each of the one or more first label groups. Note that if the camera ID from the detection unit 11 is associated with restricted area information in the camera information table 35, the determination unit 14 acquires a label associated with the person ID from a label group related to restricted area management. The determination unit 14 can acquire one label from one first label group. The determination unit 14 may acquire multiple labels from one first label group depending on the items of the label group.

The determination unit 14 refers to the feature information and acquires one or more labels corresponding to the feature information based on the label table 36. The one or more labels acquired by the determination unit 14 with reference to the feature information correspond to the labels included in each of the one or more second label groups. The determination unit 14 can acquire one label from one second label group. The determination unit 14 may acquire multiple labels from one second label group depending on the items of the label group.

The case where the determination unit 14 acquires one label will be described. The determination unit 14 determines label information based on one label in one of the one or more label groups included in the label table 36. The one label may be one label acquired from the first label group, or one label acquired from the second label group.

A case will be described where the determination unit 14 acquires multiple labels. The determination unit 14 determines label information based on a combination of one or more labels in each of at least two label groups among the one or more label groups included in the label table 36. The at least two label groups may be only at least two first label groups. The at least two label groups may be only at least two second label groups. The at least two label groups may be both one or more first label groups and one or more second label groups. The at least two label groups may be within the same division, or may span two or more different divisions.

The determination unit 14 may determine the label information according to the characteristics of the person at any one detection timing. Alternatively, the determination unit 14 may determine the label information according to the characteristics of the person at multiple detection timings for the same person. In the latter case, the determination unit 14 may determine the label information at a second detection timing after the first detection timing based on a combination of one or more labels according to the characteristics of the person at the first detection timing and one or more labels according to the characteristics of the person at the second detection timing. For example, the determination unit 14 can determine the label information based on a combination of one or more labels in each of a label group related to the number of visitors and a label group related to a lost child. In this example, the label in the label group related to the number of visitors is a label according to the characteristics of the person at the first detection timing. The label in the label group related to a lost child is a label according to the characteristics of the person at the second detection timing.

The determination unit 14 determines label information based on one or more labels obtained as described above. The determination unit 14 sends the determined label information to the judgment unit 15.

The determination unit 15 determines whether or not the person detected by the detection unit 11 is a notification target based on the notification target table 37 and the label information determined by the determination unit 14 (step S3). In step S3, for example, the determination unit 15 compares a combination of one or more labels indicated by the label information determined by the determination unit 14 with a combination of one or more labels set for each of the characteristics of the multiple notification targets included in the notification target table 37. The determination unit 15 determines whether or not there is a characteristic of the notification target person to which one or more labels that match all of the one or more labels indicated by the label information are set in the notification target table 37. The determination unit 15 may determine whether or not there is a characteristic of the notification target person to which one or more labels that match some of the one or more labels indicated by the label information are set in the notification target table 37. Hereinafter, a characteristic of the notification target person to which one or more labels that match all or some of the one or more labels indicated by the label information are set is also referred to as a characteristic of the notification target person corresponding to the label information. If there are no characteristics of the person who is a notification target that correspond to the label information, the judgment unit 15 judges that the person detected by the detection unit 11 is not a notification target. If there are characteristics of the person who is a notification target that correspond to the label information, the judgment unit 15 judges that the person detected by the detection unit 11 is a notification target. The judgment unit 15 sends the characteristics of the person who is a notification target that correspond to the label information to the notification control unit 16.

In response to the judgment unit 15 judging that the person detected by the detection unit 11 is not a notification target (step S4, NO), the processing ends. In response to the judgment unit 15 judging that the person detected by the detection unit 11 is a notification target (step S4, YES), the notification control unit 16 controls notification using a notification mode corresponding to the label information determined by the determination unit 14 (step S5). In step S5, for example, the notification control unit 16 determines a notification mode associated with the characteristics of the person who is a notification target corresponding to the label information based on the notification target table 37. The notification mode associated with the characteristics of the person who is a notification target corresponding to the label information is an example of a notification mode corresponding to the label information.

The notification control unit 16 controls notification of a person detected by the detection unit 11 in a determined notification mode. For example, the notification control unit 16 displays an icon or a message on the output device MT based on the notification mode determined according to the characteristics of the detected person. Alternatively or in addition, for example, the notification control unit 16 outputs a sound or a message from the output device MT based on the notification mode determined according to the characteristics of the detected person. Note that the notification control unit 16 may control the display of the detection result of a person detected by the detection unit 11, which is output to the detection alert output area or the long stay alert output area, based on the determined notification mode. In this example, the notification control unit 16 may control the display of an icon or a message in the detection result, or the display of the detection result, such as lighting or blinking.

FIG. 6 is a diagram showing an example of characteristics of a person who is a notification target.
The notification target table 37 includes characteristics of people who are to be notified based on the “danger” label from the label group related to the crime prevention history included in the label table 36. These characteristics are characteristics of people who are to be notified upon initial detection.
In this example, the determination unit 14 determines label information based on the "danger" label according to the characteristics of the person detected for the first time by the detection unit 11. The judgment unit 15 determines that the person has the characteristics of a person who is a notification target with the "danger" label set as indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the monitor to know in real time when a high-risk individual is approaching and take measures to prevent an incident from occurring.

The notification target table 37 includes characteristics of people who are the notification targets, which are set based on the “VVIP” label from among the label groups related to purchase amounts included in the label table 36. These characteristics are the characteristics of people who are the notification targets upon initial detection.
In this example, the determination unit 14 determines label information based on the "VVIP" label according to the characteristics of the person detected for the first time by the detection unit 11. The judgment unit 15 determines that the person has the characteristics of a person who is a notification target with the "VVIP" label set as indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the monitor to grasp in real time the presence of people with a strong desire to buy, and to take measures to proactively sell products.

The notification target table 37 includes the characteristics of the person who is the notification target, which are set based on the “fall” label from the label group related to physically disabled people included in the label table 36. The characteristics are characteristics of the person who is the notification target at a certain point in time. The characteristics are not limited to the characteristics of the person who is the notification target at the first detection.
In this example, the determination unit 14 determines label information based on the "fall" label according to the characteristics of a person detected by the detection unit 11 at any one time. The determination unit 15 determines that the person has characteristics that make him or her a notification target, to which the "fall" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the observer to detect the person who has fallen in real time and take action such as providing rescue.

The notification target table 37 includes characteristics of a person who is a notification target set based on a combination of the label "one child" from the label group related to a lost child and the label "two people" from the label group related to the number of visitors included in the label table 36. This characteristic is a characteristic of a person who is a notification target at a certain point in time. This characteristic is not limited to the characteristics of a person who is a notification target at the first detection.
In this example, the determination unit 14 determines label information based on the "one child" label and the "two children" label according to the characteristics of a person detected by the detection unit 11 at any one time. The judgment unit 15 judges that the person has characteristics that make him or her a notification target, for which a combination of the "one child" label and the "two children" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the monitor to identify lost people in real time and take appropriate action, such as providing protection for them.

The notification target table 37 includes characteristics of people who are notification targets that are set based on the “non-related person” label from the label group related to the restricted area management included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection.
In this example, the determination unit 14 determines label information based on the "non-related person" label according to the characteristics of a person detected by the detection unit 11 and determined as a long-time resident by the determination unit 13. The determination unit 15 determines that the person has the characteristics of a person who is a notification target with the "non-related person" label set as indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows supervisors to grasp in real time whether someone is staying in a restricted area for an extended period of time and take appropriate action, such as sending out patrols.

The notification target table 37 includes characteristics of persons who are notification targets set based on a combination of a label "30 times or more" from a label group related to visit frequency and a label "10 times or more" from a label group related to complaint frequency included in the label table 36. These characteristics are characteristics of persons who are notification targets in long-term detection.
In this example, the determination unit 14 determines label information based on the "30 times or more" label and the "10 times or more" label according to the characteristics of a person who is detected by the detection unit 11 and determined as a long-time resident by the determination unit 13. The determination unit 15 determines that the person has characteristics that make him or her a notification target, for which a combination of the "30 times or more" label and the "10 times or more" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the monitor to grasp in real time the long-term stay of a person who is of interest for marketing purposes and to take appropriate measures.

The notification target table 37 includes characteristics of people who are notification targets that are set based on the “person in a wheelchair” label from the label group related to physically disabled people included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection.
In this example, the determination unit 14 determines label information based on the "person in a wheelchair" label according to the characteristics of a person detected by the detection unit 11 and determined by the determination unit 13 as a long-time resident. The determination unit 15 determines that the person has the characteristics of a person who is a notification target set with the "person in a wheelchair" label indicated by the label information in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the supervisor to grasp in real time if a disabled person is staying for an extended period of time and take action such as providing assistance.

The notification target table 37 includes characteristics of people who are notification targets that are set based on a combination of a “person crouching” label from a label group related to people who are in poor physical condition and a “cough” label from a label group related to infection control measures included in the label table 36. These characteristics are characteristics of people who are notification targets in long-term detection.
In this example, the determination unit 14 determines label information based on the "person crouching" label and the "coughing" label according to the characteristics of the person detected by the detection unit 11 and determined by the determination unit 13 as a long-time resident. The determination unit 15 determines that the person has characteristics that make him or her a notification target, for which a combination of the "person crouching" label and the "coughing" label indicated by the label information is set in the notification target table 37. The notification control unit 16 controls notification in a notification mode according to the label information determined by the determination unit 14.
This allows the monitor to grasp in real time if an unwell person is staying for a long period of time and take action such as providing rescue.

As described above, the characteristics of the person to be notified can be set as appropriate. The characteristics of the person to be notified may be set not only for the immediate response as described above, but also for future use. For example, by setting the characteristics of the person to be notified in relation to marketing, data such as the number of newcomer visitors and people in various states who stay for a long time can be collected. This data can be used for marketing, such as sales strategy planning.

According to one embodiment, the Web server device SV is able to control notifications according to the person being detected. This allows the monitor to not only recognize the occurrence of a new detection, but also to easily understand what characteristics the newly detected person has. The monitor can immediately take appropriate action against the detected person.

The monitoring information processing device may be realized as a single device as described in the above example, or may be realized as multiple devices with distributed functions.

The program may be transferred in a state where it is stored in a device, or in a state where it is not stored in a device. In the latter case, the program may be transferred via a network, or in a state where it is recorded on a recording medium. The recording medium is a non-transitory tangible medium. The recording medium is a computer-readable medium. The recording medium may be in any form, such as a CD-ROM or memory card, as long as it is capable of storing the program and is computer-readable.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

SV: Web server device OT: Administrator terminal MT: Output device NW: Network C1 to Cn: Surveillance camera VE1 to VEn: Image analysis engine 1: Control unit 2: Program storage unit 3: Data storage unit 4: Input/output I/F
5...Communication I/F
6 bus 11 detection unit 12 output unit 13 determination unit 14 decision unit 15 judgement unit 16 notification control unit 31 management data 32 detection history data 33 management table 34 detection history table 35 camera information table 36 label table 37 notification target table

Claims (9)

A surveillance information processing device that processes surveillance-related information based on image data acquired by each of a plurality of distributed sensor devices,
A storage unit that stores label data including one or more labels corresponding to characteristics of a person and notification target data including characteristics of a plurality of people who are notification targets;
A detection unit that detects a person based on the image data;
an output unit that outputs a detection result relating to each of the plurality of people detected by the detection unit to a detection alert output area included in a display screen of a display unit;
a determination unit that determines label information according to characteristics of the person detected by the detection unit based on the label data;
a determination unit that determines whether or not the person detected by the detection unit is a notification target based on the notification target data and the label information determined by the determination unit;
a notification control unit that controls a display of the detection result of the person detected by the detection unit, which is output to the detection alert output area, based on a notification mode corresponding to the label information determined by the determination unit, in response to the person detected by the detection unit being determined to be a notification target by the determination unit; and
A monitoring information processing device comprising: The monitoring information processing device according to claim 1, wherein the label data includes one or more labels in each of one or more groups according to type. The surveillance information processing device according to claim 2, wherein the label data includes at least one of one or more groups including one or more labels assigned to each person and one or more groups including one or more labels according to the state of the person. the notification target data includes a characteristic of a person based on one label in one of the one or more groups;
the determination unit determines the label information based on one label in one group of the one or more groups.
The monitoring information processing device according to claim 2 . the notification target data includes characteristics of a person based on a combination of one or more labels in each of at least two groups among the one or more groups;
the determination unit determines the label information based on a combination of one or more labels in each of at least two groups among the one or more groups.
The monitoring information processing device according to claim 2 . The monitoring information processing device according to claim 1, wherein the characteristics of the people who are the targets of notification include characteristics that vary depending on the duration of stay in a monitoring area in which the sensor devices are distributed. the notification subject data includes notification modes associated with the characteristics of the persons who are the plurality of notification subjects,
The notification control unit determines a notification mode corresponding to the label information based on the notification target data.
The monitoring information processing device according to claim 1 . A surveillance information processing method executed by a surveillance information processing device that processes surveillance-related information based on image data acquired by each of a plurality of distributed sensor devices, comprising:
detecting a person based on the image data;
outputting a detection result for each of the plurality of detected persons to a detection alert output area included in a display screen of a display unit;
determining label information corresponding to the characteristics of the detected person based on label data including one or more labels corresponding to the characteristics of the person stored in a storage unit;
a step of determining whether or not a detected person is a notification target based on notification target data including characteristics of a plurality of notification targets stored in the storage unit and the determined label information;
a step of controlling a display of the detection result of the detected person output to the detection alert output area based on a notification mode corresponding to the determined label information in response to the detected person being determined to be a notification target;
A monitoring information processing method comprising: 8. A monitoring information processing program for causing a processor included in the monitoring information processing device to execute processing of each of the units included in the monitoring information processing device according to claim 1.