JP7758035B2 - Behavior detection system and behavior detection program - Google Patents

Description

This disclosure relates to a behavior detection system and a behavior detection program, and in particular to a technology for effectively notifying users of behavior detected through video surveillance.

In recent years, advances in video surveillance technology have made it possible to detect a variety of events from video data.

For example, a care support system has been proposed that constantly films care recipients in nursing facilities and analyzes the resulting footage using a machine learning model to detect whether the care recipient has assumed a specific posture and notify caregivers of the care recipient's condition (see, for example, Patent Document 1). This technology can reduce the burden on caregivers who monitor care recipients, thereby reducing the effort required to monitor the care recipient and enabling the provision of high-quality care.

However, depending on the type of behavior to be detected, it may not always be easy to properly determine whether that behavior has occurred from a single image. To address this issue, video surveillance devices have been proposed that detect behavior by combining two images (see, for example, Patent Document 2). This makes it possible to properly detect even complex behaviors that are difficult to detect from a single image.

International Publication No. 2017/026308 JP 2016-062131 A

Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", 14 Apr 2017, https://arxiv.org/pdf/1611.08050.pdf. Sijie Yan, Yuanjun Xiong, Dahua Lin, "Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition", 25 Jan 2018, https://arxiv.org/pdf/1801.07455.pdf.

Even if actions appear similar on video, their importance to the monitor is not necessarily uniform, and it can be difficult to determine the importance of an action from video alone. For example, the video surveillance device related to the above-mentioned conventional technology detects the action of unlocking a locked door (Figure 21(a)), and then determines that a burglar has entered the residence when it detects the action of entering through that door (Figure 21(b)).

However, it's not just burglars who unlock doors and enter homes; residents of those homes also engage in similar behavior. Residents also return home far more frequently than burglars. Therefore, if a video surveillance device were to notify the user of such behavior every time it detected it would be notified not only when a burglar breaks in, but also when a resident returns home, resulting in frequent and unnecessary notifications, significantly reducing user convenience and making the system impractical.

In addition, when using machine learning models for video surveillance, in order to detect behavior that suits the user's needs, it takes a huge amount of effort to prepare training data that suits the user's needs, and there is also the problem that it is not necessarily easy to implement machine learning that can finely distinguish between similar behaviors, such as the above-mentioned burglary and the resident returning home.

This disclosure has been made in consideration of the problems described above, and aims to provide a behavior detection system and behavior detection program that can appropriately notify users of the results of behavior detection through video surveillance in accordance with their needs.

In order to achieve the above-mentioned object, a behavior detection system according to one embodiment of the present disclosure comprises an identification means for using a machine learning model to identify which of predetermined behavior classes the behavior of a person included in an image corresponds to; a determination means for determining whether both a first notification condition that specifies for each behavior class whether or not notification of the detection of behavior that corresponds to the behavior class is required, and a second notification condition based on reasons other than the behavior class are satisfied; and a notification means for notifying a predetermined notification destination of the detection of the behavior if the determination result is positive, wherein the second notification condition includes at least one of the degree of congestion around the person, the positional relationship between the position of the person in the image and a specified range, the attributes of the person, the behavior classes of people around the person, and the attributes of people around the person .

In this case, if the image contains multiple people, the identification means may identify which behavior class the behavior of each person belongs to.

The machine learning model may also be a neural network. Preferably, the neural network is a convolutional neural network.

Furthermore, the first notification condition may include a specification that notification is required when the person's behavior corresponds to a combination of multiple behavior classes.

Furthermore, the second notification condition may be specified for each behavior class.

Furthermore, the second notification condition may be specified corresponding to a combination of the multiple behavior classes.

Furthermore, the second notification condition may be a combination of multiple specifications for each of the behavior classes.

Furthermore, the second notification condition may be a combination of multiple specifications corresponding to the combination of the multiple behavior classes.

Furthermore, the behavioral class may include at least one of hitting, kicking, pushing, climbing, crawling, falling, lying down, throwing, running, riding, walking, smartphone operation, sitting, grasping, and talking.

Furthermore, the surroundings of the person may be within a specified range in the image that includes the person.

Furthermore, the people around the person may be people among multiple people included in the image who belong to a cluster that includes the person.

The notification may also include information indicating which behavior class the behavior falls into.

The notification may also include information indicating the importance of the notification, which is set for each specification of whether or not notification is required in the second notification condition.

In addition, a behavior detection system according to another aspect of the present disclosure is characterized by comprising: an identification means for using a machine learning model to identify which of predetermined behavior classes the behavior of each of multiple people included in an image corresponds to; a determination means for determining whether or not both a first notification condition for specifying, for each behavior class, whether or not notification of the detection of behavior corresponding to the behavior class for one of the multiple people is required; and a second notification condition according to a combination of the behavior class for the one person and the behavior classes for the other people among the multiple people are satisfied; and a notification means for notifying a predetermined notification destination of the detection of the behavior if the determination result is positive.
In addition, a behavior detection program according to one embodiment of the present disclosure is a behavior detection program that causes a computer to execute the following steps: an identification step using a machine learning model to identify which of predetermined behavior classes the behavior of a person included in an image corresponds to; a determination step of determining whether both a first notification condition that specifies, for each behavior class, whether or not notification of the detection of behavior that corresponds to the behavior class is necessary, and a second notification condition based on reasons other than the behavior class, are satisfied; and a notification step of notifying a predetermined notification destination of the detection of the behavior if the determination result is positive, wherein the second notification condition includes at least one of the degree of congestion around the person, the positional relationship between the position of the person in the image and a specified range, the attributes of the person, the behavior classes of people around the person, and the attributes of people around the person .

In this way, the second notification condition is used in addition to the first notification condition to determine whether or not a notification is necessary, so that the results of behavior detection through video surveillance can be appropriately notified according to the user's needs.

1 is a diagram illustrating a main configuration of a behavior detection system 1 according to an embodiment of the present disclosure. FIG. 1 is a block diagram illustrating the main configuration of a behavior detection system 1. FIG. 2 is a block diagram illustrating the main functional configuration of the behavior detection system 1. FIG. 2 is a diagram illustrating the main configuration of a machine learning model that uses the behavior identification unit 301 of the behavior detection system 1. 1A to 1C are diagrams illustrating examples of surveillance images received by the behavior detection system 1 and detection results of people included in the surveillance images using a machine learning model. FIG. 6 is a diagram illustrating an example of an "action class" tab 600 on a "notification condition setting" screen 6 displayed for setting an "action class" among notification conditions. FIG. 7 is a diagram illustrating an example of an "action duration tab" 700 on a "notification condition setting" screen 6 displayed for setting the "action duration" of the notification conditions. 10A and 10B are diagrams illustrating a notification operation when an action duration is set. FIG. 9 is a diagram illustrating an example of a "Congestion level tab" 900 on a "Notification condition setting" screen 6 displayed for setting the "congestion level" of the notification conditions. FIG. 10 is a diagram illustrating an example of a "Time Period Tab" 1000 on a "Set Notification Conditions" screen 6 displayed for setting the "Time Period" among the notification conditions. FIG. 11 is a diagram illustrating an example of a "stay in specific area" tab 1100 on a "setting notification conditions" screen 6 that is displayed for setting "stay in specific area" among the notification conditions. 12 is a table illustrating the data structure of a notification condition table 1200. 10 is a flowchart illustrating the operation of the behavior detection system 1. 10A to 10C are diagrams illustrating the "Notification Condition Setting" screen 14 according to a modified example of the present disclosure, and in particular show the screen display and procedure for selecting "Surveillance Camera." 10A to 10C are diagrams illustrating a "notification condition setting" screen 14 according to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying an "action class." (a) and (b) are figures explaining the "Notification Condition Setting" screen 14 relating to a modified example of the present disclosure, and in particular, show the screen display and procedure for specifying the "Notification Condition" corresponding to the "Behavior Class." (a) and (b) are figures explaining the "Notification Condition Setting" screen 14 relating to a modified example of the present disclosure, and in particular, show the screen display and procedure for specifying the "Notification Condition" corresponding to the "Behavior Class." 10A to 10C are diagrams illustrating the "Notification Condition Setting" screen 14 according to a modified example of the present disclosure, and in particular show the screen display and procedure for specifying a combination of "behavior classes." 1 is a diagram illustrating a system configuration in a nursing care facility to which a behavior detection system 1 according to the present disclosure is applied. 10A and 10B are diagrams illustrating examples of detection results of joint information according to a modification of the present disclosure, in the order of the time at which the monitoring images from which the detection was made were captured. 1A and 1B are diagrams illustrating the operation of a video surveillance device according to the prior art, in which (a) shows the action of unlocking a door, and (b) shows the action of entering a residence.

Hereinafter, an embodiment of a behavior detection system according to the present disclosure will be described with reference to the drawings.
[1] Configuration of the Behavior Detection System First, the configuration of the behavior detection system according to this embodiment will be described.

As shown in FIG. 1, the behavior detection system 1 according to this embodiment is a so-called cloud system that sequentially acquires video images (hereinafter referred to as "monitoring images") captured by surveillance cameras 101, 102, and 103 installed on the streets. The surveillance images captured by surveillance camera 101 include, for example, people 111, 112, 113, and 114. The behavior detection system 1 detects people 111 and their behavior from the acquired surveillance images, sorts the detection results as described below, and then notifies the monitoring center 121 of the detection results. Note that instead of a cloud system, the behavior detection system 1 may be installed within the monitoring center 121.

As shown in Figure 2, the behavior detection system 1 comprises a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, etc., connected so that they can communicate with each other via an internal bus 207. The CPU 201 reads and starts up a boot program from the ROM 202, and reads and executes an OS (Operating System), application programs, etc. from the HDD 204 using the RAM 203 as a working storage area.

The NIC (Network Interface Card) 205 performs processing for communicating with the surveillance cameras 101, etc. and the monitoring center 121. The timer 206 performs the timing processing required for the CPU 201 to execute the OS, application programs, etc.

FIG. 3 is a block diagram showing the functional configuration of the behavior detection system 1. As shown in FIG. 3, the behavior detection system 1 includes a behavior identification unit 301, a notification condition determination unit 302, and a notification processing unit 303. When the behavior identification unit 301 acquires a monitoring image from the monitoring camera 101 or the like, it uses a machine learning model to recognize a person from the monitoring image and identifies to which of multiple classes the person's behavior falls. When the behavior identification unit 301 recognizes a person, the notification condition determination unit 302 determines whether the notification condition specified by the monitoring center 121 is satisfied. When the notification condition is satisfied, the notification processing unit 303 notifies the monitoring center 121.
[2] Behavior identification unit 301
The behavior identification unit 301 according to this embodiment periodically acquires surveillance images and their capture dates and times from the surveillance cameras 101, etc., and recognizes people and their behaviors from the surveillance images using a convolutional neural network (CNN) as a machine learning model. Specifically, as shown in Fig. 4, the convolutional layer 401 of the CNN 400 is used to extract features of a surveillance image 411, and the output of the convolutional layer 401 is input to a fully connected layer (output layer) 402, thereby identifying the position and behavior of the person.

This convolutional layer 401 detects characteristic patterns and various characteristic combinations of patterns in the surveillance image 411. The output layer 402 uses the output data of the convolutional layer 401 to identify the position and size of the bounding box surrounding the person included in the surveillance image 411 and the class to which the person's behavior corresponds (hereinafter referred to as the "behavior class").

CNN 400 uses images of people performing various behaviors as training data to machine-learn the position and size of the bounding box representing the area of the person included in the training data image, as well as the behavior class to which the person's behavior corresponds. CNN 400 outputs the probability that the person's behavior corresponds to each pre-specified behavior class.

In this embodiment, behavior classes include "hitting (hitting someone)," "falling (falling down)," "walking (walking)," and "smartphone operation (looking at smartphone)," and the behavior of each person is identified to which behavior class it falls. The number of behavior classes detected for each person may be two or more. For example, if the probability that CNN400 outputs for each behavior class is equal to or greater than a preset threshold, it can be determined that the behavior of that person falls into that behavior class.

When the threshold is 30%, if CNN400 outputs a behavior class with a probability of 75%, the person's behavior is determined to fall into that 75% behavior class. Also, when the threshold is 30%, if CNN400 outputs two behavior classes with a probability of 40%, the person's behavior is determined to fall into both of those 40% behavior classes. Furthermore, when the threshold is 30%, if CNN400 outputs three behavior classes with a probability of 33%, the person's behavior is determined to fall into all of those 33% behavior classes.

For example, in the example of Figure 5(a), surveillance image 500 contains two people 111 and 112, and bounding boxes 501 and 502 indicating the positions of people 111 and 112 are detected. CNN 400 identifies "hit" as the class to which the behavior of each person in bounding boxes 501 and 502 belongs. In addition, the top right corner of surveillance image 500 displays "23:55" as the time of capture.

In the example of Figure 5(a), the date of photographing is attached as separate data to the surveillance image 500 and is not displayed, but the date of photographing may be displayed along with the time of photographing. The time of photographing may also be attached as separate data to the surveillance image 500 and not displayed. In particular, when detecting a person included in the surveillance image 500 or identifying the behavior of the detected person, it is desirable to omit displaying the date of photographing and the time of photographing from the surveillance image 500. The same applies to Figures 5(b) and 5(c).

In the example of Figure 5(b), a person 113 is included in the surveillance image 510, and a bounding box 511 indicating the position of the person 113 is detected. CNN 400 identifies "fall" as the class to which the person's behavior belongs for the bounding box 511. The top right corner of the surveillance image 510 displays "09:43" as the time of capture.

In the example of Figure 5(c), a person 114 is included in the surveillance image 520, and a bounding box 521 indicating the position of the person 114 is detected. CNN 400 identifies two classes, "walking" and "smartphone operation," as the classes to which the person's behavior belongs for the bounding box 521. The top right corner of the surveillance image 510 displays "13:18" as the time of capture.

In this way, the behavior identification unit 301 accepts input of a surveillance image 411 and outputs the position and size of the bounding box for each person included in the surveillance image 411, as well as the class to which the person's behavior belongs. At this time, it also outputs the identifier of the surveillance camera that captured the surveillance image 411 and the date and time the surveillance image 411 was captured.

In addition, the behavior identification unit 301 may acquire surveillance images by requesting surveillance images from the surveillance camera 101, etc., or may acquire surveillance images by having the surveillance camera 101, etc. voluntarily transmit the surveillance images to the behavior identification unit 301.
[3] Notification condition determination unit 302
The notification condition determination unit 302 will be described below by dividing it into two processes: setting the notification condition and determining the notification condition.
(3-1) Setting of Notification Conditions When the behavior identification unit 301 detects a person, the notification condition determination unit 302 receives, from the monitoring center 121, notification conditions for determining whether to notify the monitoring center 121 of that detection. In this embodiment, an example will be described in which the monitoring center 121 accesses the behavior detection system 1 via the Web, causing a notification condition setting screen to be displayed on the browser of the monitoring center 121, and allowing the user to input the notification conditions. However, it goes without saying that the present disclosure is not limited to this, and instead, notification conditions may be set in other ways.
(3-1-1) Behavior Class Figure 6 is a diagram illustrating an example of the notification condition setting screen 6, and particularly shows the "Behavior Class" tab 600. As shown in Figure 6, the "Behavior Class" tab 600 accepts the class of behavior for which notification should be made when detected and the designation of the surveillance camera that will capture the behavior. In the example of Figure 6, of the check boxes 601, two check boxes for the behavior class, "Walking" and "Smartphone Operation," are checked.

In addition, in the list box 602 for specifying the surveillance camera, surveillance camera 103 is specified. Therefore, unless both the behavior classes of "walking" and "smartphone operation" are detected for each person included in the surveillance image captured by surveillance camera 103, even if only one of "walking" or "smartphone operation" is detected, no notification will be sent to the monitoring center 121. If other notification conditions are also specified, notification will be sent to the monitoring center 121 only if those notification conditions are also satisfied.

Regarding notification conditions other than those for the behavior classes described below, the specification may be accepted for all behavior classes in common, or may be accepted for each behavior class. The user may be allowed to select which method to use.
(3-1-2) Behavior Duration As shown in Fig. 7, the behavior duration tab 700 allows the user to set the duration of an action so that the monitoring center 121 is notified when the action of the class specified in the "behavior class" tab 600 continues for a predetermined period of time. For this reason, the behavior duration tab 700 displays a check box for specifying the duration. In addition, the upper right area of the behavior duration tab 700 displays a character string indicating which monitoring camera detected the action and for which behavior class the action duration is set.

In the example of Figure 7, the duration of the "fall" state when a "fall" is detected by the surveillance camera 102 is set to "3 to 4 minutes." Therefore, if a person who has "fallen" is detected continuously for 3 minutes or more in the surveillance image captured by the surveillance camera 102, this fact is notified to the monitoring center 121.

As shown in Figure 8, the person 113 detected in the surveillance image 800 taken at time T0 has "fallen" and is located within the bounding box 801 in the surveillance image 800.

The person 113 who has "fallen" is also detected in the surveillance images 810, 820, and 830 taken at times T1, T2, and T3, and the bounding boxes 811, 821, and 831 surrounding the person 113 match the bounding box 801 surrounding the person 113 photographed at time T0. Therefore, it is estimated that the person 113 remains motionless after falling.

However, since the time elapsed from time T0 to times T1, T2, and T3 is all shorter than the duration (threshold) set in the behavior duration tab 700, it is not possible to determine with sufficient certainty whether person 113 has fallen but is safe, or whether he or she is in danger, such as injured. For this reason, no notification is sent to the monitoring center 121.

In the surveillance image 840 captured at time T4, person 113 is also detected "falling," and the bounding box 841 surrounding person 113 matches the bounding boxes 801, 811, 821, and 831 surrounding person 113 captured at times T0, T1, T2, and T3. Furthermore, since the time elapsed from time T0 to time T4 is longer than the duration (threshold) set in the Behavior Duration tab 700, person 113 may be in danger, possibly injured. Therefore, a notification is sent to the monitoring center 121.

7 illustrates an example of the behavior duration tab 700 in which the duration is specified using a check box, but it goes without saying that the present disclosure is not limited to this, and instead of or in addition to this, a field for specifying the duration numerically may be provided. Furthermore, it goes without saying that the duration specification may be accepted using other GUI (Graphical User Interface) components, etc.
(3-1-3) Congestion Degree As illustrated in Fig. 9 , the congestion degree tab 900 allows the user to specify the number of people around the detected person, thereby accepting the specification of the congestion degree that needs to be notified to the monitoring center 121. In the example of Fig. 9 , when the behavior of the detected person falls into two behavior classes, walking and smartphone operation, and 12 or more people are detected within 8 meters of the detected person, it is determined that the area around the detected person is congested, and since using a smartphone while walking is dangerous, a notification is sent to the monitoring center 121.

Instead of using the distance from a detected person, it is also possible to detect all people in the surveillance image, identify the position of each person in the surveillance image, and use clustering based on the distribution of people's positions to determine the cluster containing the detected person as the people surrounding that person, and then determine the congestion level from the number of people. In this case, it is not necessary for the user to input the distance to identify the area around the person, improving user convenience.

When detecting the number of people in a surveillance image, a bounding box surrounding each person may be detected, and the number of bounding boxes may be used as the number of people. The number of people may also be determined by detecting the skeleton of each person.

9 illustrates a congestion level tab 900 in which the number of people in the vicinity and the distance to the vicinity are specified using check boxes, but it goes without saying that the present disclosure is not limited to this, and instead of or in addition to this, fields may be provided in which the number of people in the vicinity and the distance to the vicinity are numerically specified. Furthermore, it goes without saying that the number of people in the vicinity and the distance to the vicinity may be specified using other GUI components, etc.
(3-1-4) Time Zone The time zone tab 1000 is a screen for inputting judgment conditions (notification conditions) for determining whether or not a notification is required to the monitoring center 121 depending on the time zone in which a monitoring image including a detected person was captured. In the example of Fig. 10, four consecutive time zones are specified, and if the behavior class of the detected person is "beating," a notification is sent to the monitoring center if the monitoring image was captured between 6:00 PM and 6:00 AM.

Furthermore, instead of specifying the time itself, a time period that may vary depending on the season, such as "daytime" or "nighttime," or a time period that may vary depending on the day of the week, such as "rush hour," may be specified. When accepting the specification of multiple time periods, it goes without saying that the multiple time periods do not have to be consecutive, and multiple intermittent time periods may also be accepted. Furthermore, it goes without saying that the specification of time periods may also be accepted using other GUI components, etc.
(3-1-5) Stay in Specific Area The Stay in Specific Area tab 1100 is a screen for inputting a judgment condition (notification condition) for determining whether or not a notification is required to the monitoring center 121 depending on whether or not the detected person is located in a specific area in the monitoring image. In the example of Fig. 11 , if the behavior of the detected person 114 falls into two behavior classes, "walking" and "smartphone operation," and the person 114 is located within a specific area 1102 in the monitoring image 1101, a notification is sent to the monitoring center 121.

In the example of Figure 11, when specifying a specific area 1102 in a surveillance image 1101 captured using a surveillance camera 103, the surveillance image 1101 is displayed as an auxiliary image for the convenience of the user, but the surveillance image 1101 does not have to be displayed. Also, in Figure 11, a case is shown in which the specific area 1102 is rectangular, but the specific area may be a shape other than a rectangle, such as a circle or an ellipse.

If the specific area 1102 is rectangular, the specific area 102 can be specified, for example, by the smallest X coordinate (X1) and Y coordinate (Y1) of the four vertices of the rectangle, and the size of the rectangle in the X and Y directions (Xsize, Ysize). Needless to say, the specific area 102 may also be specified in other ways.

11 illustrates an example in which only one specific area 1102 is specified, but it goes without saying that the present disclosure is not limited to this, and it is also possible to accept the specification of two or more specific areas 1102. It goes without saying that it is also possible to accept the specification of the specific area 1102 using a method other than the above, such as accepting input of coordinate values in a monitoring image.
(3-1-6) Others Other notification conditions may include, for example, person attributes. Examples of person attributes include gender, age, physique, posture, hairstyle, and clothing. Person attributes may be detected using, for example, a machine learning model. In this case, a bounding box surrounding the person in the surveillance image may be extracted and input to the machine learning model.

Furthermore, when referring to people around the person of interest, as in the above-mentioned congestion level, the behavior class to which the behavior of the surrounding people corresponds and the attributes of the surrounding people may be referenced to determine whether to take them into consideration when determining the congestion level. Furthermore, depending on the behavior class to which the behavior of the person of interest corresponds, the behavior class to which the behavior of the surrounding people corresponds may be referenced, and if the behavior class is "sitting," notification to the monitoring center 121 may not be necessary.

By accepting the specifications as described above, a notification conditions table 1200 such as the example shown in Figure 12 is created. In the notification conditions table 1200 shown in Figure 12, one behavior class to be notified to the monitoring center 121 is specified for each surveillance camera number, but it goes without saying that two or more behavior classes may be specified, and the notification conditions other than the behavior class may be the same or different between the two or more behavior classes.

Furthermore, in this embodiment, an example will be described in which a notification is sent to the monitoring center 121 when all notification conditions corresponding to the behavior class for each surveillance camera number are met (when the AND of all notification conditions is met), but it goes without saying that the present disclosure is not limited to this, and a logical expression for the notification condition may be specified that includes not only AND but also OR and NOT.
(3-2) Determination of Notification Conditions The notification condition determination unit 302 determines whether the notification conditions are satisfied for each person identified by the behavior identification unit 301.

For example, if the behavior identification unit 301 identifies the behavior class of person 111 as "attack" for surveillance image 500 (Figure 5(a)) received from surveillance camera 101, the notification condition determination unit 302 references the column in the notification condition table 1200 where the surveillance camera number is 101 and the behavior class is "attack." In the notification condition table 1200, the corresponding notification conditions are specified as a congestion level of "two or more people within one email" from the person of interest and a time period of "6:00 PM to 6:00 AM."

In the surveillance image 500, two people, 111 and 112, are located within a one-meter radius of person 111, and the notification condition for the congestion level, "two or more people within one email," is met. Furthermore, since the image was taken at 23:55, the notification condition for the time period, "from 18:00 to 6:00," is also met. Therefore, the notification condition determination unit 302 determines that the notification condition is met.

When the behavior identification unit 301 identifies the behavior class of person 113 as "fall" for the surveillance image 510 (Figure 5(b)) received from surveillance camera 102, the notification point condition determination unit 302 references the column in the notification condition table 1200 where the surveillance camera number is 102 and the behavior class is "fall." In the notification condition table 1200, the corresponding notification condition is specified as behavior duration of "3 minutes or more."

For this reason, the notification condition determination unit 302 records on the HDD 204 the surveillance image 510, the shooting time "09:43" attached to the surveillance image 510, the position and size of the bounding box 511 surrounding the person 113 identified by the behavior identification unit 301, and the behavior class corresponding to the behavior of the person 113.

If a person is subsequently detected in a surveillance image received from the same surveillance camera 102 surrounded by a bounding box of the same size and position as the bounding box 511, and the behavior class is "falling," the shooting time "09:43" of the surveillance image 510 recorded on HDD 204 is compared with the shooting time of the newly received surveillance image to calculate the behavior duration.

If the calculated behavior duration corresponds to the behavior duration of "3 minutes or more" specified in the notification condition table 1200, the notification condition determination unit 302 determines that the notification condition is met. As a result, the notification processing unit 303 notifies the monitoring center 121. If the calculated behavior duration is less than 3 minutes, the notification condition is not met, so the monitoring center 121 is not notified, and the recording of the monitoring image 510, etc. on the HDD 204 is maintained.

If no person is detected in a surveillance image received from the same surveillance camera 102, if a person is detected but the position or size of the bounding box surrounding the person differs from the position or size of the previously detected bounding box 511, or if the person's behavior class is not "falling," it is determined that the behavior of "falling" by person 113 is not continuing, and the record of surveillance image 510, etc. is deleted from HDD 204, or the record of surveillance image 510, etc. is excluded from the determination of behavior duration. As a result, the "falling" of person 113 included in surveillance image 510 is excluded from the monitoring target for behavior duration.

When the behavior identification unit 301 identifies the behavior class of person 114 as "walking" and "smartphone operation" for the surveillance image 520 (Figure 5(c)) received from surveillance camera 103, the notification point condition determination unit 302 references the column in the notification condition table 1200 where the surveillance camera number is 103 and the behavior class is "walking" and "smartphone operation." In the notification condition table 1200, the corresponding notification conditions are specified as behavior duration of "3 minutes or more," congestion level of "12 or more people within 8 meters," and stay in specific area of "(X1, Y1, Xsize, Ysize)."

In the monitoring image 520, the bounding box 521 surrounding person 114 is included in the range (X1, Y1, Xsize, Ysize) of the specific area specified in the "Dwell in specific area" column of the notification condition table 1200. However, the only person included in the monitoring image 520 is person 114, and the notification condition of "12 or more people within 8 meters" specified in the "Congestion level" column of the notification condition table 1200 is not met.

Therefore, even if "3 minutes or more" is specified in the action duration column of the notification condition table 1200, unlike the case of the monitoring image 510, the monitoring image 520 and the shooting time "13:18" attached to the monitoring image 520 are not recorded on the HDD 204. Therefore, the notification condition determination unit 302 does not determine that the notification condition is satisfied when the monitoring image 520 is captured, and the notification processing unit 303 does not send a notification to the monitoring center 121.
[4] Notification processing unit 303
When the notification condition determination unit 302 determines that the notification conditions specified in the notification condition table 1200 are satisfied, the notification processing unit 303 notifies the monitoring center 121. In other words, the notification processing unit 303 transmits to the monitoring center 121 a signal indicating the presence or absence of an alarm that notifies that a specific behavior has been detected in the monitoring image.

It is desirable that the notification from the notification processing unit 303 to the monitoring center 121 include one or more of the following: identification information of the surveillance camera, surveillance image data, the time and date the surveillance image was taken, the behavior class identified by the behavior identification unit 301, the position and size in the surveillance image of the bounding box surrounding the person who performed the behavior corresponding to the behavior class, and information regarding which notification condition specified in the notification condition table 1200 was satisfied and how.

Furthermore, the notification from the notification processing unit 303 to the monitoring center 121 may include information indicating the importance of an alarm that notifies that a specific behavior has been detected in a monitoring image. For example, as shown in Fig. 12 , the importance of the alarm may be determined by providing a column for specifying the importance in a notification condition table 1200, and the notification processing unit 303 may refer to this column, or the notification condition determination unit 302 may refer to this column and notify the notification processing unit 303 of the importance, so that the notification processing unit 303 may notify the monitoring center 121 of the importance.
[5] Operation of the Behavior Detection System 1 Next, the operation of the behavior detection system 1 will be described.

As shown in FIG. 13, when the behavior detection system 1 acquires a surveillance image from a surveillance camera (S1301: YES), the behavior identification unit 301 checks whether a person is included in the surveillance image. If a person is detected in the surveillance image (1302: YES), the system executes the processing from step S1303 to step S1309 for each person detected in the surveillance image.

That is, the behavior identification unit 301 identifies the behavior class to which the person's behavior corresponds (S1304). The behavior class to which the person's behavior corresponds is identified using the CNN 400, as described above. Next, the notification condition determination unit 302 refers to the notification condition table 1200 to identify the notification conditions corresponding to the combination of the surveillance camera number and the behavior class (S1305), and checks whether all of the identified notification conditions are satisfied (S1306).

If all notification conditions are met (S1307: YES), the monitoring center 121 is notified that the behavior has been detected (S1308). If the notification conditions are not met (S1307: NO), the monitoring center 121 is not notified, and the processing from step S1303 to step S1309 is executed for another person detected in the monitoring image.

Once processing from step S1303 to step S1309 has been completed for all people detected in the surveillance image, return to step S1301 and repeat the processing described above.

In this way, the results of behavior detection by video monitoring using surveillance images can be appropriately notified according to the user's needs by referring to the notification condition table.
[6] Modifications The present disclosure has been described above based on the embodiments, but it goes without saying that the present disclosure is not limited to the above-described embodiments, and the following modifications can be implemented.
(6-1) In the above embodiment, an example was described in which CNN 400 was used as a machine learning model. However, it goes without saying that the present disclosure is not limited to this, and a neural network other than CNN may be used, or a machine learning model other than a neural network may be used.
(6-2) In the above embodiment, the behavior classes detected by the behavior identification unit 301 have been described as “hitting,” “falling,” “walking,” and “smartphone operation” as examples. However, it goes without saying that the present disclosure is not limited to these. Instead of or in addition to these, for example, behavior classes such as “kick,” “push,” “climb,” “crawl,” “lying,” “throw,” “run,” “ride,” “sit,” “grasp,” and “talk” may be provided to detect the behavior of people included in surveillance images.
(6-3) In the above embodiment, an example has been described in which importance is specified for each activity class in the notification condition table 1200. However, the present disclosure is not limited to this example, and the following may be used instead. An importance may be specified for each notification condition, such as "activity duration" or "congestion level." Furthermore, this importance may be specified according to the settings of the notification condition.

For example, in the case of "action duration," if the duration is between 30 and 60 seconds, the importance may be medium, and if it exceeds 60 seconds, the importance may be high. Also, instead of three levels of high, medium, and low, the importance may be set in finer levels, such as numbers from 0 to 1. In such cases, when notifying the monitoring center 121, the importance of each notification condition may also be notified to the monitoring center 121. If the importance differs between notification conditions, the notification condition with the highest importance may be notified to the monitoring center 121.

In addition, the importance of each notification condition may be multiplied by a weight corresponding to each notification condition, and then the weighted importance values may be added together, and the resulting total value may be notified to the monitoring center 121 as the importance of the notification.
(6-4) In the above embodiment, an example was given of determining notification conditions for each person included in a surveillance image, but it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

For example, if a surveillance image contains multiple people, the need to notify the monitoring center 121 may be determined based on the combination of behavior classes corresponding to the behaviors of the multiple people. For example, if a surveillance image shows a person carrying a bag and running, and another person running nearby, this may be a theft, so it is desirable to notify the monitoring center 121, check the situation at the scene, analyze the surveillance image that records the circumstances of the crime, and identify the culprit.

Furthermore, if the surveillance image contains multiple people and the behavioral class to which these people's actions correspond is "driving," there is a possibility that an emergency situation has occurred, so it is desirable to notify the monitoring center 121 to check the situation at the scene and take measures to respond to the situation.

Furthermore, if the attribute of a person included in a surveillance image taken at night is "child" and the behavior class corresponding to the person's behavior is "walking," and no other people are detected around the person in the surveillance image, or only people with the attribute other than "adult" are detected, it is possible that the "child" has become lost or may be involved in a crime, so it is desirable to notify the monitoring center 121.

In this way, the actions of multiple people included in a single surveillance image may be correlated to determine whether or not a notification is necessary. Furthermore, the actions of a person detected using a surveillance image may be correlated with the actions of a person detected using another surveillance image to determine whether or not a notification is necessary. These other surveillance images may be surveillance images captured at a different date and time from the date and time the original surveillance image was captured, using the surveillance camera that captured the original surveillance image.

Furthermore, a monitoring image taken by a monitoring camera other than the one that took the original monitoring image may be combined with the original monitoring image to determine whether or not a notification is required. In this way, it is possible to more accurately identify complex situations and determine whether or not a notification is required than when determining whether or not a notification is required based only on the behavior class to which the behavior of one person included in one monitoring image corresponds and the notification conditions corresponding to that behavior class.
(6-5) In the above embodiment, as illustrated in Figures 6 to 11, an example was described in which notification conditions were set by selecting a tab and clicking a check box. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

For example, as shown in FIG. 14(a), the "Notification Condition Settings" screen 14 displays a "Select Surveillance Camera" pull-down menu 1401. The "Select Surveillance Camera" pull-down menu 1401 is a pull-down menu that allows the behavior detection system 1 to refer to surveillance images and select a surveillance camera that can send a notification to the monitoring center 120.

When you place cursor 1411 over the "Select Surveillance Camera" pull-down menu 1401 and click, a list of surveillance cameras 1402 will be displayed, as shown in Figure 14(b). When you place cursor 1411 over the list of surveillance cameras 1402, the name of the surveillance camera the cursor is over is highlighted, indicating that it can be selected by clicking. In the example of Figure 14(b), "Survival Camera 102" is highlighted among the surveillance camera names, and clicking in this state will allow you to select the surveillance camera with that name.

Note that if there are a large number of selectable surveillance camera names, it is not possible to display all of them at once, so only some of the surveillance camera names are displayed as a list, as shown in Figure 14(b). You can also use the scroll bar 1403 to move the surveillance camera names up and down to display other surveillance camera names in a list.

When a surveillance camera is selected, the string displayed in the "Select Surveillance Camera" pull-down menu 1401 changes to the name of the surveillance camera selected from "Select Surveillance Camera," as shown in Figure 14(c). In the example of Figure 14(c), the display changes to the name of the surveillance camera, "Survival Camera 102."

When a surveillance camera ("surveillance camera 102" in Figure 15(a)) is selected and you click within the "Notification Condition Settings" screen 14, a click menu 1501 is displayed, as shown in Figure 15(b). Click menu 1501 displays a list of behavior classes that can be detected as the behavior of people included in surveillance images captured by the selected surveillance camera.

In the example of Figure 15(b), the behavior classes displayed are "hit," "fall," "walk," and "smartphone operation." Furthermore, the behavior class over which the cursor 1411 is positioned, "fall," is highlighted. Clicking on a highlighted behavior class selects that behavior class. Click menu 1501 then disappears, and the name of the selected behavior class is displayed.

In the example of Figure 15(c), the character string "Fall" is displayed as the name of the behavior class. This character string is also surrounded by a rounded rectangle 1502, indicating that it is the name of the behavior class. Needless to say, the name of the behavior class may be indicated in a manner other than by surrounding it with a rounded rectangle 1502.

Next, by hovering cursor 1411 over the name of an activity class and clicking, a click menu 1601 is displayed, as shown in Figure 16(a), for specifying the notification conditions when the activity of a person included in surveillance images captured by the surveillance camera corresponds to that activity class. In the example of Figure 16(a), the notification conditions "activity duration," "congestion level," "time period," "stay in specific area," and "person attributes" are listed in click menu 1601, and of these notification conditions, "congestion level," which cursor 1411 is hovering over, is displayed in reverse video.

When you click in this state, "Congestion" is selected from the notification conditions, and as shown in Figure 16 (b), a leader line 1511 is drawn from the rounded rectangle 1502 displaying "Fall," and the character string 1512 reading "Congestion" is displayed at the end of this leader line 1511. By repeating the same operation, you can select and display other notification conditions.

In the example of Figure 17(a), in addition to the lead line 1511, a lead line 1701 is drawn, and the character string "behavior duration" 1702 is displayed at the end of this lead line 1701. In this way, the relationship between the behavior class and the notification condition can be interactively specified in a graph structure.

When a character string displaying a notification condition is clicked on the "Notification Condition Settings" screen 14, a screen for setting that notification condition is displayed. For example, as shown in Figure 17(b), when the character string "Behavior Duration" 1702 is clicked, the behavior duration tab 700 shown in Figure 7 is displayed. In this case, the character string "Fall" is displayed in the behavior class column of the behavior duration tab 700, in line with the character string "Fall" enclosed in the rounded rectangle 1502 from which the extension line 1701 leading to the character string "Behavior Duration" 1702 is drawn.

In addition, since "Monitoring Camera 102" is displayed in the "Select Surveillance Camera" pull-down menu 1401, the string "Monitoring Camera 102" is displayed in the Surveillance Camera column of the Behavior Duration tab 700. This allows the user to set detailed behavior duration conditions as notification conditions when the behavior class corresponding to the behavior of a person included in surveillance images captured by surveillance camera 102 is "falling."

As described above, according to the above embodiment and this modified example, notification conditions can be specified interactively using check boxes, pull-down menus, value specification, screen area specification, graph structures, etc.

When combining multiple activity classes, you can also do the following. That is, by selecting the activity class names in the same way as in Figure 15(b) described above, the names of the activity classes you want to combine can be displayed on the "Notification Condition Settings" screen, and then you can combine these activity classes. In the example of Figure 18(a), the activity classes you want to combine, "Walking" and "Smartphone Operation," are each displayed on the "Notification Condition Settings" screen 18.

Next, as shown in Figure 18(b), a rectangle 1803 enclosing the activity classes "Walking" and "Smartphone Operation" is displayed by dragging from the top left (position of cursor 1812) to the bottom right (position of cursor 1813) of the activity class "Walking" on the "Notification Condition Settings" screen 18. During the drag operation, the range of activity classes to be combined is undetermined, so rectangle 1803 is displayed as a dashed line, and the start point of the drag operation (position of cursor 1812) is fixed, but the end point is undetermined.

After that, when the drag operation is completed, a rectangle 1804 is displayed in a solid line at a position and size corresponding to the rounded rectangles 1801 and 1802 that respectively enclose the behavior classes "walking" and "smartphone operation" that were enclosed by rectangle 1803 at that time (Figure 18(c)). The dashed rectangle 1803 is erased. If you then want to set notification conditions, clicking on rectangle 1804 will display candidate notification conditions for the combination of "walking" and "smartphone operation," as in the example of Figure 16(a).

In this way, according to this modification, the notification conditions can be set by intuitive operations even for combinations of behavior classes, thereby improving user convenience.
(6-6) In the above embodiment, an example was given in which surveillance images taken using a surveillance camera 101 or the like installed on the street were used. However, it goes without saying that the present disclosure is not limited to this, and surveillance images taken by a surveillance camera installed in a location other than the street may be used instead.

For example, the behavior of the care recipient may be monitored using surveillance images captured by a surveillance camera installed in a nursing facility. As shown in Figure 19, the monitoring support system MS is a system for supporting caregivers in monitoring the care recipients at a nursing facility, and is configured so that the rooms RM of each person being monitored Ob, the nurse station ST, and mobile terminals TA installed in each caregiver NS are connected to each other via a communication network NW so that they can communicate with each other.

A sensor box SB (SB-1 to SB-4) is installed in each room RM (RM-1 to RM-4) of the person being monitored Ob (Ob-1: Mr. A to SB-4: Mr. D). The sensor box SB is equipped with a camera and captures images of the inside of the room RM. When the person being monitored Ob is present in the room, the image captured by the sensor box SB will include the image of the person being monitored Ob.

The sensor box SB inputs the captured surveillance image into a machine learning model such as CNN, and outputs a bounding box surrounding the monitored person Ob in the surveillance image and the behavior class to which the monitored person Ob's behavior corresponds.

The sensor box SB is configured to determine whether or not notifications are required for each behavior class, and when it detects behavior that corresponds to a behavior class requiring notification, it sends information to the management server SV, such as the surveillance image, the position and size of the bounding box for the monitored person Ob contained in the surveillance image, the behavior class to which the behavior of the monitored person Ob corresponds, and the date and time the surveillance image was taken.

The sensor box SB may detect, for example, the behavior of the monitored person Ob going to sleep, waking up, falling out of bed, wandering around, or falling outside of bed.

When the management server SV receives information such as surveillance images from the sensor box SB, it checks whether the notification conditions set for each sensor box SB and each behavior class are met. These notification conditions may also include registered notification destinations when the notification conditions are met.

The fixed terminal device SP is used to set notification conditions in the sensor box SB and management server SV. The notification conditions set in the sensor box SB are, for example, the behavioral classes that require notification, as described above. Furthermore, the notification conditions set in the management server SV include the time period and the range of measurement values such as body temperature, blood pressure, and pulse rate measured by a biometric measurement device.

The biometric device performs biometric measurements while attached to the monitored person Ob, and can notify the sensor box SB and management server SV of the obtained measurement values. The biometric device may be attached to the monitored person Ob at all times and perform biometric measurements at pre-set times, or it may be detached as needed and perform biometric measurements only when attached. When the biometric device is detached from the monitored person Ob, it may notify the sensor box SB, etc., of this.

In this modified example, when the notification conditions are met, the management server SV sends a notification to the mobile terminal TA carried by the caregiver NS, which is the notification destination set in accordance with the notification conditions. The mobile terminal TA to which the notification is sent may be one or more. In the case of an emergency, the notification may be sent to all mobile terminals TA. In addition, the importance level may be set in accordance with the notification conditions, and the type and volume of the ringtone output by the mobile terminal TA to which the notification is sent may be changed depending on the importance level.

Care recipients housed in nursing facilities may behave in a variety of ways depending on their physical condition and other factors, but even in such cases, by applying the present disclosure, appropriate notifications can be made according to the condition of the care recipient, thereby reducing the burden on caregivers and providing high-quality care.
(6-7) In the above embodiment, the case where the surveillance camera 101 and the behavior detection system 1 are separate entities has been described as an example. However, the present disclosure is not limited to this, and the following may be adopted instead. For example, a behavior detection system may be built into each surveillance camera. Furthermore, the behavior detection system built into a surveillance camera may send notifications to multiple destinations, and in this case, the notification conditions may be different for each destination.
(6-8) In the above embodiment, the case where the notification destination of the behavior detection system 1 is the monitoring center 121 has been described as an example, but it goes without saying that the present disclosure is not limited to this, and notifications may be sent to other notification destinations instead of or in addition to the monitoring center 121. In this case, the notification destination may be a mobile communication device such as a specimen terminal.
(6-9) In the above embodiment, the position and size of a bounding box surrounding a person in a surveillance image and the behavior class to which the person's behavior corresponds are identified as an example. However, the present disclosure is not limited to this, and the following may also be used. For example, the identity of a person may be detected between surveillance images captured by the same surveillance camera at different dates and times. In this way, even if the positions and sizes of the bounding boxes match, if the people surrounded by the bounding boxes differ between the surveillance images, it is possible to prevent erroneous detection of the duration of the behavior.
(6-10) In the above variant, an example was described in which a convolutional neural network, CNN400, is used to identify people included in a surveillance image and the behavioral classes to which their behavior corresponds. However, it goes without saying that the present disclosure is not limited to this, and the following may be used instead.

For example, joint information of a person may be extracted from a surveillance image, and the obtained joint information may be used to recognize the behavior of the person.

There are two methods for extracting joint information: top-down and bottom-up. With the top-down method, each person in the surveillance image is first identified, and then the joints for each identified person are extracted. On the other hand, with the bottom-up method, joints are extracted directly from the surveillance image without identifying the people in the image, and then the extracted joints are associated with each other to identify each person's joints.

One method for extracting joint information is, for example, OpenPose (see Non-Patent Document 1), which uses a bottom-up approach. OpenPose extracts joint information from images using a neural network. OpenPose can use the Caffe (Convolutional Architecture for Fast Feature Embedding) model published by Carnegie Mellon University.

As a method for recognizing human behavior from joint information, for example, ST-GCN (Spatial Temporal Graph Convolutional Networks) can be used (see Non-Patent Document 2). In this case, video images are used as monitoring images, and joint information is extracted from the video images using OpenPose. ST-GCN estimates the behavior class to which each person's behavior corresponds from the joint information in the video images.

As shown in Figure 20, joints 2003, 2005, etc. are extracted as joint information 2000, 2010, and 2020 from a series of surveillance images that make up the video, and the connection relationship 2004 between joints 2003 and 2005, etc. is sequentially identified from their positional relationships. For example, joint information 2000 includes two people 2001 and 2002.

Similarly, joint information 2010 includes people 2011 and 2012, and joint information 2020 includes people 2021 and 2022, each of which specifies a joint and its connection relationship. From such joint information, an action class (e.g., "hit") corresponding to the action of the two people is estimated.

Note that joint information 2000, 2010, and 2020 are images that represent the position of each joint in the monitoring image and its connection relationships.When processed by a computer, the joint information, such as the coordinate values of each joint and an ID representing the connection relationships, is stored in a recording medium such as RAM 203 or HDD 204.

By using the behavioral classes estimated for each person in this way, it is possible to determine whether or not a notification is necessary, in the same way as in the above embodiment. Therefore, the results of behavior detection through video surveillance can be appropriately notified to suit the user's needs.

It goes without saying that the method for extracting joint information from surveillance images is not limited to OpenPose, and methods other than OpenPose may be used, and furthermore, methods other than the bottom-up approach may be used. It goes without saying that the method for estimating behavior classes from joint information is not limited to ST-GCN, and methods other than ST-GCN may be used.

Furthermore, it goes without saying that the machine learning model used to extract joint information from surveillance images and the machine learning model used to recognize human behavior from joint information are not limited to convolutional neural networks; neural networks other than convolutional neural networks may also be used, and machine learning models other than neural networks may also be used.
(6-11) In the above embodiment, the case of detecting a person's behavior has been described as an example, but it goes without saying that the present disclosure is not limited to this, and the following may be used instead of a person's behavior.

For example, when using a surveillance camera to detect the behavior of vermin such as bears, wild boars, or monkeys through video, a warning sound may be emitted to scare off the vermin depending on the time of year (such as the growing season of agricultural crops).Furthermore, when detecting the behavior of machinery such as automobiles through video, if a car stopped at an intersection is detected and does not move off within a predetermined time after the green light of the traffic signal has turned on, a traffic control center may be notified since there may be an abnormality with the driver or vehicle.
(6-12) As described above, the behavior detection system 1 according to the present disclosure is a computer system including a microprocessor and a memory. The memory may store a computer program, and the microprocessor may operate according to the computer program.

Here, a computer program is composed of a combination of multiple instruction codes that indicate instructions to a computer to achieve a specified function.

The computer program may also be recorded on a computer-readable recording medium, such as a flexible disk, hard disk, optical disk, semiconductor memory, etc.

Computer programs may also be transmitted via wired or wireless telecommunications lines, networks such as the Internet, data broadcasting, etc.

The present invention may also be a method used by the behavior detection system 1 by executing the computer program.
(6-13) The above-described embodiments and modifications may be combined with each other.

The behavior detection system and behavior detection program disclosed herein are useful as a technology for effectively notifying users of behavior detected through video surveillance.

1................................................................................... Behavior detection system 101, 102, 103.................................... Surveillance camera 121..................................................................... Monitoring center 301........................................................................... Behavior identification unit 302........................................................... Notification condition determination unit 303........................................................... Notification processing unit 400........................................................... Convolutional neural network 401........................................................... Convolutional layer 402........................................................... Output layer 411, 500, 5 10, 520...Monitoring images 501, 502, 511, 521...Bounding boxes 6, 14, 18.................................................... "Notification condition setting" screen 600........................................................... "Behavior class" tab 700........................................................... "Behavior duration" tab 900........................................................... "Crowdness" tab 1000........................................................... "Time period" tab 1100........................................................... "Stay in specific area" tab 1200........................................................... Notification condition table

Claims (16)

An identification means for identifying which of predetermined behavior classes a behavior of a person included in an image corresponds to using a machine learning model;
a determination means for determining whether or not both a first notification condition, which specifies for each behavior class whether or not a notification of the detection of a behavior corresponding to the behavior class is necessary, and a second notification condition, which is based on a reason other than the behavior class, are satisfied;
a notification means for notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative,
The second notification condition includes at least one of a degree of congestion around the person, a positional relationship between the position of the person in the image and a specified range, an attribute of the person, a behavior class of people around the person, and an attribute of people around the person.
A behavior detection system characterized by: The behavior detection system described in claim 1, characterized in that, when the image contains multiple people, the identification means identifies, for each person, to which behavior class the behavior belongs. The behavior detection system according to claim 1 or 2, wherein the machine learning model is a neural network. 4. The behavior detection system of claim 3, wherein the neural network is a convolutional neural network. 5. The behavior detection system according to claim 1, wherein the first notification condition includes a specification that notification is required when the behavior of the person corresponds to a combination of a plurality of behavior classes. 5. The behavior detection system according to claim 1, wherein the second notification condition is specified for each of the behavior classes. The behavior detection system described in claim 5, characterized in that the second notification condition is specified corresponding to a combination of the multiple behavior classes. 7. The behavior detection system according to claim 6, wherein the second notification condition is a combination of a plurality of specifications for each of the behavior classes. 9. The behavior detection system according to claim 8, wherein the second notification condition is a combination of a plurality of specified conditions corresponding to the combination of the plurality of behavior classes. The behavior detection system according to any one of claims 1 to 9, characterized in that the behavior classes include at least one of hitting, kicking, pushing, climbing, crawling, falling, lying down, throwing, running, riding, walking, operating a smartphone, sitting, grasping, and talking. The behavior detection system according to claim 1 , wherein the surroundings of the person are within a predetermined range that includes the person in the image. The behavior detection system according to claim 1 or 11 , wherein the people around the person are people who belong to a cluster including the person among a plurality of people included in the image. 13. The behavior detection system according to claim 1, wherein the notification includes information indicating to which behavior class the behavior corresponds. The behavior detection system according to any one of claims 1 to 13 , characterized in that the notification includes information indicating the importance of the notification set for each specification of whether or not notification is required in the second notification condition. An identification means for identifying, using a machine learning model, to which of a plurality of predetermined behavior classes the behavior of each of a plurality of people included in an image corresponds ;
a determination means for determining whether or not both a first notification condition, which specifies for each behavior class whether or not notification of detection of behavior corresponding to the behavior class for one of the plurality of persons is required, and a second notification condition according to a combination of the behavior class for the one person and the behavior classes for the other persons of the plurality of persons, are satisfied;
and a notification means for notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative.
A behavior detection system characterized by: an identifying step of identifying, using a machine learning model, which of a predetermined behavior classes the behavior of a person included in an image corresponds to;
a determination step of determining whether or not both a first notification condition that specifies, for each behavior class, whether or not a notification of the detection of a behavior corresponding to the behavior class is necessary, and a second notification condition based on a reason other than the behavior class are satisfied;
a notification step of notifying a predetermined notification destination of the detection of the behavior when the determination result is affirmative,
The second notification condition includes at least one of a degree of congestion around the person, a positional relationship between the position of the person in the image and a specified range, an attribute of the person, a behavior class of people around the person, and an attribute of people around the person.
A behavior detection program characterized by: