JP7459558B2 - Abnormal behavior detection method, abnormal behavior detection program, and abnormal behavior detection device - Google Patents

Description

The present invention relates to an abnormal behavior detection method, an abnormal behavior detection program, and an abnormal behavior detection device.

At police boxes where police officers are stationed, there have been incidents in which police officers have been attacked by suspicious individuals, resulting in death or injury. For example, there was an incident in which a suspicious person appeared from the back door of a police box and attacked a responding police officer with a knife. There have also been incidents in which suspicious individuals who visited a police box claiming to have found money and goods attacked responding police officers with knives and model guns. In order to deal with these incidents, consideration is being given to installing security cameras at police boxes and police stations.

There is technology that detects suspicious individuals from security camera footage and notifies administrators and potential victims of the detection information. Such detection technology includes technology that sets no-entry areas and detects people who enter these no-entry areas as suspicious. This technology can reduce false positives of suspicious individuals compared to when infrared cameras or other cameras are used to detect moving objects as suspicious.

The detection technology also includes a technology that registers face information in advance and detects a person whose face information does not match the registered face information as a suspicious person. According to this technology, it is possible to reduce false detection of a suspicious person rather than detecting a person who has entered a residence as a suspicious person.

JP2014-044542 International Publication No. 2015-068302 JP2015-138295

"Fujitsu Develops Actlyzer, an AI Technology that Recognizes Various Human Behaviors from Video," [online], November 25, 2019, Fujitsu Ltd., [Retrieved January 16, 2020], Internet (URL: https://pr.fujitsu.com/jp/news/2019/11/25.html)

The above detection technology includes a technique that sets a target area in an area where an unspecified number of people enter and exit, calculates the average value of physical quantities that represent the behavioral state of people present in the target area, and detects people who deviate from the average value as suspicious. The physical quantities are physical quantities obtained from the location of the visitor, such as a person's movement speed, movement direction, passing position, dwell time, stay time, and dwell position. This technology makes it possible to identify suspicious people without having people wear special devices.

However, conventional technology that identifies suspicious individuals based on physical quantities cannot identify them when the physical quantities obtained from the visitor's position do not deviate from the average value, for example when the suspicious individual is disguised as a harmless visitor.

One aspect of the present invention is to make it possible to identify a suspicious person, even when the suspicious person is pretending to be a harmless visitor.

One embodiment includes a motion information acquisition unit that acquires motion information representing a motion of a person included in a video based on skeletal information of the person included in a video captured in a predetermined area. The present embodiment also includes a feature derivation unit that derives a feature amount representing that a person included in the video performs abnormal behavior based on the acquired motion information, and and a notification unit that provides notification when it is suggested that a person will engage in abnormal behavior.

In one aspect, the present invention makes it possible to identify suspicious persons, even when the suspicious persons are acting under the guise of benign visitors.

FIG. 1 is a conceptual diagram illustrating an abnormal behavior detection system according to the present embodiment. 1 is a block diagram illustrating a functional configuration of an abnormal behavior detection device according to a first embodiment. FIG. 1 is a block diagram illustrating a hardware configuration of an abnormal behavior detection device according to the present embodiment. It is a conceptual diagram explaining the outline of abnormal behavior detection processing of a 1st embodiment. 2 is a flowchart illustrating the flow of abnormal behavior detection processing according to the first embodiment. 12 is a flowchart illustrating the flow of responder determination processing according to the present embodiment. FIG. 2 is a block diagram illustrating a functional configuration of an abnormal behavior detection device according to a second embodiment. FIG. 13 is a conceptual diagram illustrating an overview of abnormal behavior detection processing according to the second embodiment. 13 is a flowchart illustrating a flow of an abnormal behavior detection process according to the second embodiment.

[First embodiment]

(System configuration)
Fig. 1 illustrates an example of a hardware configuration of an abnormal behavior detection system 10 according to a first embodiment. The abnormal behavior detection system 10 includes a video capture device 11, an alarm device 12, and an abnormal behavior detection device 13, and is installed in, for example, a police box 14. The police box 14 is an example of a predetermined area where an unspecified number of visitors enter and exit. Fig. 1 illustrates a state in which a police officer 15, an example of a person stationed at the police box 14, and a visitor 16 are present. The abnormal behavior detection device 13 may be located at a location separated from the police box 14 via a network.

(Functional configuration)
FIG. 2 illustrates the functional configuration of the abnormal behavior detection device 13. The abnormal behavior detection device 13 includes a video capturing section 21, a skeletal information acquisition section 22, a motion information acquisition section 23, a police officer determination section 24, a responder determination section 25, a suspicious person degree derivation section 26, an abnormal behavior degree derivation section 27, and a notification section 28.

The video capture unit 21 uses the video capture device 11 to capture images inside the police box 14.

The skeletal information acquisition unit 22 uses existing technology to acquire skeletal information of people appearing in the video captured by the video capture unit 21 for each image frame that constitutes the video. Skeletal information is information that represents the three-dimensional position of each of the multiple joints that form the skeleton. The motion information acquisition unit 23 uses the time-series skeletal information acquired by the skeletal information acquisition unit 22 to acquire motion information of people appearing in the video captured by the video capture unit 21.

The motion information is information representing the state of each body part obtained from time-series skeletal information. For example, raising hands, bending legs, standing, running, etc. To obtain motion information, a rule-based method may be used in which rules are created in advance such as the distance or angle of movement of the skeleton, or the position, distance, or angle of the skeleton and variations thereof may be used as features. A learning-based method may be used in which the information is learned in advance. Further, for example, another behavior analysis technique may be used to acquire the behavior information. Another behavior analysis technique, for example, defines multiple types (for example, 100 types) of basic movements that are the elements that constitute a complex behavior, and uses the skeletal information of each image frame to calculate the basic movements for each image frame. This is a recognition technology that uses a learning model.

In addition, as useful additional information to the motion information, information on weapons, bags, or other personal belongings carried by people in the video may also be obtained.

The police officer determination unit 24 determines whether or not a person appearing in the video captured by the video capture unit 21 is a police officer. To determine whether a person is a police officer, a histogram-based method that utilizes statistics of color information and brightness information of human areas may be used, taking advantage of the fact that police officers wear uniforms of a specific color. To determine whether a person is a police officer, a template matching-based method that uses the video of the police officer as a reference image, or a learning-based method that learns characteristic information of police officers in advance may be used.

The responder determining unit 25 uses the determination result of the police officer determining unit 24 to determine the responder to whom the police officer shown in the video is responding. Information such as the distance between the police officer and the visitor, the direction of the body or face of the police officer and the visitor, and the line of sight may be used to determine who is responding. Furthermore, if the desk, chair, etc. to be used during reception are determined, information on the desk and chair may be used.

The suspiciousness degree derivation unit 26 derives the suspiciousness degree of the visitor when it is determined from the judgment result of the responder response judgment unit 25 that a visitor is captured in the video, that is, when a person other than a police officer is captured in the video. The abnormal behavior degree derivation unit 27 uses the suspiciousness degree derived by the suspiciousness degree derivation unit 26 to derive the abnormal behavior degree, which is the attack risk of a police officer being attacked by a suspicious person, and uses the attack risk to determine whether or not to issue a warning. The suspiciousness degree derivation unit 26 and the abnormal behavior degree derivation unit 27 are included in the feature derivation unit.

When the abnormal behavior degree deriving unit 27 determines to issue a warning, the notification unit 28 uses the notification device 12 to issue a warning.

(Hardware configuration diagram)

As an example, as shown in FIG. 3, the abnormal behavior detection device 13 includes a CPU (Central Processing Unit) 51, a primary storage unit 52, a secondary storage unit 53, and an external interface 54. The CPU 51 is an example of a processor, which is hardware. The CPU 51, the primary storage unit 52, the secondary storage unit 53, and the external interface 54 are connected to each other via a bus 59. The CPU 51 may be a single processor or multiple processors. In addition, for example, a GPU (Graphics Processing Unit) may be used instead of the CPU 51.

The primary storage unit 52 is, for example, a volatile memory such as a RAM (Random Access Memory). The secondary storage unit 53 is, for example, a non-volatile memory such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive).

The secondary storage unit 53 includes a program storage area 53A and a data storage area 53B. The program storage area 53A stores, for example, a program such as an abnormal behavior detection program. The data storage area 53B stores, for example, video information of a police box photographed by the video photographing device 11.

The CPU 51 reads out the abnormal behavior detection program from the program storage area 53A and loads it into the primary storage unit 52. By loading and executing the abnormal behavior detection program, the CPU 51 operates as the video capture unit 21, skeletal information acquisition unit 22, motion information acquisition unit 23, police officer determination unit 24, responder determination unit 25, suspicious person degree derivation unit 26, abnormal behavior degree derivation unit 27, and notification unit 28 in FIG. 2.

Note that programs such as the abnormal behavior detection program may be stored in an external server and expanded to the primary storage unit 52 via a network. Further, programs such as the abnormal behavior detection program may be stored in a non-temporary recording medium such as a DVD (Digital Versatile Disc), and expanded to the primary storage unit 52 via a recording medium reading device.

An external device is connected to the external interface 54, and the external interface 54 controls transmission and reception of various information between the external device and the CPU 51. In FIG. 4, an example is shown in which a camera 55A, which is an example of a video photographing device, and a speaker 55B, which is an example of a notification device, are connected to the external interface 54.

The camera 55A may be, for example, a digital camera, an infrared camera, or an RGB-D (Red Green Blue - Depth) camera. For example, a patrol lamp may be used instead of the speaker 55B.

In addition, either or both of the camera 55A and the speaker 55B may be built into the abnormal behavior detection device 13, or may be located at a location separated from the abnormal behavior detection device 13 via a network. The abnormal behavior detection device 13 may be, for example, a personal computer, a server, etc.

(Summary of abnormal behavior detection processing)
In the first embodiment, the attack risk, which is an example of the degree of abnormal behavior, is derived using the degree of suspicious person. The degree of suspicious person and the degree of abnormal behavior are examples of feature amounts. FIG. 4 illustrates an overview of abnormal behavior detection processing according to the first embodiment. The CPU 51 uses the camera 55A to photograph a video 101 inside the police box. In the example of FIG. 4, two people 101A and 101B are shown in the video 101. The two people 101A, 101B are police officers or visitors.

The CPU 51 detects people from the video 101 and obtains rectangular areas 102A and 102B that contain the people. To detect people, a template matching method may be used in which an image of the person to be detected is used as a reference image, or a learning-based method may be used in which characteristic information of people is learned in advance.

The CPU 51 uses the rectangular areas 102A and 102B to obtain human skeletal information 103A and 103B. To obtain skeletal information, you can use a method that finds the contour of the person and uses the center or center of gravity of the contour as the skeleton, or you can use images of body parts as reference images, estimate each body part, and then calculate the skeletal structure. A template matching based approach to estimation may be used. Further, to acquire the skeleton information, a learning-based method may be used in which characteristic information of the skeleton is learned in advance. The CPU 51 uses the skeleton information 103A and 103B to obtain human motion information.

The CPU 51 uses rectangular areas 102A and 102B to determine whether or not the person in the rectangular area is a police officer, and outputs rectangular area 104A that includes a visitor who is not a police officer. The CPU 51 uses the acquired person's movement information and visitor information to derive behavior information of the visitor, and derives a suspiciousness level based on the behavior information. The derivation of the suspiciousness level will be described later.

The CPU 51 derives the attack risk using the derived suspicious person degree. The derivation of attack risk will be discussed later. The CPU 51 determines whether or not to issue a warning based on the derived attack risk value, and if it is determined to issue a warning, the CPU 51 uses the speaker 55B, which is an example of a notification device, to issue a warning to the police officer. Announce a warning.

In addition, instead of using the speaker 55B to notify the warning, the warning may be notified using a mobile phone or mobile terminal held by the police officer, or the warning can be sent to the abnormal behavior detection device 13 via the network. A warning may be sent to the information terminal of the connected management facility. Further, in addition to or instead of notifying the warning, a warning log may be generated and stored in the abnormal behavior detection device 13 or the information terminal of the managed facility. The warning log can be used in conjunction with the video captured by the video camera 11, for example, for training police officers on how to deal with visitors and avoid attacks.

(Derivation of suspicious person level)
The suspiciousness level d _suspicious of a visitor is calculated using n types of behavioral information about the visitor obtained from the video of the police officer interacting with the visitor using the camera 55A installed to take pictures inside the police box 14. It is derived from equation (1). f _i () (i=1,...,n) is a function that receives as input the set B of behavior information calculated from the skeleton information and outputs behavior information i regarding the visitor. k _i is a weighting coefficient of behavior information i. Here, the behavioral information is information representing complex body changes obtained from a collection of multiple pieces of behavioral information. For example, if a person frequently changes between a raised hand state and a hand lowered state in a short period of time, it is considered as flailing the arm.

Functions that output n types of behavior information are illustrated in equations (2) to (8).

Formula (2) outputs 1 if the visitor is carrying a weapon, since there is a possibility that the visitor may injure a police officer, and outputs 0 if the visitor is not carrying a weapon. For example, using weapon information and human skeletal information obtained from video, if the skeletal position of the person's hand overlaps with the position of the weapon, it is determined that the visitor is holding a weapon, and if this state, which is motion information, continues, it is determined that the visitor is carrying a weapon and outputs 1.

Formula (3) outputs 1 if a visitor is violent, as there is a possibility that a police officer may be injured or property may be damaged, and outputs 0 if the visitor is not violent. For example, using skeletal information of a person obtained from a video, the shoulder angle is obtained from the skeletal positions of the elbow, shoulder, and waist, and if the shoulder angle is equal to or greater than a predetermined angle, the arm is considered to be raised, and if it is less than the predetermined angle, the arm is considered to be lowered. If the movement information, that is, the arm raised or arm lowered state, changes a predetermined number of times within a predetermined time, it is determined that the visitor is violent. For example, the predetermined angle may be 100 degrees, the predetermined time may be three seconds, and the predetermined number of times may be three.

Since there is a possibility that a visitor may attack a police officer immediately after a sudden stand-up, formula (4) outputs 1 when a police officer is nearby, i.e., when the distance between the police officer and the visitor is within a specified distance, if the visitor suddenly stands up. If a police officer is not nearby, the movement is one that requires caution, so 0.5 is output, and if the visitor does not suddenly stand up, 0 is output. The specified distance may be, for example, 1 m.

Using human skeletal information obtained from the video, the bending angle of the knee is obtained from the skeletal positions of the heel, knee, and hip, and the bending angle of the hip is obtained from the skeletal positions of the knee, hip, and shoulder. First, if the bending angle of the knee and the bending angle of the hip are each within a predetermined angle, the state is determined to be sitting. The predetermined bending angles of the knee and the hip may each be 80 degrees to 110 degrees. Next, if the bending angle of the knee and the bending angle of the hip are each within a predetermined angle, the state is determined to be standing. The predetermined bending angles of the knee and the hip may each be 170 degrees to 190 degrees. If there is a transition from a sitting state, which is motion information, to a standing state between video frames within a predetermined time, it is determined that the person has suddenly stood up. The predetermined time may be, for example, one second.

When a visitor puts his/her hand in the bag, it may be an action to take out a weapon or dangerous object, so equation (5) outputs 1 if a police officer is nearby and the visitor puts his/her hand in the bag, and outputs 0.5 even when no police officer is nearby, as this is a movement that requires caution. Equation (5) outputs 0 if the visitor does not put his/her hand in the bag. For example, using bag information obtained from video, if the skeletal position of the person's hand does not overlap with the position of the bag, it is determined that the hand is not in the bag, and if the skeletal position of the person's hand overlaps with the position of the bag, it is determined that the hand is in the bag. If the action information changes from a state in which the hand is not in the bag to a state in which the bag is in the hand, it is determined that the hand is in the bag.

Equation (6) shows that when a visitor's gaze wanders, it may be an action to check the police officer's condition and the location of the camera, etc., and when the visitor's gaze wanders when a police officer is nearby. , outputs 1 because it may be an action to confirm the position of a camera, etc. To let your gaze wander means to perform the action of looking around here and there. Equation (6) outputs 0.5 if the visitor's line of sight wanders when the police officer is not nearby, and outputs 0 if the visitor does not let his or her line of sight wander.

Equation (7) outputs 1 when a visitor peers into the back of a room when a police officer is nearby, as this may be an action to check for the presence of other police officers, and outputs 0.5 when a visitor peers into the back of a room when no police officer is nearby, as this may be an action to check for the presence of a police officer. Equation (7) outputs 0 if the visitor does not peer into the back of the room. For example, it is possible to determine whether a visitor is peering into the back of the room by using information on the area the visitor is gazing at, obtained from the video, and information on the direction of the visitor's face, which is motion information.

Equation (8) outputs 1 if a visitor looks at the camera when a police officer is nearby, since it is possible that the visitor is concerned about the attack being recorded when the visitor looks at the camera. If the police officer looks at the camera when there is no one nearby, outputs 0.5, and if the police officer does not look at the camera, outputs 0.

The weight k _i in equation (1) may be 1, for example. Alternatively, k ₁ and k ₂ may be set to 1, k ₃ may be set to 0.8, k ₄ may be set to 0.5, and k ₅ , k ₆ , and k ₇ may be set to 0.3.

(Derivation of attack risk)

The attack risk _drisk , which indicates the degree of risk of a police officer being attacked by a suspicious person, is derived using the visitor's suspicious degree _dsuspicious derived from equation (1), for example, using equation (9). do.

T _visitor is a threshold value for attack risk d _risk determination. T _visitor may be 1, for example.

Note that this embodiment is not limited to the derivation of the suspiciousness level and the attack risk and the exemplified values described above. In the above, a case is exemplified where the suspiciousness level increases as the possibility that the visitor is a suspicious person increases, but for example, a value may be set so that the suspiciousness level decreases as the possibility that the visitor is a suspicious person increases. In this case, for example, the attack risk d _risk may be derived using formula (10).

When the attack risk d _risk is 1, the CPU 51 determines that a warning will be notified, and when the attack risk d _risk is 0, the CPU 51 determines that a warning will not be reported.

(Abnormal behavior detection process)
5 illustrates an example of the flow of the abnormal behavior detection process. In step 201, the video capture unit 21 captures video of the inside of the police box 14 using the camera 55A. In step 202, the police officer determination unit 24 detects a human area from the captured video. The human area is, for example, a rectangular area that includes a person.

In step 203, the police officer determining unit 24 determines whether the number of people shown in the video is zero. If the determination in step 203 is affirmative, that is, if no person is shown in the video, the police officer determining unit 24 ends the abnormal behavior detection process. If the determination in step 203 is negative, that is, if a person is shown in the video, the police officer determining unit 24 determines in step 204 whether or not the person in the video is a police officer.

In step 205, the police officer determination unit 24 determines whether the number of visitors shown in the video is zero. The number of visitors can be obtained by subtracting the number of police officers from the number of people shown in the video. If the determination in step 205 is positive, i.e., if no visitors are shown in the video, the police officer determination unit 24 ends the abnormal behavior detection process.

If the determination in step 205 is negative, that is, if a visitor is shown in the video, then in step 206, the skeletal information acquisition unit 22 uses the video of the human area to identify all the people shown in the video. Obtain the skeletal information of. In step 207, the motion information acquisition unit 23 uses the skeletal information to acquire motion information of all the people shown in the video.

In step 208, the respondent determination unit 25 determines whether the number of police officers appearing in the video is zero. If the determination in step 208 is positive, i.e., if no police officers appear in the video, the respondent determination unit 25 proceeds to step 211, which will be described later. If the determination in step 208 is negative, i.e., if it is determined that a police officer appears in the video, the respondent determination unit 25 determines in step 209, as will be described later, whether a police officer is responding to a visitor.

In step 210, the suspiciousness degree derivation unit 26 judges whether or not the processing target person, which is one of the people shown in the video, is a visitor. If the judgment in step 210 is positive, i.e., if the processing target person is a visitor, the suspiciousness degree derivation unit 26 derives the suspiciousness degree of the visitor in step 211. In step 212, the abnormal behavior degree derivation unit 27 judges whether or not the suspiciousness degree is equal to or greater than a threshold value A. The threshold value A corresponds to the above-mentioned T _visitor .

If the determination in step 212 is negative, or if the determination in step 210 is negative, the abnormal behavior degree derivation unit 27 determines in step 213 whether or not all people appearing in the video have been processed. If the determination in step 212 is negative, it means that the suspiciousness level is less than threshold A, and if the determination in step 210 is negative, it means that the person to be processed is a police officer. If the determination in step 213 is positive, i.e., if all people appearing in the video have been processed, the abnormal behavior degree derivation unit 27 ends the abnormal behavior detection process. If the determination in step 213 is negative, i.e., if there are people that have not yet been processed, the abnormal behavior degree derivation unit 27 returns to step 210.

If the judgment in step 212 is positive, i.e., if the suspiciousness level is equal to or greater than threshold A, in step 214, the notification unit 28 uses the speaker 55B to issue a warning to the police officer, and ends the abnormal behavior detection process.

(Responder Determination Process)
6 illustrates an example of the flow of the respondent determination process of the police officer in step 209. In step 221, the respondent determination unit 25 selects one of the police officers appearing in the video. In step 222, the respondent determination unit 25 obtains the distance between the selected police officer and the visitor. In step 223, the respondent determination unit 25 selects the visitor who is the shortest distance from the selected police officer.

In step 224, the respondent determination unit 25 determines whether the distance is equal to or less than a threshold C. The threshold C may be, for example, 1 m. If the determination in step 224 is positive, that is, if the distance is equal to or less than the threshold C, the respondent determination unit 25 determines in step 225 that the visitor is a respondent being served by a police officer. In step 226, the respondent determination unit 25 determines whether processing of all police officers has been completed.

If the determination in step 224 is negative, i.e., if the distance between the police officer and the visitor is longer than threshold C, the responder determination unit 25 also proceeds to step 226. If the determination in step 226 is positive, i.e., if processing of all police officers has been completed, the responder determination unit 25 ends the responder determination process for police officers. If the determination in step 226 is negative, i.e., if there are unprocessed police officers, the responder determination unit 25 returns to step 221.

(Suspiciousness Degree Deriving Process)
In step 211, the suspiciousness degree derivation unit 26 derives the suspiciousness degree using formula (1), more specifically, formulas (2) to (8). All of formulas (2) to (8) may be used, or any of them may be selectively used, or the suspiciousness degree may be derived using behavioral information other than the behavioral information used in formulas (2) to (8). In formulas (4) to (8), if it is determined in step 209 that the visitor is a person who is being greeted by a police officer, it may be determined that a police officer is nearby, and if it is not determined that the visitor is a person who is being greeted by a police officer, it may be determined that a police officer is not nearby. Also, for example, formulas (4) to (8) may be used only when it is determined in step 209 that the visitor is a person who is being greeted by a police officer.

A suspicious person who attacks a police officer working at a police box visited by an unspecified number of visitors is likely to pose as a normal visitor and carry out the attack while the police officer is off guard. For this reason, it is difficult to distinguish between suspicious people and normal visitors, and it is difficult to appropriately notify police officers of the risk of an attack. In this embodiment, movement information that represents the movements of a person included in the video is obtained based on skeletal information of a person included in the video captured in a predetermined area. In this embodiment, a feature value that indicates that a person included in the video is behaving abnormally is also derived based on the obtained movement information, and an alert is issued if the derived feature value suggests that a person included in the video is behaving abnormally.

In this embodiment, a suspiciousness level indicating the degree to which a person in the video is suspicious is also derived, and an abnormal behavior level indicating whether the person in the video is behaving abnormally is derived as a feature based on the derived suspiciousness level.

In this embodiment, this makes it possible to identify suspicious individuals, even when they are pretending to be harmless visitors.

[Second embodiment]

In the second embodiment, the risk of attack is derived based on the degree of suspiciousness of a visitor, which is an example of a person other than a resident, as well as the degree of caution of a police officer, which is an example of a resident. Explanations of configurations and operations similar to those of the first embodiment will be omitted. The degree of suspiciousness, the degree of caution, and the risk of attack are examples of feature quantities.

(Functional configuration diagram)
FIG. 7 illustrates a functional configuration diagram of the second embodiment. The video shooting unit 31, the skeletal information acquisition unit 32, the motion information acquisition unit 33, the police officer determination unit 34, and the responder determination unit 35 in FIG. 7 are the same as the video shooting unit 21, the skeletal information acquisition unit 22, the motion information acquisition unit 23, the police officer determination unit 24, and the responder determination unit 25 in FIG. 2. The suspiciousness degree derivation unit 36, the abnormal behavior degree derivation unit 37, and the notification unit 38 are the same as the suspiciousness degree derivation unit 26, the abnormal behavior degree derivation unit 27, and the notification unit 28 in FIG. 2. The second embodiment differs from the first embodiment in that it includes the attention degree derivation unit 39 in FIG. 7. In the second embodiment, the suspiciousness degree derivation unit 36, the abnormal behavior degree derivation unit 37, and the attention degree derivation unit 39 correspond to the feature derivation unit.

When a police officer is shown in the video, the attention level deriving unit 39 uses the police officer's motion information, which is an example of the first motion information, obtained from the police officer's skeleton information, which is an example of the first skeleton information. Then, the degree of caution representing the degree of caution of the police officer is derived. The abnormal behavior degree deriving unit 37 calculates the suspicious person degree derived using the visitor's behavior information acquired from the visitor's skeletal information by the suspicious person degree deriving unit 36 and the caution derived by the caution degree deriving unit 39. The attack risk, which is an example of the degree of abnormal behavior, is derived using the degree. The visitor's skeletal information is an example of second skeletal information, and the visitor's behavior information is an example of second behavior information. The abnormal behavior degree deriving unit 37 uses the attack risk to determine whether or not to issue a warning.

(Outline of abnormal behavior detection process)
An overview of the abnormal behavior detection process of the second embodiment will be described with reference to Fig. 8. In Fig. 8, the CPU 51 uses rectangular areas 102A and 102B that contain a person to determine whether or not the person in the rectangular area is a police officer, and outputs a rectangular area 104B that contains the police officer. The CPU 51 uses the acquired motion information of the person and information of the police officer to derive behavior information of the police officer, and derives a caution level based on the behavior information. The derivation of the caution level will be described later. The second embodiment differs from the first embodiment in that the risk of attack is derived using the caution level in addition to the suspiciousness level.

(Deriving the level of attention)
The police officer's level of caution d _careful is derived by equation (11) using m types of behavioral information on the police officer obtained from video of the police officer interacting with a visitor captured by camera 55A installed to capture the inside of police box 14. g _j () is a function that takes as input a set B of motion information calculated from skeletal information and outputs behavioral information on the police officer. l _j is a weighting coefficient for behavioral information j, and may be, for example, 0.3.

An example of a function that outputs m types of behavior information is shown below.

Equation (12) outputs 1 when a police officer is in the safe area, and outputs 0 when a police officer is not in the safe area. A safe area is an area where a police officer can easily avoid being attacked by a suspicious person. For example, a place close to an entrance is a safe area, and a place close to a wall or furniture and difficult to avoid is not a safe area. For example, it is determined whether a police officer is in a safe area based on information about the safe area obtained from the video and information about the movement of the police officer. The safe area may be specified in advance in pixel units in the video, or a moving object shown in the video, such as a movable furniture such as a chair and the standing position of a visitor, may be recognized by existing image recognition processing, and may be changed according to the position of the moving object.

Equation (13) outputs a caution level of 1 when the distance between the police officer and the visitor is a predetermined length or more, and outputs a caution level of 0 when the distance is less than the predetermined length. The predetermined length may be, for example, 1 m. In addition, existing image recognition processing can recognize whether there are fixtures or counters between the police officer and the visitor in the video, and adjust the caution level value according to the recognition result. good.

Equation (14) outputs 1 if the police officer is looking at the visitor, and 0 if he is not. This is because if the police officer is looking at the visitor, he is more likely to be able to understand the suspicious person's movements and avoid an attack.

Equation (15) outputs 1 when the police officer is in an actionable posture, and outputs 0 when the police officer is not in an actionable posture. An actionable posture is a posture that allows the police officer to respond to a sudden action by a visitor. For example, a standing posture may be an actionable posture, and a sitting posture may not be an actionable posture. In addition, an actionable posture may be determined based on the work content or the indoor environment.

Equation (16) outputs 1 when the police officer is in an actionable state, and outputs 0 when the police officer is not in an actionable state. The actionable state is a state in which the visitor can respond to sudden actions. For example, a state in which the user is talking face-to-face with a visitor is an actionable state, and a state in which the user is creating a report without looking at the visitor is not an actionable state. Furthermore, the actionable state may be determined depending on the work content or the indoor environment.

(Deriving the risk of attack)

_{The risk d risk} of a police officer being attacked by a suspicious person is derived using equation (17) using the visitor's suspiciousness d _suspicious derived using equation (1) and the police officer's caution d _careful derived using equation (11).

T _visitor and T _police are thresholds for determining _{the risk of attack d risk} . T _visitor and T _police may be 1, for example.

Note that this embodiment is not limited to the derivation and exemplified values of the suspicious person degree, caution level, and attack risk described above. The above example shows a case where the lower the police officer's level of caution, the lower the level of caution. For example, the value may be set so that the lower the police officer's level of caution, the higher the level of caution. Good too. In this case, for example, the attack risk d _risk may be derived using equation (18).

Furthermore, if the value is set such that the lower the police officer's degree of caution, the higher the degree of caution, the attack risk may be derived using equation (19). h _s is a weighting coefficient for the suspicious person level d _suspicious , h _c is a weighting coefficient for the caution level d _careful , and T _sc is a threshold value for determining the attack risk d _risk .

(Flow of abnormal behavior detection process)
9 illustrates an example of a flow of abnormal behavior detection processing according to the second embodiment. Steps 251 to 264 in FIG. 9 are similar to steps 201 to 214 in FIG.

In the second embodiment, if the determination in step 260 is negative, that is, if the person to be processed is a police officer, the attentiveness deriving unit 39 derives the attentiveness of the police officer at step 265. The derivation of the degree of attention will be described later. In step 266, the abnormal behavior degree deriving unit 37 determines whether the police officer's attentiveness level is less than or equal to threshold B. The threshold B is T _police , which may be 1, for example. If the determination in step 266 is negative, that is, the degree of caution is greater than the threshold B, the abnormal behavior degree deriving unit 37 proceeds to step 263, and if the determination in step 266 is affirmative, that is, the degree of caution is greater than the threshold B. If it is below, the abnormal behavior degree deriving unit 37 proceeds to step 264.

(Attention Level Deriving Process)
In step 265, the attention level derivation unit 39 derives the attention level, for example, by using formula (11), and more specifically, formulas (12) to (16). All of formulas (12) to (16) may be used, any of them may be selectively used, or the attention level may be derived by using behavior other than the behavior information used in formulas (12) to (16).

In this embodiment, motion information representing the motion of a person included in the video is obtained based on skeletal information of the person included in the video captured in a predetermined area. In this embodiment, a feature indicating that the person included in the video is behaving abnormally is also derived based on the obtained motion information, and an alert is issued when the derived feature suggests that the person included in the video is behaving abnormally.

In this embodiment, the degree of suspicious person indicating the degree to which the person included in the video is suspicious is derived, and based on the derived degree of suspicious person, the person included in the video exhibits abnormal behavior as a feature value. Derive the degree of abnormal behavior that represents what to do.

In this embodiment, the skeletal information also includes first skeletal information of a person residing in a predetermined area and second skeletal information of a person other than the residing person, and the skeletal information includes the first skeletal information as motion information. First motion information indicating the motion of the corresponding person is acquired. In this embodiment, second motion information indicating the motion of the person corresponding to the second skeleton information is also acquired as the skeleton information. In this embodiment, the degree of caution representing the degree of caution of the person corresponding to the first skeletal information with respect to the person corresponding to the second skeletal information is derived based on the first motion information. In the present embodiment, the degree indicating the possibility that the person corresponding to the second skeleton information is a suspicious person is derived, and based on the derived suspicious person degree and the derived caution degree, as a feature quantity, A degree of abnormal behavior indicating the possibility that a person corresponding to the second skeleton information will perform abnormal behavior is derived.

In this embodiment, this makes it possible to more appropriately identify suspicious individuals, even when they are pretending to be harmless visitors.

In order to protect the safety of local residents, police officers working at police stations are trained to arrest criminals when a crime occurs. Therefore, police officers are generally able to respond appropriately to attacks by criminals. However, if a suspicious person poses as an ordinary visitor and attacks a police officer while he or she is off guard, it can be difficult even for police officers to respond to the sudden attack.

As illustrated in the first embodiment, when the visitor is highly suspicious, and as illustrated in the second embodiment, when the visitor is highly suspicious and the police officer is not very careful, there is a high risk of an attack by a suspicious person. That is, in this embodiment, in order to protect police officers from attacks by suspicious people, when the risk of an attack becomes high, the police officer is notified and warned to take appropriate action, such as by increasing the level of caution.

This allows police officers to prepare for attacks by suspicious individuals and respond in a way that does not give them an opportunity to attack. For example, they can keep an eye on visitors and be ready to act immediately.

In this embodiment, an example has been described in which the predetermined area is a police box and the person stationed there is a police officer, but the application is not limited to this. For example, examples of the predetermined areas include pachinko prize exchange places, lottery shops, convenience stores open late at night, and fast food. Further, an example of a stationed person is a store clerk. In this case as well, the person stationed at the store may be determined based on the uniform of the clerk, or the person stationed at the store may wear an IC (Integrated Circuit) tag in advance, and the person stationed at the store may be determined based on the IC tag. Further, in this embodiment, an example in which the abnormal behavior is an attack has been described, but the present invention is not limited to this. The abnormal behavior may be, for example, theft, property damage, etc.

The flowcharts in FIGS. 5, 6, and 9 are examples, and the flow of processing can be changed as appropriate.
(Additional note 1)
a motion information acquisition unit that acquires motion information representing the motion of the person included in the video based on skeletal information of the person included in the video captured in a predetermined area;
a feature amount deriving unit that derives a feature amount representing that a person included in the video performs abnormal behavior based on the behavior information acquired by the behavior information acquisition unit;
a notifying unit that notifies when the feature derived by the feature deriving unit suggests that a person included in the video engages in abnormal behavior;
Abnormal behavior detection device including.
(Additional note 2)
The feature amount deriving unit is
a suspicious person degree deriving unit that derives a suspicious person degree indicating the degree to which a person included in the video is a suspicious person;
an abnormal behavior degree deriving unit that derives, as the feature amount, an abnormal behavior degree representing that a person included in the video performs abnormal behavior based on the suspicious person degree derived by the suspicious person degree deriving unit;
The abnormal behavior detection device according to Supplementary Note 1, including:
(Additional note 3)
The skeletal information includes first skeletal information of a person residing in the predetermined area and second skeletal information of a person other than the residing person,
The motion information acquisition unit acquires, as the motion information, first motion information indicating a motion of the person corresponding to the first skeleton information and second motion information indicating the motion of the person corresponding to the second skeleton information. death,
The feature amount deriving unit is
an attentiveness deriving unit that derives an attentiveness level representing a degree of attentiveness of the person corresponding to the first skeletal information to the person corresponding to the second skeletal information based on the first motion information;
further including;
The suspicious person degree deriving unit derives a degree indicating the possibility that the person corresponding to the second skeleton information is a suspicious person,
The abnormal behavior degree deriving unit corresponds to the second skeleton information as the feature amount based on the suspicious person degree derived by the suspicious person degree deriving unit and the caution level derived by the caution level deriving unit. Deriving the degree of abnormal behavior that indicates the possibility that a person will engage in abnormal behavior.
Abnormal behavior detection device in Appendix 2.
(Additional note 4)
the predetermined area is inside a police box;
The stationed person is a police officer;
Abnormal behavior detection device in Appendix 3.
(Appendix 5)
The abnormal behavior detection device according to appendix 3 or 4, wherein the degree of abnormal behavior is a value representing a risk that a person other than the resident person will attack the resident person.
(Appendix 6)
Supplementary Note 3, wherein the notification unit notifies a warning using an alarm device installed in the area and an alarm device connected to the area via a network and located at a position distant from the area. ~An abnormal behavior detection device according to any of Appendix 5.
(Appendix 7)
Based on the second movement information, the visitor who is a person other than the resident person has a weapon, the visitor is acting violently, the visitor suddenly stands up, or the visitor puts his hand in his bag. Deriving information on at least one of the following actions: the visitor's gaze wandering, the visitor's looking into the back of the room, and the visitor's attention to the camera;
Abnormal behavior detection device according to any of Supplementary notes 3 to 6.
(Appendix 8)
Based on the first operation information, the resident person who is the resident person is in a safe area, and the distance between the resident person and the visitor who is a person other than the resident person is a predetermined length or more. is looking at the visitor, the resident is in a posture ready for action, and the resident is in a state ready for action.
Abnormal behavior detection device according to any of Supplementary notes 3 to 7.
(Appendix 9)
Based on skeletal information of a person included in a video taken of a predetermined area, obtain motion information representing the motion of the person included in the video;
Based on the acquired motion information, derive a feature amount indicating that the person included in the video performs abnormal behavior;
Notifying when the derived feature amount suggests that a person included in the video engages in abnormal behavior;
An abnormal behavior detection program that causes a computer to perform abnormal behavior detection processing.
(Appendix 10)
Deriving the feature amounts is as follows:
Deriving a suspicious person degree indicating the degree to which a person included in the video is a suspicious person,
Based on the derived suspicious person degree, derive an abnormal behavior degree representing that a person included in the video performs abnormal behavior as the feature amount;
The abnormal behavior detection program according to supplementary note 9, which includes:
(Appendix 11)
The skeletal information includes first skeletal information of a person residing in the predetermined area and second skeletal information of a person other than the residing person,
As the motion information, first motion information indicating the motion of the person corresponding to the first skeleton information and second motion information indicating the motion of the person corresponding to the second skeleton information are obtained;
Based on the first motion information, derive a degree of caution representing a degree of caution of the person corresponding to the first skeleton information with respect to the person corresponding to the second skeleton information,
Deriving a degree indicating the possibility that the person corresponding to the second skeleton information is a suspicious person,
Based on the derived degree of suspicious person and the derived degree of caution, derive a degree of abnormal behavior indicating the possibility that the person corresponding to the second skeleton information will behave abnormally as the feature quantity;
Abnormal behavior detection program in Appendix 10.
(Appendix 12)
the predetermined area is inside a police box;
The stationed person is a police officer;
Abnormal behavior detection program in Appendix 11.
(Appendix 13)
The abnormal behavior detection program according to appendix 11 or 12, wherein the degree of abnormal behavior is a value representing a risk that a person other than the resident person will attack the resident person.
(Appendix 14)
The computer is
Based on skeletal information of a person included in a video taken of a predetermined area, obtain motion information representing the motion of the person included in the video;
Based on the acquired motion information, derive a feature amount indicating that the person included in the video performs abnormal behavior;
Notifying when the derived feature amount suggests that a person included in the video engages in abnormal behavior;
Abnormal behavior detection method.
(Appendix 15)
Deriving the feature amounts is as follows:
Deriving a suspicious person degree indicating the degree to which a person included in the video is a suspicious person,
Based on the derived suspicious person degree, derive an abnormal behavior degree representing that a person included in the video performs abnormal behavior as the feature amount;
The abnormal behavior detection method according to supplementary note 14, which includes:
(Appendix 16)
The skeletal information includes first skeletal information of a person residing in the predetermined area and second skeletal information of a person other than the residing person,
As the motion information, first motion information indicating the motion of the person corresponding to the first skeleton information and second motion information indicating the motion of the person corresponding to the second skeleton information are obtained;
Based on the first motion information, derive a degree of caution representing a degree of caution of the person corresponding to the first skeleton information with respect to the person corresponding to the second skeleton information,
Deriving a degree indicating the possibility that the person corresponding to the second skeleton information is a suspicious person,
Based on the derived degree of suspicious person and the derived degree of caution, derive a degree of abnormal behavior indicating the possibility that the person corresponding to the second skeleton information will behave abnormally as the feature quantity;
Supplementary Note 15: Abnormal behavior detection method.
(Appendix 17)
the predetermined area is inside a police box;
The stationed person is a police officer;
Supplementary Note 16: Abnormal behavior detection method.
(Appendix 18)
The abnormal behavior detection method according to appendix 16 or 17, wherein the degree of abnormal behavior is a value representing a risk that a person other than the resident person will attack the resident person.

22, 32: skeleton information acquisition unit 23, 33: motion information acquisition unit 26, 36: suspicious person degree derivation unit 27, 37: abnormal behavior degree derivation unit 28, 38: notification unit 39: caution degree derivation unit

Claims (10)

a motion information acquisition unit that acquires first motion information representing a motion of a first person and second motion information representing a motion of a second person other than the first person ;
a suspiciousness degree derivation unit that derives a suspiciousness degree representing a degree to which the second person is a suspicious person based on the second action information;
an attention level deriving unit that derives an attention level representing a level of attention of the first person to the second person based on the first action information;
a risk derivation unit that derives a risk of the first person being attacked by the second person based on the suspiciousness degree derived by the suspiciousness degree derivation unit and the caution degree derived by the caution degree derivation unit;
An abnormal behavior detection device comprising: The motion information acquisition unit is
acquiring the first motion information representing a motion of the first person included in a video captured in a predetermined area based on skeletal information of the first person included in the video;
acquiring the second motion information representing a motion of the second person included in the video based on skeletal information of the second person included in the video;
The abnormal behavior detection device according to claim 1 . a notification unit that notifies when the risk derived by the risk derivation unit suggests that the first person will be attacked by the second person;
The abnormal behavior detection device according to claim 2 , further comprising: the predetermined area is inside a police box;
the first person is a person stationed in the predetermined area ;
The abnormal behavior detection device according to claim 2 . The stationed person is a police officer;
The abnormal behavior detection device according to claim 4. The abnormal behavior detection device of claim 3, wherein the alarm unit issues an alert using an alarm device installed within the area and an alarm device connected to the area via a network and located at a location separated from the area . Based on the second action information, at least one of the following behavior information is obtained: the visitor, who is the second person, has a weapon; the visitor is violent; the visitor suddenly stands up; the visitor puts his/her hand in his/her bag; the visitor's gaze wanders; the visitor peers into the back of the room; and the visitor is concerned about the camera.
The abnormal behavior detection device according to any one of claims 1 to 6. Based on the first operation information, the resident resident who is the first person is in a safe area, the distance from the visitor is a predetermined length or more, and the resident person is looking toward the visitor. seeking information on at least one of the following: the expatriate is present, the expatriate is in an actionable posture, and the expatriate is in an actionable state;
The abnormal behavior detection device according to claim 7 . obtaining first motion information representing a motion of a first person and second motion information representing a motion of a second person other than the first person ;
Deriving a suspicious person degree representing the degree to which the second person is a suspicious person based on the second behavior information,
Deriving a degree of attention representing the degree of attention of the first person to the second person based on the first motion information,
Deriving a risk of the first person being attacked by the second person based on the suspicious person degree and the caution level;
An abnormal behavior detection program that causes a computer to perform abnormal behavior detection processing. The computer is
First motion information representing a motion of a first person and second motion information representing a motion of a second person other than the first person are acquired.
Deriving a suspicious person degree representing the degree to which the second person is a suspicious person based on the second behavior information,
Deriving a degree of attention representing the degree of attention of the first person to the second person based on the first motion information,
Deriving a risk of the first person being attacked by the second person based on the suspicious person degree and the caution level;
Abnormal behavior detection method.