JP7539779B2 - Anomaly Detection System - Google Patents

Description

The present invention relates to technology for detecting abnormalities in passersby.

For example, if a facility manager could notice that the behavior or state of people passing through the facility was abnormal, it would be desirable to be able to respond to the abnormality promptly.

For example, if a passerby repeatedly travels back and forth to the same place within the facility, that passerby may be lost. In such a case, it would be beneficial for the passerby if the administrator could go to that passerby and provide directions. Also, if a passerby stays in a blind spot where a restroom is located for a long period of time, that passerby may become unwell inside the restroom and be unable to move. In such a case, if the administrator visits the restroom to check on the passerby's safety, the passerby can receive care before he or she falls ill and falls into a critical condition.

For example, Patent Document 1 proposes an anomaly determination device that determines whether a person's behavior is abnormal based on the movement path of a person detected from images captured by multiple cameras placed discretely within a monitored area and the time the person spends in blind spots on that movement path.

JP 2019-109724 A

A passerby may trip over something and fall, or suddenly feel unwell and crouch down. Some of these abnormal conditions require a response as quickly as possible. Therefore, it would be desirable to have a mechanism that can quickly detect abnormal conditions in passersby, such as a sudden fall or crouching down.

For example, the abnormality determination device proposed in Patent Document 1 detects abnormal behavior of a passerby based on the path of the passerby and the time the passerby stays in blind spots on that path, so it takes time to detect the abnormal behavior. Therefore, the abnormality determination device proposed in Patent Document 1 cannot quickly detect sudden abnormal behavior or states, such as a passerby falling over or suddenly crouching down.

In view of the above, the present invention provides a means for quickly detecting abnormal behavior or conditions that suddenly occur in passersby.

The present invention proposes, as a first aspect, an anomaly detection system that determines whether or not a passerby 's posture is abnormal based on the ratio of the height from the floor of a specific part of the passerby 's upper body as indicated by the skeletal coordinates detected from a single image captured by a camera, to the height of the passerby as indicated by the skeletal coordinates.

According to the anomaly detection system according to the first aspect, if a person with an abnormal posture is present within the imaging area of the camera in one frame of an image, the person can be detected.
Furthermore, the anomaly detection system according to the first aspect can detect, for example, a person who has fallen on the floor with their head touching the floor.
Furthermore, according to the abnormality detection system according to the first aspect, even if the height of a particular part of a person, such as the head, changes depending on the person's height, a person with abnormal posture can be correctly detected.

In addition, the present invention proposes, as a second aspect, an anomaly detection system which determines whether or not the posture of a passerby is abnormal based on the ratio between the height from the floor of a specific part of the passerby 's upper body indicated by the skeletal coordinates detected from a single image captured by a camera, and the height from the floor of a specific part of the passerby's lower body indicated by the skeletal coordinates.

According to the anomaly detection system of the second aspect, if a person with an abnormal posture is present within the imaging area of the camera in one frame of an image, the person can be detected.
Furthermore, the abnormality detection system according to the second aspect can detect, for example, a person who has fallen on the floor with their head touching the floor.
Furthermore, the anomaly detection system according to the second aspect can detect, for example, a person crouching on the floor with their head near their knees.

In the anomaly detection system according to the first or second aspect, a configuration may be adopted as a third aspect in which it is determined whether or not the posture of the passerby is abnormal based on the position of coordinates obtained by transforming the skeletal coordinates into global coordinates.

In the anomaly detection system according to the first or second aspect, a configuration in which the camera generates a distance image and identifies the skeletal coordinates of the passerby based on the distance indicated by the distance image may be adopted as a fourth aspect.

According to the anomaly detection system of the third or fourth aspect, the position of a person's skeletal structure can be correctly identified regardless of the installation position of the camera, so that people with abnormal posture can be detected with high accuracy.

In the anomaly detection system according to the second aspect, a configuration may be adopted as a fifth aspect in which, when the attribute of the passerby recognized from the single image indicates that the passerby is a wheelchair user , the ratio of the comparison standard between the height of a specific part of the passerby 's upper body from the floor and the height of a specific part of the passerby 's lower body from the floor is changed to determine whether the posture of the passerby is abnormal or not.

According to the anomaly detection system of the fifth aspect, for example, if an elderly person with a cane is leaning forward, this is not judged to be an abnormal posture, whereas if an elementary school student carrying a school bag is leaning forward, this is judged to be an abnormal posture, and people with abnormal postures are detected with high accuracy.

In the anomaly detection system according to the first or second aspect, when multiple passersby are recognized from the single image, a configuration may be adopted as a sixth aspect in which the skeletal coordinates of the multiple passersby are detected in an order determined based on the attributes of the multiple passersby recognized from the image.

According to the anomaly detection system of the sixth aspect, by determining whether or not there is an abnormal posture in, for example, elderly people or visually impaired people using a white cane, compared to other people, abnormal postures of people who require more immediate attention as possible if they become abnormally ill can be detected more quickly than other people.

In the anomaly detection system according to any of the first to sixth aspects, a seventh aspect may be adopted in which, for each of a plurality of images taken sequentially at different times by a camera, it is determined whether or not the posture of a passerby is abnormal based on the position of the skeletal coordinates of the passerby detected from the image, and if the frequency with which the posture of the passerby has been determined to be abnormal in the past satisfies a predetermined condition, a notification of the abnormality is sent.

According to the anomaly detection system of the seventh aspect, for example, if a person stumbles and loses balance for an instant, but does not suffer injury, quickly regains normal posture, and resumes walking, the person will not be mistakenly determined to be in an abnormal posture.

FIG. 1 is a diagram illustrating an overall configuration of an anomaly detection system according to an embodiment. FIG. 2 is a diagram illustrating an example of an image captured by a camera according to an embodiment. FIG. 2 is a diagram showing a functional configuration of a server device according to an embodiment. 1 is a diagram illustrating two-dimensional coordinates of a predetermined part of a person's skeleton identified by a skeleton coordinate identification unit according to an embodiment. FIG. 4 is a diagram illustrating an example of a data configuration of a human pose table according to an embodiment. 11 is a flowchart of a process performed by a server device according to an embodiment to determine whether a posture of a person appearing in an image is abnormal. FIG. 13 is a diagram illustrating a notification screen displayed by a terminal device according to an embodiment.

[Embodiment]
An anomaly detection system 1 according to an embodiment of the present invention will be described below. Fig. 1 is a diagram showing a schematic diagram of the overall configuration of the anomaly detection system 1. The anomaly detection system 1 is a system that, with respect to each of a plurality of monitored areas, quickly detects a person who is in an abnormal posture within the monitored area. A space S shown in Fig. 1 is one of the multiple monitored areas of the anomaly detection system 1.

The anomaly detection system 1 comprises a server device 11, a camera 12, and a terminal device 13. Note that while FIG. 1 shows only one camera 12 placed in one of the multiple monitored areas, the anomaly detection system 1 comprises multiple cameras 12 placed in each of the multiple monitored areas. Also, while FIG. 1 shows only one terminal device 13, the number of terminal devices 13 corresponds to the number of destinations for notifications sent when the anomaly detection system 1 detects a person in an abnormal posture.

The server device 11 is a device that identifies the posture of a person in a monitored area from an image captured by the camera 12, judges whether the posture is abnormal or not, and if it is abnormal, notifies the terminal device 13 to warn the person. For this reason, the server device 11 is connected to each of the camera 12 and the terminal device 13 so that data communication is possible.

The camera 12 photographs the area to be monitored and transmits the images generated by the photographing to the server device 11. The terminal device 13 is used by the manager who manages the area to be monitored (for example, if the area to be monitored is within a station, then an employee of the railway company in charge of managing the station), receives notifications transmitted from the server device 11, and displays the received notifications on the screen and sounds an alarm to alert the manager.

The hardware of the server device 11 is a typical computer for server devices equipped with a processor, memory, and a communication interface, and the processor performs data processing according to the program stored in the memory, thereby operating as a device equipped with the above-mentioned functions in the anomaly detection system 1.

Camera 12 is a general network camera capable of transmitting images to an external device. Camera 12 captures a predetermined number of images per second (e.g., 10 images per second) and transmits these images sequentially to server device 11. Figure 2 is a schematic diagram showing an example of an image captured by camera 12.

The hardware of the terminal device 13 is a general computer for terminal devices equipped with a processor, memory, a communication interface, a display, a touch panel, and other operation reception devices, and the processor processes data according to the programs stored in the memory, thereby operating as a device equipped with the above-mentioned functions in the anomaly detection system 1. Examples of this include a monitor terminal installed in a facility management room and a mobile terminal carried by a facility staff member.

In the monitored area exemplified by space S in FIG. 1, six or more markers M are placed at positions within the angle of view of camera 12. The markers M are placed at two points on the XY plane, two points on the YZ plane, and two points on the ZX plane shown in FIG. 1. Server device 11 stores coordinates indicating the three-dimensional position of camera 12 in space S, and internal parameters that associate the two-dimensional coordinates (u, v) on the image of marker M captured by camera 12 with the position in space S (three-dimensional coordinates X, Y, Z). Furthermore, the two-dimensional coordinates use a coordinate system corresponding to the image elements of the image captured by camera 12.

The server device 11 uses internal parameters to convert two-dimensional coordinates in the image captured by the camera 12 into three-dimensional coordinates in the space S using a known coordinate conversion method (referred to as "global coordinate conversion" in this application).

Figure 3 is a diagram showing the functional configuration of server device 11. The functional configuration of server device 11 will be described below. The control unit 110 controls the operation of the other components of server device 11 and performs various data processing. The memory unit 111 stores various data. The image acquisition unit 112 acquires images from camera 12. The images acquired by the image acquisition unit 112 are processed by the control unit 110 and stored in the memory unit 111.

The clock unit 113 continuously measures the current time. When the image acquisition unit 112 acquires an image from the camera 12, it associates the image with the current time measured by the clock unit 113 and stores it in the memory unit 111.

The skeletal coordinate identification unit 114 identifies the two-dimensional coordinates of a specific part of the skeleton of a person appearing in an image stored in the storage unit 111 using a known method, and performs global coordinate conversion on the identified two-dimensional coordinates to identify the three-dimensional coordinates of the specific part of the skeleton of the person in the space S. The three-dimensional coordinates identified by the skeletal coordinate identification unit 114 are stored in the storage unit 111. The method used by the skeletal coordinate identification unit 114 to identify the two-dimensional coordinates of a specific part of the skeleton of a person from an image is a method using deep learning called OpenPose. However, the skeletal coordinate identification unit 114 may use other methods such as VisonPose (registered trademark, Next System Co., Ltd.) instead of OpenPose.

Figure 4 is a diagram illustrating the two-dimensional coordinates of specific parts of a person's skeleton identified by the skeleton coordinate identification unit 114. Figure 4(a) is still image information of the subject of skeleton detection, and Figure 4(b) shows the subject's skeleton coordinates detected by skeleton detection. Figure 4(c) shows the body parts corresponding to each position of the skeleton coordinates shown in Figure 4(b).

The attribute identification unit 115 (FIG. 3) identifies the attributes of people in images stored in the memory unit 111 using a known image recognition method. Specific examples of the attributes of people identified by the attribute identification unit 115 include "child," "elderly," "visually impaired," and "wheelchair user." For example, if the attribute identification unit 115 detects a school bag from a captured image by object detection, it identifies the attribute of the person carrying the school bag as "child." If the attribute identification unit 115 detects a pushcart, it identifies the attribute of the person who owns the pushcart as "elderly." If the attribute identification unit 115 detects a white cane, it identifies the attribute of the person carrying the white cane as "visually impaired." If the attribute identification unit 115 detects a wheelchair, it identifies the attribute of the person who owns the wheelchair as "wheelchair user." The attributes identified by the attribute identification unit 115 are stored in the memory unit 111.

The abnormal posture determination unit 116 determines whether or not the posture of a person is abnormal, based on the three-dimensional coordinates in the space S of a predetermined part of the person's skeleton identified by the skeletal coordinate identification unit 114. In this embodiment, the abnormal posture determination unit 116 determines that the posture of each person captured in the image is abnormal when either of the following conditions (A) and (B) is satisfied.
Condition (A): The height of the neck is less than one-tenth of the person's height.
Condition (B): The height of the neck is less than twice the height of the person's knees.

Note that condition (A) is a condition for detecting a state in which a person has fallen on the floor. Condition (B) is a condition for detecting a state in which a person is crouching.

The height of the neck and knees is indicated by the three-dimensional coordinates of those parts identified by the skeletal coordinate identification unit 114. If the coordinate information of the neck is missing, the coordinate information of the parts adjacent to the neck, such as the face, ears, and nose, is used. If the coordinate information of the knees is missing, the coordinate information of the adjacent parts, such as the feet and waist, is used. Height is calculated as the difference between the floor of space S and the coordinate of the top of the head. Alternatively, it is calculated by multiplying the difference between the floor of space S and the coordinate of the neck by a multiplying factor (e.g., 1.1) for adding the length from the neck to the top of the head.

The memory unit 111 stores a person posture table, which is a table for storing attributes identified by the attribute identification unit 115 and the results of the determination by the abnormal posture determination unit 116. FIG. 5 is a diagram illustrating an example of the data configuration of the person posture table. The person posture table is a collection of data records corresponding to each person who appeared in any of the most recent 10 images. Each data record in the person posture table stores a person ID that identifies a person, an attribute identified for that person by the attribute identification unit 115, and the result of the determination made by the abnormal posture determination unit 116 for that person.

The judgment results stored in the person posture table are the judgment results for each of the most recent 10 images. In FIG. 5, the "First Image" column stores the judgment result made by the abnormal posture judgment unit 116 based on the three-dimensional coordinates identified by the skeletal coordinate identification unit 114 from the image last captured by the camera 12. The "Second Image" column stores the judgment result made by the abnormal posture judgment unit 116 based on the three-dimensional coordinates identified by the skeletal coordinate identification unit 114 from the second-to-last image captured by the camera 12. That is, the "ith Image" column stores the judgment result made by the abnormal posture judgment unit 116 based on the three-dimensional coordinates identified by the skeletal coordinate identification unit 114 from the i-th image captured from the last image captured by the camera 12.

The notification necessity determination unit 117 (Fig. 3) determines that an abnormality should be notified if the frequency of past detections of a person with an abnormal posture meets a predetermined condition based on the data stored in the person posture table. Specifically, the notification necessity determination unit 117 determines that an abnormality should be notified if a predetermined number or more (e.g., seven or more) of the ten judgment results stored in the "judgment result" column for any data record in the person posture table are "abnormal". Alternatively, it determines that an abnormality should be notified if a predetermined number of consecutive judgment results (e.g., three consecutive) stored in the "judgment result" column are "abnormal".

When the notification necessity determination unit 117 determines that an abnormality should be notified, the notification unit 118 (Figure 3) sends a notification to the terminal device 13 to alert the user.

Figure 6 is a flow diagram of the process performed by the server device 11 to determine whether the posture of a person in an image is abnormal. Each time the server device 11 receives a new image from the camera 12, it performs the process according to the flow diagram in Figure 6 using that image.

When the image acquisition unit 112 acquires a new image from the camera 12 and stores the image in the memory unit 111, the skeletal coordinate identification unit 114 identifies two-dimensional skeletal coordinates (two-dimensional coordinates in the image) for each person appearing in the image (step S01). The two-dimensional skeletal coordinates identified in step S01 are temporarily stored in the memory unit 111.

In parallel with the processing of step S01, the attribute identification unit 115 recognizes people from new images acquired by the image acquisition unit 112 from the camera 12, and identifies attributes for each of the recognized people (step S02). The attributes identified in step S02 are stored in the person posture table.

The abnormal posture determination unit 116 then assigns the initial value "1" to the counter j. In addition, the abnormal posture determination unit 116 assigns, for example, the number of people recognized by the attribute identification unit 115 to the variable n. (Step S03).

Next, the abnormal posture determination unit 116 determines the priority order for performing the following process among the n people appearing in the image based on the attributes identified in step S02 (step S04). The attribute identification unit 115 determines the priority order, for example, in the following order: "visually impaired," "wheelchair user," "elderly," "child," and "other."

Next, the abnormal posture determination unit 116 selects one of the undetermined persons as a processing target, in order of the priority determined in step S04 (step S05). Next, the skeletal coordinate identification unit 114 converts the two-dimensional skeletal coordinates (two-dimensional coordinates in the image) of the person selected in step S05 into three-dimensional skeletal coordinates (three-dimensional coordinates in space S) (step S06).

Next, the abnormal posture determination unit 116 determines whether the processing target person satisfies the following condition (A) based on the three-dimensional skeletal coordinates of the processing target person (step S07).
Condition (A): The height of the neck is less than one-tenth of the person's height.

If the processing target person satisfies the condition (A) (step S07; Yes), the abnormal posture determination unit 116 determines that the processing target person has an abnormal posture (step S08). On the other hand, if the processing target person does not satisfy the condition (A) (step S07; No), the abnormal posture determination unit 116 determines whether the processing target person satisfies the following condition (B) from the processing target person's three-dimensional skeletal coordinates (step S09).
Condition (B): The height of the neck is less than twice the height of the person's knees.

If the processing target person satisfies condition (B) (step S09; Yes), the abnormal posture determination unit 116 determines that the processing target person's posture is abnormal (step S08). On the other hand, if the processing target person does not satisfy condition (B) (step S09; No), the abnormal posture determination unit 116 determines that the processing target person's posture is normal (step S10). The determination result in step S10 is stored in the person posture table.

The abnormal posture determination unit 116 then determines whether the current counter j matches the variable n indicating the number of people, i.e., whether processing has been completed for all people recognized from the image (step S11). If counter j does not match the variable n (step S11; No), the server device 11 adds "1" to counter j (step S12) and performs processing from step S05 onwards for unprocessed people. If counter j matches the variable n (step S11; Yes), the server device 11 ends the series of processes.

The notification necessity determination unit 117 determines whether or not a notification is required each time data in the "Determination result" column of the person posture table is updated by processing according to the flow in FIG. 6. Specifically, the notification necessity determination unit 117 determines that a notification is required when seven or more "abnormalities" are stored in the "Determination result" column of any data record in the person posture table.

If the notification necessity determination unit 117 determines that a notification is necessary, the notification unit 118 sends a notification to the terminal device 13 that includes the most recent image at that time, a frame surrounding the person with seven or more "abnormalities" stored, and data indicating the installation location of the camera 12 that captured the image.

When the terminal device 13 receives a notification from the notification unit 118 of the server device 11, it uses the image included in the notification to display a notification screen as shown in FIG. 7, and also emits an alarm to alert the administrator, who is the user of the terminal device 13.

In response to the warning sound emitted from the terminal device 13, the administrator looks at the image included in the notification screen displayed on the display of the terminal device 13, determines whether any person is in a state where they need help, and if it is determined that they are in a state where they need help, takes the necessary action immediately.

According to the above-described anomaly detection system 1, for example, if the camera 12 captures 10 images per second, when any person in the monitored area assumes an abnormal posture, the administrator is notified of this within one second. Therefore, if a passerby in the monitored area falls or suddenly crouches down, the administrator is immediately informed of the abnormality and can quickly take the necessary measures.

[Modification]
The above-described embodiment may be modified in various ways within the scope of the technical concept of the present invention. The modifications are shown below. Two or more of the modifications shown below may be combined as appropriate.

(1) The conditions (A) and (B) shown in the above-described embodiment are examples of conditions for determining whether the posture of a person in an image is abnormal, and various other conditions may be used instead of or in addition to conditions (A) and (B).

The method of calculating the subject's height is not limited to the calculation method described in the above embodiment. For example, the height may be calculated by multiplying the sum of the length from the neck to the waist, the length from the waist to the knee, and the length from the knee to the ankle, which are indicated by the three-dimensional coordinates identified by the skeletal coordinate identification unit 114, by a multiplying factor (e.g., 1.1) for adding the length from the neck to the top of the head.

(2) In the above-described embodiment, the two-dimensional skeletal coordinates determined from the image captured by the camera 12, which is an optical camera, are converted into three-dimensional skeletal coordinates by global coordinate transformation. Alternatively, a camera that generates a distance image (distance image sensor) may be used as the camera 12, and the three-dimensional skeletal coordinates of the person appearing in the distance image may be determined based on the distance indicated by the distance image generated by the camera 12.

(3) In the above-described embodiment, a determination is made as to whether a person's posture is abnormal based on the three-dimensional skeletal coordinates of specific parts, such as the neck and knees. The three-dimensional skeletal coordinates of a part to be used in determining whether a person's posture is abnormal may be changed in various ways.

(4) Whether or not a person's posture is abnormal may be determined based on the attributes of the person identified from the image. For example, in a normal state, a wheelchair user often has a neck height that is lower than twice the height of the knees. Therefore, for wheelchair users, instead of condition (B), a configuration may be adopted in which whether or not the posture is abnormal is determined using, for example, the following condition (C).
Condition (C): The height of the neck is 1.5 times or less than the height of the person's knees.

(5) In the above-described embodiment, a determination is made as to whether a person's posture is abnormal based on the ratio of the height of the neck or knees to the height. However, for example, a determination may be made as to whether a person's posture is abnormal based on the absolute value of the height of the neck or knees. For example, a configuration may be adopted in which, regardless of the height of a person, if the height of the neck from the floor is 20 cm or less, the posture of the person is determined to be abnormal.

(6) In the above-described embodiment, the marker M is installed in the space S, but singular points such as facility equipment that has been installed in advance in the space S can also be used as the marker M. For example, fixed signs, guide boards, and floor patterns are examples of this. Also, even if there is no specific object, it is possible to link the angle of view of the camera 12 with survey data within the space S and use it as the marker M.

(7) In the above-described embodiment, the two-dimensional skeletal coordinates determined from the image captured by the optical camera 12 are converted into three-dimensional skeletal coordinates by global coordinate transformation. By using a stereo camera as the camera 12, it is also possible to obtain the three-dimensional coordinates of the subject without the marker M.

(8) In the above-described embodiment, the server device 11 is a separate device from the camera 12 and the terminal device 13, but the server device 11 may be a single device integrated with the camera 12 or the terminal device 13. For example, if the camera 12 has a built-in computer, the computer built into the camera 12 may play the role of the server device 11. Also, if the terminal device 13 is a monitor terminal installed in a facility management room, the terminal device 13 may also play the role of the server device 11.

1... anomaly detection system, 11... server device, 12... camera, 13... terminal device, 110... control unit, 111... storage unit, 112... image acquisition unit, 113... timing unit, 114... skeletal coordinate identification unit, 115... attribute identification unit, 116... abnormal posture determination unit, 117... notification necessity determination unit, 118... notification unit.

Claims (7)

An anomaly detection system that determines whether a passerby 's posture is abnormal based on the ratio of the height from the floor of a specific part of the passerby 's upper body as indicated by the skeletal coordinates of the passerby detected from a single image captured by a camera to the height of the passerby as indicated by the skeletal coordinates. An anomaly detection system that determines whether a passerby 's posture is abnormal based on the ratio between the height from the floor of a specific part of the passerby 's upper body indicated by the skeletal coordinates detected from a single image captured by a camera, and the height from the floor of a specific part of the passerby 's lower body indicated by the skeletal coordinates . The anomaly detection system according to claim 1 or 2, further comprising: determining whether or not a posture of the passerby is abnormal based on a coordinate position obtained by performing global coordinate transformation on the skeleton coordinates. The anomaly detection system according to claim 1 or 2, wherein the camera generates a distance image, and identifies skeletal coordinates of the passerby based on a distance indicated by the distance image. 3. The anomaly detection system according to claim 2, wherein, when the attribute of the passerby recognized from the single image indicates that the passerby is a wheelchair user, the ratio of the comparison standard between the height of a specific part of the passerby 's upper body from the floor and the height of a specific part of the passerby 's lower body from the floor is changed to determine whether the posture of the passerby is abnormal. 3. The anomaly detection system according to claim 1, wherein, when a plurality of passersby are recognized from the single image, skeletal coordinates of the plurality of passersby are detected in an order determined based on attributes of the plurality of passersby recognized from the image. 7. The anomaly detection system according to claim 1, further comprising: a step of determining whether or not a posture of a passerby is abnormal based on the skeletal coordinate position of the passerby detected from each of a plurality of images captured sequentially at different times by a camera; and notifying the user of the abnormality when the frequency with which the posture of the passerby has been determined to be abnormal in the past satisfies a predetermined condition.

Images