JP7822834B2 - Monitoring system, monitoring software, and monitoring method - Google Patents

Description

The present invention relates to a monitoring system, monitoring software, and monitoring method.

Patent Document 1 discloses a danger detection device that detects the posture and bedding height of a care recipient by analyzing images captured by a camera, and determines whether the care recipient is in a dangerous position based on these data, and issues an alert if the position is determined to be dangerous.

JP 2017-10383 A

Pose estimation technology, which detects people and their postures in captured images, can sometimes mistakenly identify non-human objects as people. Such false human detections can be reduced to some extent by removing two-dimensional people (or objects) based on 3D distance information from a TOF camera or similar, or by removing stationary objects using frame-to-frame differences. However, two-dimensional person removal cannot remove false detections of three-dimensional objects such as chairs, and frame-to-frame differences cannot remove cases where differences occur because something passes in front of a three-dimensional object such as a chair. In such cases, objects other than people may be mistakenly continued to be monitored.

Therefore, surveillance systems that detect people in captured images using pose estimation technology, etc., have room for improvement in terms of reducing the rate of false person detection.

The present invention was made in consideration of the above-mentioned current situation, and aims to provide a surveillance system, surveillance software, and surveillance method that can reduce the rate of false person detection.

In order to solve the above-mentioned problems and achieve the object, (1) a surveillance system according to a first aspect of the present disclosure is a surveillance system comprising a storage device, an imaging device that generates captured images, and a calculation device that acquires the captured images generated by the imaging device, wherein the calculation device further performs the following: when a candidate for an unmoving object is detected in the captured image, a determination process that determines whether the candidate for an unmoving object is a moving object or a stationary object; and, if the result of the determination process is a stationary object, a storage process that stores the candidate for an unmoving object in the storage device as one of the stationary objects; when a candidate for a moving object is detected in the captured image, a comparison process that compares the candidate for an unmoving object with one or more of the unmoving objects stored in the storage device; and, if the result of the comparison process shows that there is no matching stationary object, a monitoring process that determines that the candidate for an unmoving object is a moving object and monitors the moving object; and, if the result of the comparison process shows that there is a matching stationary object, the computing device does not monitor the candidate for an unmoving object.

(2) In the monitoring system described in (1) above, the computing device may further delete the oldest of the stored unmoving objects when the number of unmoving objects stored in the storage device reaches a predetermined value.

(3) In the monitoring system described in (1) or (2) above, the calculation device may further perform a second storage process to store the non-moving body candidate determined to be a moving body as a result of the determination process and the moving body candidate determined to be a moving body as a result of the matching process in the storage device, respectively, as one of the moving bodies. The determination process may also perform a second matching process to match the non-moving body candidate with one or more of the moving bodies stored in the storage device. If the second matching process does not result in a matching moving body, the non-moving body candidate may be determined to be a non-moving body. If the second matching process results in a matching moving body, the non-moving body candidate may be determined to be a moving body.

(4) In the monitoring system described in (3) above, the computing device may further delete the oldest stored moving object when the number of moving objects stored in the storage device reaches a predetermined value.

(5) In the monitoring system described in any one of (1) to (4) above, if the result of the determination process is an unmoving object, the computing device may not monitor the candidate unmoving object, and if the result of the determination process is a moving object, the computing device may determine that the candidate unmoving object is a moving object and monitor the moving object.

(6) In the monitoring system described in any one of (1) to (5) above, the computing device may further perform a detection process to detect an object whose posture has been estimated in the captured image, and a primary determination process to determine whether the object detected by the detection process is a candidate for a moving object or a candidate for a stationary object.

(7) Furthermore, the monitoring software according to a second aspect of the present disclosure is monitoring software that runs on a computing device connected to a storage device and an imaging device that generates captured images, and causes the computing device to perform the following: a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage process for storing the candidate for an unmoving object in the storage device as one of the unmoving objects if the result of the determination process is that the candidate for an unmoving object is an unmoving object; a matching process for comparing the candidate for a moving object with one or more unmoving objects stored in the storage device when the candidate for a moving object is detected in the captured image; and a monitoring process for determining that the candidate for a moving object is a moving object and monitoring the moving object if the result of the matching process is that there is no matching unmoving object; and does not cause the computing device to perform the process for monitoring the candidate for a moving object.

(8) In the monitoring software described in (7) above, the arithmetic device may further execute a process of deleting the oldest of the stored unmoving objects when the number of unmoving objects stored in the storage device reaches a predetermined value.

(9) In the monitoring software described in (7) or (8) above, the arithmetic device may further execute a second storage process to store the non-moving body candidate determined to be a moving body as a result of the determination process and the moving body candidate determined to be a moving body as a result of the matching process in the storage device, respectively, as one of the moving bodies. In the determination process, the arithmetic device may execute a second matching process to match the non-moving body candidate with one or more of the moving bodies stored in the storage device. If the second matching process does not find a matching moving body, the arithmetic device may execute a process to determine the non-moving body candidate as a non-moving body. If the second matching process finds a matching moving body, the arithmetic device may execute a process to determine the non-moving body candidate as a moving body.

(10) In the monitoring software described in (9) above, the arithmetic device may further execute a process of deleting the oldest of the stored moving objects when the number of moving objects stored in the storage device reaches a predetermined value.

(11) In the monitoring software described in any of (7) to (10) above, if the result of the determination process is an unmoving object, the arithmetic device may not be caused to execute a process for monitoring the candidate unmoving object, and if the result of the determination process is a moving object, the arithmetic device may be caused to determine that the candidate unmoving object is a moving object and execute a process for monitoring the moving object.

(12) In the surveillance software described in any of (7) to (11) above, the arithmetic device may further execute a detection process for detecting an object whose posture has been estimated in the captured image, and a primary determination process for determining whether the object detected by the detection process is a candidate for a moving object or a candidate for a stationary object.

(13) Furthermore, a monitoring method according to a third aspect of the present disclosure includes an image acquisition step of acquiring a captured image generated by an imaging device; a determination step of determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image; a storage step of storing the candidate for an unmoving object in a storage device as one of the unmoving objects when the result of the determination step is that the candidate for an unmoving object is an unmoving object; a comparison step of comparing the candidate for a moving object with one or more of the unmoving objects stored in the storage device when the result of the comparison step is that there is no matching unmoving object; and a monitoring step of determining that the candidate for a moving object is a moving object and monitoring the moving object.

(14) The monitoring method described in (13) above may further include a step of deleting the oldest of the stored unmoving objects when the number of unmoving objects stored in the storage device reaches a predetermined value.

(15) The monitoring method described in (13) or (14) above may further include a step of performing a second storage process to store the candidate unmoving body determined to be a moving body as a result of the determination process and the candidate moving body determined to be a moving body as a result of the matching process in the storage device, respectively, as one of the moving bodies; the determination process may perform a second matching process to match the candidate unmoving body with one or more of the moving bodies stored in the storage device; if the second matching process results in no matching moving body, the candidate unmoving body may be determined to be an unmoving body; and if the second matching process results in a matching moving body, the candidate unmoving body may be determined to be a moving body.

(16) The monitoring method described in (15) above may further include a step of deleting the oldest stored moving object when the number of moving objects stored in the storage device reaches a predetermined value.

(17) The monitoring method described in any of (13) to (16) above may further include a step of determining, if the result of the determination step is a moving object, that the candidate unmoving object is a moving object and monitoring the moving object.

(18) The monitoring method described in any of (13) to (17) above may further include a detection step of detecting an object whose pose has been estimated in the captured image, and a primary determination step of determining whether the object detected by the detection step is a candidate for a moving object or a candidate for a stationary object.

This disclosure provides a monitoring system, monitoring software, and monitoring method that can reduce the false person detection rate.

1 is a schematic diagram illustrating a configuration of a monitoring system according to a first embodiment. 10 is a flowchart illustrating a process of storing an unmoving object by the arithmetic device of the monitoring system according to the first embodiment. 10 is a flowchart illustrating a specific example of a storage process performed by a computing device of the monitoring system according to the first embodiment. 10A and 10B are schematic diagrams showing examples of candidates for unmoving objects detected in a captured image; 3 is a schematic diagram showing an example of an unmoving object already stored in a storage device of the surveillance system according to the first embodiment. FIG. FIG. 10 is a schematic diagram for explaining a method for calculating an area match rate using IoU. 10 is a flowchart illustrating a process of matching a candidate moving object by the arithmetic device of the surveillance system according to the first embodiment. 10 is a flowchart illustrating a specific example of a matching process performed by the arithmetic device of the monitoring system according to the first embodiment. 10 is a schematic diagram illustrating a case where a non-human, immobile object is erroneously detected as a candidate moving object, showing a scene in which a jacket draped over a chair is erroneously detected. 10 is a schematic diagram illustrating a case where a non-human, stationary object is erroneously detected as a candidate moving object, showing a scene in which a person moves while overlapping with a jacket that has been erroneously detected. 10 is a schematic diagram illustrating a case where a non-human, immobile object is erroneously detected as a candidate moving object, showing a scene in which a person passes by a jacket that has been erroneously detected. 10 is a flowchart illustrating processing by a computing device of a monitoring system according to a second embodiment. 10 is a flowchart illustrating a specific example of a monitoring process performed by a computing device of a monitoring system according to a second embodiment.

Embodiments of the monitoring system, monitoring software, and monitoring method according to the present disclosure will be described in detail below with reference to the drawings. The monitoring target of the monitoring system, monitoring software, and monitoring method according to the present disclosure is not particularly limited as long as it is a moving object, but the following description will focus on an example in which a person is the target.

Note that in the following explanation, repeated explanations of components with the same or similar functions will be omitted as appropriate.

(Embodiment 1)
First, an overview of the monitoring system according to the first embodiment will be described. In the monitoring system according to this embodiment, image patterns of non-human, stationary objects that have been mistakenly detected in the past are registered, and image patterns of moving objects detected thereafter are compared with the registered patterns. If a matching registered pattern is found, the moving object is not considered a monitoring target. This makes it possible to remove objects (e.g., non-human objects) that have been mistakenly detected as moving objects from the monitoring targets.

Figure 1 is a schematic diagram showing the configuration of a monitoring system according to embodiment 1.

The monitoring system 100 according to this embodiment is used in facilities such as nursing homes and hospitals. The monitoring system 100 detects monitored individuals, such as residents and hospitalized patients, as moving objects from captured images and monitors them. As shown in FIG. 1, the monitoring system 100 according to this embodiment includes a storage device 10, an image capture device 20 that generates captured images, and a computing device 30 that acquires the captured images generated by the image capture device 20.

The imaging device 20 is a camera installed in the monitored person's room and captures images of the monitored person within the room area. The imaging device 20 generates captured images, such as infrared images, and transmits the generated captured images to the computing device 30 via a communication network. The captured images are moving images. The imaging device 20 may capture images and generate captured images continuously at all times, or may capture images when the monitored person enters the imaging range.

Note that a room area is an area where a monitored person resides, is hospitalized, stays for an extended period of time, etc. A room area includes both a room where a monitored person stays alone (a private room) and a room where multiple monitored people stay together. If a facility has multiple room areas, a camera device 20 may be installed in each room area, or a camera device 20 may be installed in some of the room areas.

Furthermore, the imaging device 20 may be installed in a common area of the facility and may capture images of monitored persons within the common area. A common area is an area where monitored persons are permitted to enter, pass through, or stay, excluding living rooms, and examples include corridors, dining rooms, rehabilitation rooms, recreation rooms, visiting areas, etc. Multiple imaging devices 20 may be installed in one common area.

The computing device 30 is a computer located in an office or other location in the facility. The computing device 30 analyzes the captured images generated by the image capture device 20 and performs monitoring if the monitored person in the captured images is a moving object.

The computing device 30 may include a communication unit, a display unit (display device), an input unit, and a control unit.

The communication unit is a mechanism that sends and receives data to and from the imaging device 20 and storage device 10 via a communication network, etc.

The display unit is a mechanism that displays various screens to present information to facility staff, and specifically includes, for example, an LCD display device.

The input unit is a mechanism that accepts operational input, and specific examples include a keyboard and mouse.

The control unit is a mechanism that controls the operation of the computing device 30 and executes various processes described below. In practice, each process is executed by storing a program corresponding to each process, such as a monitoring program, in ROM or non-volatile memory, and loading and executing the program into the CPU.

The monitoring program may be pre-installed in the monitoring system 100, or may be provided to the user as an application program that can run on a general-purpose OS, recorded on a computer-readable recording medium, or via a network.

The storage device 10 is a device that stores data and is composed of storage devices such as an HDD, non-volatile RAM, and ROM. The storage device 10 is used to store programs executed by the calculation device 30 (control unit) and to temporarily store data when the control unit is operating.

Next, the processing performed by the arithmetic device 30 will be explained using Figures 2 to 11. The arithmetic device 30 is primarily characterized by performing a process of storing (registering) unmoving objects and a process of matching moving object candidates with the stored unmoving objects.

First, the process of storing unmoving objects will be described. Figure 2 is a flowchart illustrating the process of storing unmoving objects performed by the computing device of the surveillance system according to embodiment 1.

As shown in FIG. 2, the computing device 30 first acquires a captured image generated by the imaging device 20 (S10), and when a candidate for an unmoving object is detected in the captured image, performs a determination process to determine whether the candidate for an unmoving object is a moving object or a stationary object (S11). This determination process may be performed by comparing the candidate for an unmoving object with the history of past moving objects stored in the storage device 10. This comparison process will be described in detail as a second comparison process in embodiment 2.

Note that here, "candidate for unmoving object" refers to an object that has been detected as an unmoving object at least once before the determination process, such as an unmoving object detected by inter-frame differences before the determination process.

If the result of the determination process is an unmoving object (S12: unmoving object), the calculation device 30 performs a storage process to store (register) the candidate unmoving object in the storage device 10 as one of the unmoving objects (S13). Examples of information to be registered in the storage device 10 include a captured image of the unmoving object, the posture coordinates of the unmoving object, and the angle (orientation) of the imaging device 20 when the unmoving object was captured.

As a result of the above, image patterns of immovable objects (e.g., non-human objects) to be removed can be registered in the storage device 10.

As will be described later, if the result of the determination process is an unmoving object, the calculation device 30 may calculate the match rate (e.g., area match rate) between the unmoving object candidate and each unmoving object already stored in the storage device 10, and determine whether or not to store the unmoving object candidate in the storage device 10 based on the calculated match rate. For example, if there is an unmoving object whose match rate is equal to or greater than a predetermined threshold, the unmoving object candidate may not be stored in the storage device 10, and if there is no unmoving object whose match rate is equal to or greater than the threshold, the unmoving object candidate may be stored in the storage device 10 as one of the unmoving objects.

Also, as shown in FIG. 2, if the result of the determination process is an unmoving object, the calculation device 30 does not need to monitor the candidate unmoving object (S14). On the other hand, if the result of the determination process is a moving object (S12: moving object), the calculation device 30 may determine that the candidate unmoving object is a moving object and monitor the moving object (S15).

The computing device 30 may perform the storage process according to the flow shown in Figure 3. Figure 3 is a flowchart illustrating a specific example of the storage process performed by the computing device of the surveillance system according to embodiment 1. Figure 4 is a schematic diagram showing an example of a candidate unmoving object detected in a captured image.

In this case, as shown in Figure 4, the calculation device 30 first calculates the area 41 of the candidate immobile object within the captured image (S20). More specifically, the coordinates of the smallest rotated rectangle that contains all of the pose-estimated parts (skeleton points) are found. Figure 4 shows an example in which a jacket (a three-dimensional object) draped over a chair has been mistakenly detected.

Figure 5 is a schematic diagram showing examples of unmoving objects already stored in the storage device of the surveillance system according to embodiment 1. Figure 6 is a schematic diagram for explaining a method for calculating the area match rate using IoU.

Next, the calculation device 30 calculates the area match rate between the calculated area 41 of the candidate stationary object and the area 42 of each stationary object already stored in the storage device 10, as shown in Figure 5 (S21). Figure 5 shows an example in which a person (flat surface) on a poster hanging on the wall is stored in the storage device 10 as a stationary object. As the area match rate, for example, IoU (Intersection over Union) can be used, as shown in Figure 6.

As a result, if there is an immovable object whose area match rate is equal to or greater than a predetermined threshold (e.g., 50%) (S22: Yes), the calculation device 30 terminates the storage process without storing the immovable object candidate in the storage device 10 (S23); if there is no immovable object (S22: No), the calculation device 30 stores the immovable object candidate in the storage device 10 as one of the immovable objects (S24). In the example shown in Figures 4 to 6, the area between the area 41 of the erroneously detected jacket and the area 42 of the person on the registered poster is 0%, so if there are no other immovable objects in the storage device 10, the erroneously detected jacket will be stored in the storage device 10 as an immovable object.

The computing device 30 may perform the storage process using a FIFO (First In, First Out) method, and if, as a result of storing the immovable objects in S24, the number of immovable objects (items) stored in the storage device 10 reaches a predetermined value (e.g., 50 items) (S25: Yes), the oldest immovable object stored in the storage device 10 may be deleted (S26). On the other hand, if the number of immovable objects (items) stored in the storage device 10 is less than the predetermined value (e.g., 50 items), the storage process may be terminated without deletion.

Next, we will explain the process of matching moving object candidates with stored unmoving objects. Figure 7 is a flowchart explaining the process of matching moving object candidates performed by the computing device of the surveillance system according to embodiment 1.

As shown in FIG. 7 , when a candidate moving object is detected in a captured image, the computing device 30 performs a matching process to match the candidate moving object with one or more immovable objects stored in the storage device 10 (S30). This matching process may be performed by calculating the difference between the captured image of the candidate moving object and the captured image of the immovable object to be matched, as described below.

Note that here, "candidate for moving object" means an object that has been detected as a moving object at least once before the matching process, such as a moving object detected by inter-frame differences before the matching process.

If the result of the matching process shows that there is no matching stationary object (S31: No), the calculation device 30 determines that the candidate moving object is a moving object and performs a monitoring process to monitor the moving object (S32). On the other hand, if the result of the matching process shows that there is a matching stationary object (S31: Yes), the calculation device 30 does not monitor the candidate moving object (S33). As a result, even if a stationary object is mistakenly detected as a candidate moving object (for example, if a three-dimensional object that is not a person is mistakenly detected as a candidate moving object because a person or the like is passing by), if the stationary object is registered in the storage device 10, it can be removed from the list of objects to be monitored as not being a moving object (person). This reduces the rate of false human detection by the monitoring system 100.

Here, in the matching process, a moving object candidate matches an immovable object stored in the storage device 10, but this does not necessarily mean that the two (their captured images) match perfectly; it may also mean that the difference between the two captured images is below a predetermined threshold. In other words, the calculation device 30 may determine that a matching immovable object exists if the difference between the captured image of the moving object candidate and the captured image of the immovable object to be matched is below a predetermined threshold, and may determine that a matching immovable object does not exist if the difference exceeds the predetermined threshold.

Furthermore, at this time, the immovable objects to be compared may be all immovable objects stored in the storage device 10, or at least some of the immovable objects stored in the storage device 10, for example, immovable objects whose area matching rate with the moving object candidates is equal to or exceeds a predetermined threshold.

The computing device 30 may perform the matching process according to the flow shown in FIG. 8. FIG. 8 is a flowchart illustrating a specific example of matching process performed by the computing device of the surveillance system according to embodiment 1. First, a specific example in which a non-human, stationary object is erroneously detected as a candidate moving object will be described. FIG. 9 is a schematic diagram illustrating a case in which a non-human, stationary object is erroneously detected as a candidate moving object, showing a scene in which a jacket draped over a chair is erroneously detected. FIG. 10 is a schematic diagram illustrating a case in which a non-human, stationary object is erroneously detected as a candidate moving object, showing a scene in which a person moves while overlapping with the erroneously detected jacket. FIG. 11 is a schematic diagram illustrating a case in which a non-human, stationary object is erroneously detected as a candidate moving object, showing a scene in which a person passes beside the erroneously detected jacket.

As shown in Figure 9, if a jacket (a three-dimensional object) draped over a chair is mistakenly detected as a person, the jacket is a stationary object and can be removed from the monitoring target if left as is. However, as shown in Figures 10 and 11, if a person moves while overlapping with the erroneously detected jacket, i.e., if at least part of the jacket is hidden, inter-frame differences occur, and as a result, the jacket is detected as a candidate moving object. In such cases, the person could not be removed from the monitoring target using planar person removal or inter-frame differences, but can be removed as a stationary object using the matching process according to this embodiment.

In the matching process shown in Figure 8, as shown in Figure 11, the calculation device 30 first calculates the area 43 of the moving object candidate within the captured image (S40). In detail, the coordinates of the smallest rotated rectangle that contains all of the pose-estimated parts (skeleton points) are found.

Next, the calculation device 30 calculates the area match rate between the calculated area area 43 of the candidate moving object and the area area of each immovable object stored in the storage device 10 (S41). Immovable objects stored in the storage device 10 include, for example, the jacket draped over the chair shown in Figure 4 above and the person on the poster shown in Figure 5. The area match rate can be, for example, IoU.

Next, the calculation device 30 extracts immovable objects from the immovable objects stored in the storage device 10 that have an area matching rate equal to or greater than a predetermined threshold (e.g., 25%) (S42). In the example shown in Figure 11, the area matching rate with the person on the poster shown in Figure 5 is 0%, so they are not extracted, but the area matching rate with the jacket shown in Figure 4 is approximately 80%, so they are extracted.

At this time, the extracted stationary objects may be limited to those for which the angle of the camera device 20 at the time of shooting matches the angle of the camera device 20 at the time of shooting the candidate moving object (for example, when the difference between the two angles is within a predetermined threshold). In other words, stationary objects for which the angle of the camera device 20 at the time of shooting does not match the angle of the candidate moving object may be excluded from processing in the next step S43 and subsequent steps.

Next, the calculation device 30 calculates the difference between the captured image of the candidate moving object and the captured image of each extracted stationary object (S43). In more detail, the calculation device 30 calculates the difference in brightness between the area of the candidate moving object in the captured image and the area of the stationary object in the captured image.

As a result, if all differences are below a predetermined threshold (S44: Yes), the calculation device 30 determines that there is a matching immovable object and ends the matching process (S45); if there is even one difference that exceeds the threshold (S44: No), the calculation device 30 determines that there is no matching immovable object and ends the matching process (S46).

In the example shown in Figure 11, the difference between area 43 and area 41 of the jacket shown in Figure 4 is below the threshold, so a matching immobile object exists. In other words, the jacket shown in Figure 11 is determined to be an immobile object and is excluded from monitoring.

In this embodiment, when performing monitoring processing of a moving object, the computing device 30 may detect the behavior of the moving object from the captured image and determine whether the monitored person is in danger based on the detection results. If it is determined that the monitored person is in danger, an alarm may be activated to notify the facility manager of this. Note that, as a method for detecting the behavior of a moving object from the captured image, for example, a behavior detection model based on machine learning may be used.

The alarm device is a device that alerts facility staff and others when a monitored person is in a dangerous state, and is placed in the facility's staff room, nurse station, etc. The alarm device may be an LCD display device, an information terminal carried by staff (smartphone, tablet, etc.), a speaker that emits sound, a light that emits light, etc. The alarm device issues an alert to staff about a monitored person who is determined to be in a dangerous state. The alert may be issued by displaying an alert on the display unit of the alarm device, or by sound, light emission, vibration, etc.

As described above, the monitoring system 100 according to this embodiment can reduce the false person detection rate.

(Embodiment 2)
The monitoring system of embodiment 2 differs from the monitoring system of embodiment 1 in that the calculation device 30 further performs a process of detecting candidates for moving or stationary objects from the captured image before the judgment process, and a process of storing (registering) objects that are judged to be moving objects, and also performs a process of comparing candidates for stationary objects with the stored moving objects in the judgment process.

Furthermore, in this embodiment, the imaging device 20 is equipped with a depth sensor such as a FOT camera in addition to an image sensor to acquire the three-dimensional coordinates of the joints. The depth sensor recognizes the position and depth (depth) of the subject. Note that the depth sensor may also be a pattern illumination type sensor.

The processing performed by the arithmetic device 30 in this embodiment will be described using Figures 12 and 13. Figure 12 is a flowchart explaining the processing performed by the arithmetic device of the monitoring system according to embodiment 2. Note that in Figure 12, the same reference numerals are used to designate processes that are common to embodiment 1.

As shown in FIG. 12 , in this embodiment, after acquiring a captured image generated by the image capture device 20 (S10), the computing device 30 first performs a detection process to detect an object whose pose has been estimated in the captured image (S50). In particular, an object estimated to be a person is detected from the captured image using a person detection model based on machine learning, and multiple skeleton points of the object are extracted for each frame in a time series, and the coordinate displacement of the skeleton points over time is estimated. The coordinates of each skeleton point may be three-dimensional coordinates including depth (distance) information based on a depth sensor. Note that the object detected by this detection process may be an actual person or a non-human object.

Then, the computing device 30 performs a tracking process for the object detected by the detection process (S51). In detail, if the detected object is an object that has already been detected, the location information of the detected object is added to the registered ID. If the detected object is an object that has not yet been detected, a new ID is assigned, and the location information of the detected object is added to that ID.

Next, the calculation device 30 performs a primary determination process to determine whether the object detected by the detection process is a candidate for a moving object or a candidate for a stationary object (S52). In detail, the calculation device 30 determines whether the detected object is a moving object or a stationary object by taking the difference between frames of the captured image, and sets it as a candidate for a moving object or a candidate for a stationary object, respectively.

If the result of the primary determination process is a moving object candidate (S53: moving object), the calculation device 30 performs the matching process described in embodiment 1 (S30). That is, the moving object candidate is matched with one or more immobile objects stored in the storage device 10.

If the result of the primary determination process is a candidate for an unmoving object (S53: moving object), the calculation device 30 performs the determination process described in embodiment 1 (S11). That is, it determines whether the candidate for an unmoving object is a moving object or a unmoving object. In this embodiment, the calculation device 30 thus determines whether the detected object is a moving object or a unmoving object in the primary determination process and the subsequent determination process (secondary determination process).

Furthermore, the calculation device 30 performs a second storage process in which the candidate stationary object determined to be a moving object as a result of the determination process (S12: moving object) and the candidate moving object determined to be a moving object as a result of the matching process (S31: moving object) are each stored (registered) as one of the moving objects in the storage device 10 (S54).

Specifically, for example, the activity amount of the moving object is stored as a person activity amount in an ID determined in the object tracking process. Here, the person activity amount is a value based on the image difference amount calculated in the primary determination process or the matching process. The person activity amount is updated, for example, at 5 FPS.
It is calculated using the formula: Person activity amount (after update) = Person activity amount (before update) + Image difference amount - Activity amount attenuation constant. The activity amount attenuation constant is a constant obtained experimentally. In the primary determination process, the calculation device 30 calculates this person activity amount, and if the person activity amount is equal to or greater than a predetermined threshold, determines that the object is a candidate for a moving object, and if it is less than the threshold, determines that the object is a candidate for a stationary object.

The computing device 30 may perform the second storage process using a FIFO method, and if, as a result of storing the moving objects in S54, the number of moving objects (items) stored in the storage device 10 reaches a predetermined value, the oldest moving object stored in the storage device 10 may be deleted. On the other hand, if the number of moving objects (items) stored in the storage device 10 is less than the predetermined value, the second storage process may be terminated without deletion.

Then, in the determination process (secondary determination process) of S11, the calculation device 30 performs a second matching process to match the candidate unmoving object with one or more moving objects stored in the storage device. If the result of the second matching process is that there is no matching moving object, the calculation device 30 determines that the candidate unmoving object is an unmoving object (S12: unmoving object). If the result of the second matching process is that there is a matching moving object, the calculation device 30 determines that the candidate unmoving object is a moving object (S12: moving object). In more detail, the calculation device 30 checks whether an object with the same ID as the candidate unmoving object is stored in the storage device 10 as a moving object (an object whose human activity level is above a predetermined threshold), and determines that a matching moving object exists if a history of such a moving object exists, and determines that a matching moving object does not exist if a history of such a moving object does not exist.

Furthermore, in this embodiment, after the second storage process, the calculation device 30 performs a moving object monitoring process (S55).

The computing device 30 may execute the monitoring process (S55) according to the flow shown in FIG. 13. FIG. 13 is a flowchart illustrating a specific example of the monitoring process performed by the computing device of the monitoring system according to embodiment 2.

In this case, the computing device 30 first checks the reliability of the 3D coordinates of the moving object (S60). More specifically, based on the 3D landmark information of the moving object acquired by the depth sensor, it determines whether the object is valid as a person. For example, if the moving object exists on a flat surface or if the moving object is exhibiting an unpredictable posture, it is determined to be unreliable. Then, if the 3D coordinates of the moving object are reliable (S61: Yes), the computing device 30 may detect the behavior of the moving object from the captured image and determine whether the monitored person is in danger based on the detection results (S62). On the other hand, if the 3D coordinates of the moving object are unreliable (S61: Yes), the computing device 30 may continue to monitor the moving object without detecting its behavior (S63).

Similar to embodiment 1, this embodiment can also reduce the false person detection rate.

Although the embodiments have been described above with reference to the drawings, the present disclosure is not limited to the above-described embodiments. Furthermore, the configurations of the respective embodiments may be combined or modified as appropriate within the scope of the gist of the present disclosure.

As described above, this disclosure provides a technology that is useful for reducing the false human detection rate in surveillance systems that use captured images.

100: Monitoring system 10: Storage device 20: Photography device 30: Computing device 41, 42, 43: Area

Claims (8)

A storage device;
an imaging device that generates a captured image;
a computing device that acquires the captured image generated by the imaging device,
The computing device further includes a detection process for detecting an object whose pose has been estimated in the captured image;
a tracking process of the object detected by the detection process;
a primary determination process for determining whether the object detected by the detection process is a moving object candidate or a stationary object candidate;
a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image;
a storage process for storing the candidate for the unmoving object in the storage device as one of the unmoving objects when the result of the determination process is an unmoving object;
a comparison process for comparing, when a candidate moving object is detected in the captured image, the candidate moving object with one or more of the unmoving objects stored in the storage device;
If there is no matching unmoving object as a result of the comparison process, a monitoring process is performed in which the candidate moving object is determined to be a moving object and the moving object is monitored;
If there is a matching non-moving object as a result of the matching process, the candidate moving object is not monitored;
In the primary determination process, the arithmetic device calculates a person activity amount based on an image difference amount, which is an inter-frame difference of the captured image, and an experimentally determined activity amount attenuation constant, and updates the person activity amount based on the calculated person activity amount, the image difference amount, and the activity amount attenuation constant, and determines that the object detected by the detection process is a candidate for a moving object if the updated person activity amount is equal to or greater than a predetermined threshold, and determines that the object detected by the detection process is a candidate for a non-moving object if the updated person activity amount is less than the predetermined threshold.
A monitoring system characterized by: The monitoring system described in claim 1, characterized in that the computing device further deletes the oldest stored unmoving object when the number of unmoving objects stored in the storage device reaches a predetermined value. the computing device further performs a second storage process to store the candidate for the non-moving object determined to be a moving object as a result of the determination process and the candidate for the moving object determined to be a moving object as a result of the matching process in the storage device as one of the moving objects;
In the determination process, a second matching process is performed to match the candidate unmoving object with one or more moving objects stored in the storage device;
If there is no matching moving object as a result of the second matching process, the candidate for the non-moving object is determined to be a non-moving object;
3. The monitoring system according to claim 1, wherein if a matching moving object is found as a result of the second comparison process, the candidate for the unmoving object is determined to be a moving object. The surveillance system described in claim 3, characterized in that the computing device further deletes the oldest stored moving object when the number of moving objects stored in the storage device reaches a predetermined value. When the result of the determination process is an unmoving object, the calculation device does not monitor the unmoving object candidate,
5. The monitoring system according to claim 1, wherein, when the result of the determination process indicates a moving object, the candidate for the unmoving object is determined to be a moving object, and the moving object is monitored. The monitoring system according to any one of claims 1 to 5, wherein in the primary determination process, the arithmetic device calculates the updated person activity amount by adding an image difference amount to the person activity amount before the update and subtracting the activity amount attenuation constant from the person activity amount before the update. A monitoring software that runs on a computing device connected to a storage device and an image capturing device that generates captured images,
a detection process for detecting an object whose pose has been estimated in the captured image, in the computing device;
a tracking process of the object detected by the detection process;
a primary determination process for determining whether the object detected by the detection process is a moving object candidate or a stationary object candidate;
a determination process for determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image;
a storage process for storing the candidate for the unmoving object in the storage device as one of the unmoving objects when the result of the determination process is an unmoving object;
a comparison process for comparing, when a candidate moving object is detected in the captured image, the candidate moving object with one or more of the unmoving objects stored in the storage device;
If there is no matching unmoving object as a result of the comparison process, the candidate moving object is determined to be a moving object, and a monitoring process is executed to monitor the moving object;
If there is a matching unmoving object as a result of the matching process, the process of monitoring the moving object candidate is not executed,
In the primary determination process, a person activity amount is calculated based on an image difference amount, which is an inter-frame difference of the captured image, and an experimentally determined activity amount attenuation constant, and the person activity amount is updated based on the calculated person activity amount, the image difference amount, and the activity amount attenuation constant, and if the updated person activity amount is equal to or greater than a predetermined threshold, the object detected by the detection process is determined to be a candidate for a moving object, and if the updated person activity amount is less than the predetermined threshold, the object detected by the detection process is determined to be a candidate for a non-moving object.
The monitoring software is characterized by: an image acquisition step of acquiring a photographed image generated by the photographing device;
a detection step of detecting an object whose pose has been estimated in the captured image;
a tracking step of the object detected by the detecting step;
a primary determination step of determining whether the object detected by the detection step is a moving object candidate or a non-moving object candidate;
a determination step of determining whether a candidate for an unmoving object is a moving object or a unmoving object when the candidate for an unmoving object is detected in the captured image;
a storage step of storing the candidate for the unmoving object in a storage device as one of the unmoving objects when the result of the determination step is an unmoving object;
a matching step of matching a candidate moving object with one or more of the unmoving objects stored in the storage device when the candidate moving object is detected in the captured image;
a monitoring step of determining that the candidate moving object is a moving object and monitoring the moving object when there is no matching unmoving object as a result of the comparing step,
In the primary determination step, a person's activity amount is calculated based on an image difference amount, which is a difference between frames of the captured image, and an experimentally determined activity amount attenuation constant, and the person's activity amount is updated based on the calculated person's activity amount, the image difference amount, and the activity amount attenuation constant. If the updated person's activity amount is equal to or greater than a predetermined threshold, the object detected in the detection step is determined to be a candidate for a moving object, and if the updated person's activity amount is less than the predetermined threshold, the object detected in the detection step is determined to be a candidate for a non-moving object.
A monitoring method comprising: