JP7707958B2 - Correction system, correction method, and correction program - Google Patents

Description

The present invention relates to a correction system, a correction method, and a correction program.

In Japan, the high economic growth after the war has led to improved living standards, improved sanitary conditions, and improved medical standards, resulting in a notable increase in life expectancy. This, combined with a declining birth rate, has resulted in an aging society with a high aging rate. In such an aging society, it is expected that the number of people who require nursing care due to illness, injury, aging, etc. will increase.

In facilities such as hospitals and elderly care facilities, people who require care may fall while walking or fall out of bed and injure themselves. For this reason, systems are being developed that can detect the condition of people who require care from captured images so that caregivers, nurses, and other staff can rush to the scene immediately when a person who requires care falls into such a state. In order for such a system to detect the condition of a person who requires care, it is necessary to detect the posture and behavior of the person being detected from the image with a high degree of accuracy.

The following prior art is disclosed in Patent Document 1 below. A moving object is detected from a moving image, the movement line of the detected object is obtained, and the type or attribute of the object is identified based on the shape of the movement line. In this way, the type or attribute of the moving object is identified.

JP 2016-57998 A

However, while the above prior art can identify the type or attributes of a moving object, it has the problem that it cannot improve the accuracy of detecting an object regardless of whether the object is moving or not.

The present invention has been made to solve these problems. In other words, the objective is to provide a correction system, a correction method, and a correction program that can improve the accuracy of object detection regardless of whether the object is moving or not.

The above-mentioned problems of the present invention are solved by the following means.

(1) A correction system having an acquisition unit that acquires an image in which a specified area is captured, a detection unit that detects position information of a person and an object from the image including the person and the object using a trained model that has been trained to detect position information of the person and the object, respectively, a switching unit that switches a correction process for the object's position information according to the object's position information detected by the detection unit, and a correction unit that corrects the object's position information detected by the detection unit using the correction process after switching by the switching unit.

(2) The correction system described in (1) above, wherein the switching unit includes a moving object determination unit that determines whether the object is moving based on the position information of the object detected by the detection unit, and a correction process switching unit that switches the correction process of the position information of the object by the correction unit depending on the determination result by the moving object determination unit.

(3) The correction system described in (2) above, in which the position information includes the attribute class and position of each of the person and the object, the detection unit has an attribute class identification unit that identifies the attribute class of the person and the object based on the image, and a position identification unit that identifies the position of the person and the object based on the image, and the moving object determination unit determines whether or not the object can move based on the attribute class of the object identified by the attribute class identification unit, using a reference table in which the relationship between the attribute class and whether or not the object can move is set.

(4) The correction system described in (2) above, in which the switching unit has a movement likelihood calculation unit that calculates the likelihood of the object's movement from the position information of the object, and the moving object determination unit determines whether the object is capable of moving based on the likelihood of the object calculated by the movement likelihood calculation unit.

(5) The correction system described in (1) above, wherein the correction unit includes a stationary object correction unit that performs a correction to remove erroneous detection of the position information of the object when it is determined that there is contact between the person and the object, when the switching unit has switched to the correction process of the position information of the object that is not moving, and a moving object correction unit that performs a correction to complement non-detection of the position information of the object when it is determined that there is contact between the person and the object, when the switching unit has switched to the correction process of the position information of the object that is moving.

(6) The correction system described in (5) above, in which the stationary object correction unit determines whether an object determined to be in contact with the person is detected based on the position information of the object detected in the past image taken before the image in which the position information of the object was detected, and performs correction to delete the position information of the object determined to be in contact with the person, and the moving object correction unit does not delete the position information of the object determined to be in contact with the person even when it determines that the object is in contact with the person.

(7) The correction system described in (5) or (6) above, in which the moving object correction unit, when it determines that there is contact between the person and the object, tracks the object determined to have been in contact with the person in the past image in which the specified area was captured before the image in which the position information was detected, and performs correction to complement the position information of the object that cannot be confirmed by tracking.

(8) A correction method comprising the steps of: (a) acquiring an image in which a predetermined area is captured; (b) detecting position information of a person and an object from the image including the person and the object using a trained model that has been trained to detect position information of the person and the object, respectively; (c) switching a correction process for the object's position information according to the object's position information detected in (b); and (d) correcting the object's position information detected in (b) by the correction process after switching in (c).

(9) The correction method described in (8) above, in which the step (c) includes a step (c1) of determining whether the object can move based on the position information of the object detected in the step (b), and a step (c2) of switching the correction process of the position information of the object in the step (d) depending on the determination result in the step (c1).

(10) The correction method described in (9) above, in which the position information includes the attribute class and position of each of the person and the object, and the step (b) includes a step (b1) of identifying the attribute class of the person and the object based on the image, and a step (b2) of identifying the position of the person and the object based on the image, and the step (c1) determines whether or not the object can be moved based on the attribute class of the object identified in the step (b1) using a reference table in which the relationship between the attribute class and whether or not the object can be moved is set.

(11) The correction method described in (9) above, in which the step (c) includes a step (c3) of calculating the likelihood of the object's movement from the position information of the object, and the step (c1) determines whether the object can move based on the likelihood of the object calculated in the step (c3).

(12) The correction method described in (8) above, in which the step (d) includes a step (d1) of performing a correction to remove a false positive in the position information of the object when it is determined that the person and the object are in contact with each other when the step (c) is switched to the correction process of the position information of the object that is not moving, and a step (d2) of performing a correction to complement a non-detection in the position information of the object when it is determined that the person and the object are in contact with each other when the step (c) is switched to the correction process of the position information of the object that is moving.

(13) The correction method described in (12) above, in which, in step (d1), a false detection is determined based on the position information of the object detected in the past image taken before the image in which the position information of the object is detected, and a correction is performed to delete the position information of the object determined to be a false detection, and in step (d2), even if it is determined that the person and the object are in contact, the position information of the object determined to be in contact with the person is not deleted.

(14) In the correction method described in (12) or (13) above, in the step (d2), if it is determined that there is contact between the person and the object, the object determined to have been in contact with the person is tracked in the past image in which the specified area was captured before the image in which the position information was detected, and a correction is made to complement the position information of the object that cannot be confirmed by tracking.

(15) A correction program for causing a computer to execute the correction method described in any one of (8) to (14) above.

Using a trained model that has been trained to detect the position information of people and objects from images that include people and objects, the correction process for the object's position information is switched according to the object's position information detected from an image of a specified area, and the object's position information is corrected by the correction process after switching. This makes it possible to improve the detection accuracy of objects regardless of whether the object is moving or not.

FIG. 1 is a diagram showing a schematic configuration of a detection system. FIG. 2 is a block diagram showing a configuration of the imaging device. FIG. 2 is a block diagram showing the configuration of a server. FIG. 2 is a functional block diagram of a control unit. FIG. 11 is an explanatory diagram for explaining a correction process for a stationary object. An explanatory diagram showing an image of training data used to learn the learned model of the detection unit. 10 is an explanatory diagram showing an example of position information normally detected by a detection unit, together with an image acquired by an acquisition unit; FIG. 11 is an explanatory diagram showing an example of position information including an object rectangle erroneously detected by a detection unit, together with an image acquired by an acquisition unit. FIG. 11 is an explanatory diagram for explaining a correction process regarding a moving object. FIG. FIG. 2 is a block diagram showing a configuration of a mobile terminal. 4 is a flowchart showing the operation of the detection system. FIG. 2 is a functional block diagram of a control unit.

Below, a correction system, a correction method, and a correction program according to an embodiment of the present invention will be described with reference to the drawings. Note that in the drawings, identical elements are given the same reference numerals, and duplicate explanations will be omitted. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation, and may differ from the actual ratios.

First Embodiment
FIG. 1 is a diagram showing a schematic configuration of a detection system 10. As shown in FIG.

The detection system 10 includes an image capture device 100, a server 200, a communication network 300, and a mobile terminal 400. The image capture device 100 is connected to the server 200 via the communication network 300 so that they can communicate with each other. The mobile terminal 400 can be connected to the communication network 300 via an access point 310. The server 200 constitutes a correction system. Note that the image capture device 100 may be configured to perform some or all of the functions of the server 200. In this case, the image capture device 100 alone or together with the server 200 may constitute the correction system.

(Photographing device 100)
Fig. 2 is a block diagram showing the configuration of the image capturing device 100. As shown in the example of Fig. 2, the image capturing device 100 includes a control unit 110, a communication unit 120, and a camera 130, which are interconnected by a bus.

The control unit 110 is composed of a CPU (Central Processing Unit) and memories such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and controls each part of the image capturing device 100 and performs calculation processing according to a program. The control unit 110 transmits an image 600 (see Figures 5, 7, and 8) obtained by the camera 130 capturing an image of a specific area to the server 200 or the like via the communication unit 120. Hereinafter, the image 600 captured by the camera 130 will also be referred to simply as "image 600". The specific area is, for example, a three-dimensional area that includes the entire floor surface of the person 500's room.

The communication unit 120 is an interface circuit (e.g., a LAN card, etc.) for communicating with the mobile terminal 400, etc., via the communication network 300.

The camera 130 is, for example, a wide-angle camera. The camera 130 is installed in a position overlooking a specific area by installing the imaging device 100 on the ceiling or the like of the room of the person 500, and captures the specific area. The person 500 is, for example, a person who requires care or nursing from a staff member or the like, and is the subject of behavior detection. The camera 130 may be a standard camera with a narrower angle of view than a wide-angle camera. The image 600 may include the person 500 and the object 700, such as a bed 710 and a chair 720. The image 600 includes still images and videos. The camera 130 is a near-infrared camera that can capture a specific area by irradiating a capture area with near-infrared light using an LED (Light Emitting Device) and receiving the near-infrared light reflected by an object in the capture area using a CMOS (Complementary Metal Oxide Semiconductor) sensor. The image 600 can be a monochrome image in which each pixel represents the reflectance of near-infrared light. The camera 130 can be a visible light camera instead of a near-infrared camera, or a combination of the two.

(Server 200)
3 is a block diagram showing the configuration of the server 200. The server 200 includes a control unit 210, a communication unit 220, and a storage unit 230. The components are connected to each other via a bus.

The basic configuration of the control unit 210 and the communication unit 220 is similar to that of the control unit 110 and the communication unit 120, which are the corresponding components of the imaging device 100, so a duplicated explanation will be omitted. The storage unit 230 is composed of a RAM, a ROM, a HDD (Hard Disc Drive), etc.

Figure 4 is a functional block diagram of the control unit 210. The control unit 210 functions as an acquisition unit 211, a detection unit 212, a switching unit 213, and a correction unit 214. The control unit 210 also functions as a behavior estimation unit (not shown).

The acquisition unit 211 acquires the image 600 by receiving it from the image capture device 100 via the communication unit 220. If the image 600 received from the image capture device 100 is stored in the storage unit 230, the acquisition unit 211 may acquire the image 600 by reading it from the storage unit 230. The image 600 includes an image 600 including a person 500 and an object 700. The object includes an object 700 that may move (hereinafter referred to as a "moving object") and an object 700 that may not move (hereinafter referred to as a "stationary object"). The moving object includes, for example, a wheelchair or a walker. The stationary object includes a chair 720 or a bed 710. Note that the stationary object also includes an object 700 that is relatively unlikely to move. The moving object constitutes a moving object. The stationary object constitutes an object that does not move.

The detection unit 212 detects the position information of the person 500 and the object 700 from the image 600 including the person 500 and the object 700 using a trained model that has been trained to detect the position information of the person 500 and the object 700. For example, a Region Proposal Network (RPN), which is a neural network model, may be used as the trained model.

The position information is information on the attribute class and position of each of the person 500 and the object 700. That is, the position information is information indicating the position for each attribute class. The attribute class is a classification of the person 500 and each object 700, such as "person," "wheelchair," "chair," "walker," and "bed." The position may be a coordinate on the image 600.

The detection unit 212 may include an attribute identification unit 2121 and a position identification unit 2122. The attribute identification unit 2121 identifies (estimates) the attribute classes of the person 500 and the object 700 based on the image 600. The position identification unit 2122 identifies (estimates) the positions of the person 500 and the object 700 (i.e., the positions for each attribute class) based on the image 600. The attribute identification unit 2121 constitutes an attribute class identification unit.

The attribute identification unit 2121 can detect candidate rectangles that are areas that include a person 500 or an object 700 from the image 600, and identify (estimate) an attribute class for each candidate rectangle. A candidate rectangle that includes a person 500 is also referred to as a "person rectangle 610" (see Figures 5, 7, and 8). A candidate rectangle that includes an object 700 is also referred to as an "object rectangle 620" (see Figures 5 and 8).

The position identification unit 2122 identifies the coordinates of the human rectangle 610 and the object rectangle 620 (e.g., each coordinate of a pair of diagonal corners of the rectangle) as the positions of the human rectangle 610 and the object rectangle 620, respectively.

The switching unit 213 switches the correction process for the position information of the object 700 according to the position information of the object 700 detected by the detection unit 212 (specifically, the attribute class of the object rectangle 620). The switching unit 213 may include a moving object determination unit 2131 and a correction process switching unit 2132.

The moving object determination unit 2131 determines whether the object 700 is moving or not based on the position information of the object 700. Specifically, the moving object determination unit 2131 determines whether the object 700 detected as the object rectangle 620 is a moving object or a stationary object based on the position information of the object 700.

Specifically, the moving object determination unit 2131 can determine whether the object 700 can move, based on the attribute class of the object 700 identified by the attribute identification unit 2121, using a reference table in which the relationship between the attribute class and whether the object can move is set. That is, the moving object determination unit 2131 uses the reference table to determine whether the object 700 detected as the object rectangle 620 is a moving object or a stationary object.

The correction process switching unit 2132 switches the correction process of the position information of the object 700 by the correction unit 214 according to the determination result by the moving object determination unit 2131. Specifically, the correction process switching unit 2132 switches the correction process by the correction unit 214 to either the correction process of the position information of a moving object or the correction process of the position information of a stationary object. The correction process switching unit 2132 outputs correction process information to the correction unit 214 for controlling whether the correction process is to be performed, the correction process of the position information of a moving object or the correction process of the position information of a stationary object.

The correction unit 214 corrects the position information of the object 700 detected by the detection unit 212 using a correction process after switching by the switching unit 213, and outputs the corrected position information. That is, the correction unit 214 corrects the position information of the object 700, among the position information detected by the detection unit 212, using a correction process after switching, and outputs the corrected position information. The correction unit 214 may include a stationary object correction unit 2141 and a moving object correction unit 2142. Note that the attribute class of the object 700 output from the detection unit 212 to the switching unit 213 and the position information output from the detection unit 212 to the correction unit 214 are linked by a unique ID or the like.

When the switching unit 213 switches to the correction process for the position information of a stationary object, the stationary object correction unit 2141 performs a correction (hereinafter referred to as a "correction process for a stationary object") on the position information of the object 700 to remove a false detection of the object 700 when it is determined that the person 500 is in contact with the object 700. When the switching unit 213 switches to the correction process for the position information of a moving object, the moving object correction unit 2142 performs a correction (hereinafter referred to as a "correction process for a moving object") on the position information of the object 700 to complement the non-detection of the object 700. The correction process for a moving object may be performed on a past image 600. The past image 600 may be an image 600 taken before the image 600 for which the position information that was the basis for switching the correction process by the switching unit 213 was detected.

Figure 5 is an explanatory diagram for explaining the correction process for a stationary object. In Figure 5, for ease of explanation, a person rectangle 610 and an object rectangle 620 are shown on an image 600.

5, in the current image 600, the object 700 is detected as an object rectangle 620. However, if the detected object 700 is a stationary object, it may have been erroneously detected for the reasons described below. Whether or not it is an erroneous detection can be determined by whether or not an object 700 of the same attribute class is detected at the same position (including approximately the same position) in the past image 600 going back a predetermined time. The predetermined time can be set to an appropriate value by experimentation from the viewpoint of the detection accuracy of the object 700. Since there is no object rectangle 620 (the object 700 is not detected) in the past image 600 during movement (the person 500 is moving), it can be determined that the object 700 detected in the current image 600 is an erroneous detection. This is because if it is a stationary object (e.g., a chair), the object 700 detected in the current image 600 should be detected at the same position in the past image 600 because the stationary object does not move, but it is not detected. Therefore, in the correction process for stationary objects, when an object 700 detected in the current image 600 has not been detected in the past image 600, the position information of the object 700 included in the position information detected in the current image 600 is deleted. That is, the stationary object correction unit 2141 determines whether an object 700 determined to be in contact with a person 500 is erroneously detected based on the position information of the object 700 detected in the past image 600 taken before the image 600 in which the position information of the object 700 was detected, and performs correction to delete the position information of the object determined to be erroneously detected. In FIG. 5, the object rectangle 620 corresponding to the position information of the object 700 to be deleted is shown as a rectangle with dashed lines.

In FIG. 5, object 700 is also detected in image 600 before movement (before person 500 moves), but since object 700 is determined to be a stationary object, correction can be made to remove object 700, similar to the correction made to current image 600.

The correction process for stationary objects corresponds to a correction process for correcting false detection of object 700.

Examples of why stationary objects may be erroneously detected are described with reference to Figures 6 to 8.

Figure 6 is an explanatory diagram showing an image 650 of training data used to learn the trained model of the detection unit 212. In order to simplify the explanation, in Figure 6, a person rectangle 610 and an object rectangle 620 indicating the positions of the person 500 and the object 700 (chair 720), respectively, which are the correct answer labels, are also shown on the image 650.

As shown in the example of FIG. 6, a neural network model can be trained using a relatively large amount of training data that combines an image 650 including a person 500 and an object 700 with correct labels (correct attribute classes and positions) of a person rectangle 610 and an object rectangle 620, thereby generating a trained model.

Figure 7 is an explanatory diagram showing an example of position information normally detected by the detection unit 212, together with an image 600 acquired by the acquisition unit 211. In Figure 7, for ease of explanation, a human rectangle 610, which is the detected position information, is shown on the image 600. If the image 600 acquired by the acquisition unit 211 contains a person 500 but does not contain an object 700, then if the position information is detected normally, the position etc. of the person 500 will be detected as a human rectangle 610, and a non-existent object will not be detected as an object rectangle 620.

8 is an explanatory diagram showing an example of position information including an object rectangle 620 erroneously detected (abnormally detected) by the detection unit 212, together with an image 600 acquired by the acquisition unit 211. In FIG. 8, for ease of explanation, the detected human rectangle 610 and object rectangle 620 are shown on the image 600. In the example shown in FIG. 8, when the image 600 acquired by the acquisition unit 211 includes a person 500 and does not include an object 700, the position of the person 500 is correctly detected as the human rectangle 610. On the other hand, a chair 720, which is a non-existent object 700, is erroneously detected as the object rectangle 620 of the chair 720. Such erroneous detection of the object 700 is considered to be due to, for example, the use of an image including both the person 500 and the object 700 in the image 650 of the training data used for learning the learned model, whereby the features of the person 500 and the features of the object 700 around the person 500 are associated and learned. This indicates that, depending on the movement and posture of the person 500, there is a possibility that an object 700 may be erroneously detected in the space around the person 500 where nothing is present.

When a chair 720 or the like, which is such a non-existent object 700, is erroneously detected, in an estimation of the posture of the person 500 based on the position information of the person 500 and the object 700, the posture of sitting on the floor is erroneously estimated as the posture of sitting on a chair. As a result, in an estimation of the behavior of the person 500 based on the posture of the person 500, the behavior of the person 500 falling (or the posture of sitting on the floor) may be erroneously estimated as the behavior of sitting on the chair 720 (or the posture of sitting on a chair).

When it is determined that there is contact between the person 500 and the object 700, the correction process for a stationary object may determine whether the object 700 has been erroneously detected based on whether an object 700 of the same attribute class has been detected at the same position in the past image 600. That is, the contact between the person 500 and the object 700 may trigger processing to determine whether the object 700 has been erroneously detected. Whether there is contact between the person 500 and the object 700 may be determined based on whether the size of the overlap between the person rectangle 610 and the object rectangle 620 exceeds a predetermined threshold. The predetermined threshold may be set to an appropriate value through experiments in terms of the detection accuracy of the object 700.

Figure 9 is an explanatory diagram for explaining the correction process for a moving object. In Figure 9, for ease of explanation, a person rectangle 610 and an object rectangle 620 are shown on an image 600.

9, in the current image 600, the object 700 is detected as an object rectangle 620. If the detected object 700 is a moving object, for example, a wheelchair, it is assumed that the person rectangle 610 and the object rectangle 620 will be located close to each other. In addition, since the object 700 moves together with the person 500, it is naturally assumed that the object 700 of the same attribute class will not be detected at the same position (including approximately the same position) in the past image 600 going back a predetermined time. Therefore, in the correction process for the moving object, even if it is determined that the person 500 and the object 700 are in contact with each other, the position information of the object 700 determined to be in contact with the person 500 is not deleted. Therefore, in FIG. 9, the moving object 700 detected as the object rectangle 620 from the image 600 before and after the movement (current) is not deleted in the correction process for the moving object.

In the correction process for moving objects, an undetected object 700 (an object 700 that actually exists but has not been detected) is corrected to be complemented with position information detected from the past image 600. Specifically, when the moving object correction unit 2142 determines that there is contact between the person 500 and the object 700, it tracks the object 700 determined to have been in contact with the person 500 in the past image 600 in which a predetermined area was captured before the image 600 in which the position information was detected, and performs a correction to complement the position information of the object 700 that cannot be confirmed by tracking. The object 700 in the past image 600 can be tracked using a known tracking method. For example, SORT (Simple Online and Realtime Tracking) can be used to track the object 700 in the past image 600.

In FIG. 9, object 700 is not detected as object rectangle 620 in image 600 during movement. Therefore, as shown by the dashed-dotted rectangle, the object rectangle 620 of the moving object is complemented in the position information detected from image 600 during movement. In other words, the position information of object 700, which is a moving object, is complemented in the position information detected from image 600 during movement.

The object 700 can be supplemented in the correction process for a moving object as follows. For example, the position information of the object 700 is supplemented to the position information detected from the past image 600 in which the face object 700 is not detected, so that the relative positional relationship between the person 500 and the object 700, which are in contact with each other, is maintained in the past image 600 in which the face object 700 is not detected.

In this way, the correction process for the moving object corresponds to the correction process for correcting the non-detection of the object 700.

The corrected location information can be used by a behavior estimation unit (not shown) to detect the behavior of person 500.

The behavior estimation unit estimates the behavior of the person 500 based on the corrected position information. Specifically, the behavior estimation unit may estimate the behavior of the person 500 as follows. The behavior estimation unit estimates the joint points based on the human rectangle 610 (more specifically, the image included in the human rectangle 610). The behavior estimation unit may estimate the joint points of the person 500 using a neural network model that has been trained to estimate the joint points of the person 500 from the human rectangle 610. As the neural network model, a well-known model such as Deep Pose may be used. The behavior estimation unit may estimate the behavior of the person 500 based on the relationship between the joint points and the object rectangle 620. Specifically, the behavior of the person 500 may be estimated based on the relationship between the posture of the person 500 indicated by the joint points and the position and attribute class of the object 700 indicated by the object rectangle 620. For example, if the object indicated by the object rectangle 620 is a chair 720 and the posture indicated by the joint points is a sitting position, the behavior estimation unit may estimate that the person 500 is performing a sitting position on a chair if a predetermined number or more of the joint points overlap with the object rectangle 620.

When the control unit 210 determines that the detected behavior of the person 500 corresponds to an event that has been preset as a behavior that requires notification to care staff, such as a fall or a trip, the control unit 210 may transmit an event notification to the mobile device 400, including the behavior corresponding to the event, the name of the person 500, the room number, etc. Furthermore, the control unit 210 may notify the mobile device 400 of the detected behavior of the person 500 periodically or when the behavior changes.

(Mobile terminal 400)
10 is a block diagram showing the configuration of mobile terminal 400. Mobile terminal 400 includes a control unit 410, a wireless communication unit 420, a display unit 430, an input unit 440, and an audio input/output unit 450. The components are connected to each other via a bus. Mobile terminal 400 may be configured by a communication terminal device such as a tablet computer, a smartphone, or a mobile phone, for example.

The control unit 410 has basic components such as a CPU, RAM, and ROM, similar to the configuration of the control unit 110 of the imaging device 100.

The wireless communication unit 420 has a function for performing wireless communication according to standards such as Wi-Fi and Bluetooth (registered trademark), and performs wireless communication with each device directly or via the access point 310. The wireless communication unit 420 receives event notifications from the server 200.

The display unit 430 and the input unit 440 are touch panels, and a touch sensor as the input unit 440 is provided on the display surface of the display unit 430, which is composed of a liquid crystal or the like. The display unit 430 displays the behavior of the subject 510 and the event notification received from the server 200. The display unit 430 and the input unit 440 may display an input screen that prompts the subject 510 to respond to the event notification, and may also accept the staff's intention to respond to the event notification, which is input on the input screen, and transmit it to the server 200. In this case, the server 200 may determine one of the staff members as the staff member in charge of responding to the event notification, and transmit a notification to the mobile terminals 400 of all staff members, including the determined staff member, informing them of the staff member in charge of responding to the event notification.

The audio input/output unit 450 is, for example, a speaker and a microphone, and enables voice communication between staff members with other mobile terminals 400 via the wireless communication unit 420.

The operation of the detection system 10 is explained.

Figure 11 is a flowchart showing the operation of the detection system 10. This flowchart can be executed by the control unit 210 of the server 200 according to a program. Note that if some or all of the functions shown in Figure 4 are executed by the imaging device 100, some or all of this flowchart can be executed by the control unit 110 of the imaging device 100 according to a program.

The control unit 210 acquires the image 600 by receiving it from the imaging device 100 (S101).

The control unit 210 detects the position information of the person 500 and the object 700 from the acquired image 600 using a trained model that has been trained to detect the position information of the person 500 and the object 700 from the image 600 including the person 500 and the object 700 (S102).

Based on the position information of the object 700, the control unit 210 uses the reference table to determine whether the object 700 can be moved (S103).

The control unit 210 determines whether the object 700 is likely to move (S104). That is, the control unit 201 determines whether the object 700 is a moving object.

When the control unit 210 determines that the object 700 is not a moving object (when it is determined that the object 700 is a stationary object) (S104: NO), it performs a correction process for stationary objects on the position information detected in step S102 (S105).

When the control unit 210 determines that the object 700 is a moving object, it performs a correction process for the moving object on the position information detected in step S102 (S106).

Second Embodiment
The second embodiment will be described. This embodiment differs from the first embodiment in the following respects. In the first embodiment, a reference table is referenced based on an attribute class included in the position information of the object 700 to determine whether the object 700 can move. On the other hand, in this embodiment, a likelihood of the object 700's ease of movement (hereinafter referred to as "movement likelihood") is calculated from the position information of the object 700, and whether the object 700 can move is determined based on the calculated likelihood. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted.

Figure 12 is a functional block diagram of the control unit 210. The control unit 210 functions as an acquisition unit 211, a detection unit 212, a switching unit 213, and a correction unit 214.

The switching unit 213 switches the correction process for the position information of the object 700 according to the position information of the object 700 detected by the detection unit 212. The switching unit 213 may include a movement likelihood calculation unit 2133, a moving object determination unit 2131, and a correction process switching unit 2132.

The movement likelihood calculation unit 2133 calculates the movement likelihood from the position information of the object 700 (i.e., the attribute class and position of the object 700). The movement likelihood calculation unit 2133 can detect the position information of the person 500 and the object 700 using a trained model that has been trained to calculate the movement likelihood from the position information of the object 70 (i.e., the attribute class and position of the object 70).

The moving object determination unit 2131 determines whether the object 700 is moving based on the movement likelihood calculated by the movement likelihood calculation unit 2133. That is, the moving object determination unit 2131 determines whether the object 700 is a moving object or a stationary object based on the movement likelihood of the object 700 detected as the object rectangle 620. Specifically, the moving object determination unit 2131 determines an object 700 whose movement likelihood is equal to or greater than a predetermined threshold as a moving object, and determines an object 700 whose movement likelihood is less than the predetermined threshold as a stationary object.

The embodiment provides the following advantages:

Using a trained model that has been trained to detect position information of people and objects from images that include people and objects, the correction process for the object's position information is switched according to the object's position information detected from an image of a specified area, and the object's position information is corrected by the correction process after the switch. This makes it possible to improve the detection accuracy of objects regardless of whether the object is moving or not.

Furthermore, based on the object's position information detected by the detection unit, it is determined whether the object is moving, and the correction process for the object's position information is switched depending on the determination result. This makes it possible to simply and efficiently improve the object detection accuracy.

Furthermore, the location information includes the attribute class and location of each of the person and object, and the attribute class of the person and object is identified based on the image, and the location of the person and object is identified based on the image. Then, based on the identified attribute class of the object, a reference table in which the relationship between the attribute class and whether or not it can be moved is used to identify whether or not the object can be moved. This makes it easier to identify whether or not an object can be moved.

Furthermore, the likelihood of an object's ease of movement is calculated from the object's position information, and whether or not the object can move is determined based on the calculated likelihood of the object's ease of movement. This makes it easier to determine whether or not an object can move.

Furthermore, when the process is switched to correcting the position information of a stationary object, a correction is made to remove erroneous detections of the object's position information when it is determined that there is contact between a person and an object. And when the process is switched to correcting the position information of a moving object, a correction is made to complement undetected portions of the object's position information when it is determined that there is contact between a person and an object. This makes it possible to improve the object detection accuracy more easily and effectively.

Furthermore, for stationary objects, among the objects determined to have come into contact with a person, a false detection is determined based on the position information of the object detected in a past image taken before the image in which the object's position information was detected, and a correction is made to delete the position information of the object determined to have been a false detection, and for moving objects, even if it is determined that there is contact between a person and the object, the position information of the object determined to have come into contact with a person is not deleted. This makes it possible to suppress false detection of stationary objects and suppress a decrease in the detection accuracy of moving objects.

Furthermore, if it is determined that there is contact between a person and an object, the system tracks the object determined to have been in contact with the person in a past image in which a specific area was captured prior to the image in which the position information was detected, and performs a correction to complement the position information of the object that cannot be confirmed by tracking. This improves the accuracy of detecting moving objects.

The system configuration described above is the main configuration described in order to explain the features of the above-mentioned embodiment, but is not limited to the above-mentioned configuration and can be modified in various ways within the scope of the claims. Furthermore, it does not exclude configurations that are equipped with general human and object detection systems.

The means and methods for performing various processes in the above-mentioned system can be realized by either a dedicated hardware circuit or a programmed computer. The above-mentioned programs may be provided by computer-readable recording media such as USB memory or DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the programs recorded on the computer-readable recording media are usually transferred to and stored in a storage unit such as a hard disk. The above-mentioned programs may be provided as standalone application software, or may be incorporated as a function into the software of a server or other device.

10 detection system,
100 imaging device,
110 control unit,
120 Communication department,
130 cameras,
200 servers,
210 control unit,
220 Communication department,
230 storage section,
300 communication network,
400 Mobile terminal,
500 people,
600 images,
610 person rectangle,
620 object rectangle,
700 objects,
710 beds,
720 chair.

Claims (15)

An acquisition unit that acquires an image of a predetermined area;
A detection unit that detects position information of the person and the object from the image using a trained model that has been trained to detect position information of the person and the object from the image including the person and the object, respectively;
a switching unit that switches a correction process for the position information of the object in response to the position information of the object detected by the detection unit;
a correction unit that corrects the position information of the object detected by the detection unit by the correction process after switching by the switching unit;
A correction system having The switching unit is
a moving object determination unit that determines whether the object is movable based on the position information of the object detected by the detection unit;
a correction process switching unit that switches the correction process of the position information of the object by the correction unit according to a determination result by the moving object determination unit;
The correction system of claim 1 , comprising: The location information includes attribute classes and locations of the person and the object, respectively;
The detection unit is
an attribute class identification unit that identifies the attribute classes of the person and the object based on the image;
a position identification unit that identifies the positions of the person and the object based on the image,
The correction system according to claim 2 , wherein the moving object determination unit determines whether or not the object is capable of moving based on the attribute class of the object identified by the attribute class identification unit, using a reference table in which a relationship between the attribute class and whether or not the object is capable of moving is set. the switching unit includes a movement likelihood calculation unit that calculates a likelihood of a movement of the object from the position information of the object,
The correction system according to claim 2 , wherein the moving object determination unit determines whether or not the object has moved based on the likelihood of the object calculated by the movement likelihood calculation unit. The correction unit is
a stationary object correction unit that performs a correction to remove a false detection of the position information of the object when it is determined that the person is in contact with the object, when the switching unit switches to the correction process of the position information of the object that is not moving; and
2. The correction system according to claim 1, further comprising: a moving object correction unit that, when the switching unit switches to the correction process for the position information of the moving object, performs a correction to complement non-detection of the position information of the object when it is determined that there is contact between the person and the object. the stationary object correction unit determines whether the object is detected as a false detection based on the position information of the object detected from the past image taken before the image in which the position information of the object is detected, among the objects determined to be in contact with the person, and performs a correction to delete the position information of the object determined to be a false detection;
The correction system according to claim 5 , wherein the moving object correction unit does not delete the position information of the object determined to be in contact with the person even when it is determined that the person and the object are in contact with each other. The correction system according to claim 5 or 6, wherein, when the moving object correction unit determines that the person and the object are in contact with each other, the moving object correction unit tracks the object determined to be in contact with the person in the past image in which the specified area was captured before the image in which the position information was detected, and performs correction to complement the position information of the object that cannot be confirmed by tracking. A step (a) of acquiring an image of a predetermined area;
(b) detecting position information of the person and the object from the image using a trained model trained to detect position information of the person and the object, respectively, from the image including the person and the object;
a step (c) of switching a correction process for the position information of the object in response to the position information of the object detected in the step (b);
a step (d) of correcting the position information of the object detected in the step (b) by the correction process after switching in the step (c);
The correction method has the following features: The step (c)
A step (c1) of determining whether or not the object can be moved based on the position information of the object detected in the step (b);
a step (c2) of switching the correction process of the position information of the object in the step (d) according to a result of the determination in the step (c1);
The method of claim 8 , further comprising: The location information includes attribute classes and locations of the person and the object, respectively;
The step (b)
(b1) identifying the attribute classes of the people and the objects based on the images;
(b2) determining the location of the person and the object based on the image,
10. The correction method according to claim 9, wherein the step (c1) determines whether or not the object can be moved based on the attribute class of the object identified in the step (b1) by using a reference table in which a relationship between the attribute class and whether or not the object can be moved is set. The step (c) includes a step (c3) of calculating a likelihood of the object's ease of movement from the position information of the object;
The correction method according to claim 9 , wherein the step (c1) specifies whether or not the object is capable of moving based on the likelihood of the object calculated in the step (c3). The step (d)
In the step (c), when the process is switched to the correction process of the position information of the object that does not move, when it is determined that the person is in contact with the object, a correction step (d1) is performed to remove a false detection of the position information of the object;
9. The correction method according to claim 8, further comprising: a step (d2) of performing a correction to complement non-detection of the position information of the object when it is determined that there is contact between the person and the object when the step (c) is switched to the correction process of the position information of the moving object. In the step (d1), among the objects determined to have come into contact with the person, a false detection is determined based on the position information of the object detected in the past image taken before the image in which the position information of the object is detected, and a correction is performed to delete the position information of the object determined to have been a false detection;
The correction method according to claim 12 , wherein in the step (d2), even if it is determined that there is contact between the person and the object, the position information of the object determined to be in contact with the person is not deleted. The correction method according to claim 12 or 13, wherein, in step (d2), if it is determined that there is contact between the person and the object, the object determined to have been in contact with the person is tracked in the past image in which the predetermined area was captured before the image in which the position information was detected, and a correction is made to complement the position information of the object that cannot be confirmed by tracking. A correction program for causing a computer to execute the correction method according to any one of claims 8 to 14.