JP6823686B2 - Object detection device and method and storage medium - Google Patents

Description

The present invention relates to image processing, in particular, object detection processing, for example.

In surveillance systems, people are generally the primary monitoring target. In general, people routinely wear, hold, and use some objects (eg, glasses, bags, suitcases, wheelchairs, etc.), so these objects monitor people in an auxiliary way. Used to do. As used herein, these objects are referred to, for example, as attachments of related persons. In the monitoring process, the person recognition process is generally the main processing operation, and the most necessary basic information used for the person recognition process is information such as the position of the person in the moving image and related objects. Therefore, the ability to detect people and related objects from video / images with high recall has a direct impact on the accuracy of the person recognition process. In the present specification, the person recognition process means, for example, attribute recognition of a person, verification of a person (verification of the ID of a target person), search of a person image, recognition or analysis of a person's behavior / behavior (for example, the target person Includes whether you have any object, and an analysis of the behavior between the subject and other objects).

Non-Patent Document 1 discloses an exemplary object detection technique for detecting a person and related objects from a video / image with a high recall rate. Generally, it is as follows. First, a neural network is used to extract various levels of features from the input image. For example, extract low-level features for small-scale objects, medium-level features for medium-scale objects, and high-level features for large-scale objects. Next, using the corresponding pre-generated candidate region generation network, the relevant information of the candidate region of the object (for example, the position of the candidate region, the score of the candidate region, and the characteristics of the candidate region) is extracted from the characteristics of each level. To do.

In the object detection technique (for example, the above-exemplified technique), generally, only the candidate area having a score not lower than the preset threshold value or only the candidate area having the score ranked in the upper N is the final output. Become. In other words, the candidate area of the final output will be regarded as the area of objects (people and objects) that can be detected from the image. However, objects obstructed by other objects in the image, such as suitcases obstructed by stakeholders, wheelchairs obstructed by seated persons, bags affected by shadows placed on the ground, etc. For objects that are affected by lighting, the features of those objects cannot be completely extracted from the image. Therefore, even if the candidates of those objects can be detected from the image by using the above-exemplified technique, the score of the candidate area of those objects is low, so that the score of the candidate area of those objects is lower than the preset threshold value. Alternatively, the scores of the candidate areas of those objects will not be ranked in the top N. As a result, the candidate area for such an object is not the final output, the object from the image cannot be detected, and ultimately the high recall of that object is affected.

"Feature Pyramid Networks for Object Detection" (Tsung-Yi Lin, Piotr Dollar, Ross Girshick, Kaiming He, Bharath Hariharan, Serge Belongie, CVPR 2017).

In view of the related art, the present invention solves at least one of the above problems.

The object detection device provided in one embodiment of the present invention is an object detection device, which is an extraction means for extracting features from an image and a space showing a specific positional relationship between two related objects belonging to a predetermined category. An acquisition means for acquiring the spatial relationship between the extracted features based on a pre-generated model in which the target relationship is learned, and a detection means for detecting an object candidate region from the image based on the extracted features. And the score of the candidate area indicating the probability that an object of a predetermined category exists in the area of the image based on the spatial relationship between the features included in the two candidate areas of the detected candidate areas. It is characterized by having an update means for updating and a determination means for determining an area of an object existing in the image based on the score of the updated candidate area .

The object detection method provided in another embodiment of the present invention is a pre-generation that learns an extraction step of extracting features from an image and a spatial relationship indicating a specific positional relationship between two related objects belonging to a predetermined category. An acquisition step of acquiring the spatial relationship between the extracted features based on the model, a detection step of detecting a candidate region of an object from the image based on the extracted features, and the detected candidate. An update step of updating the score of a candidate region indicating the probability that an object of a predetermined category exists in the region of the image based on the spatial relationship between the features included in the two candidate regions of the region, and the above. It is characterized by having a determination step of determining an area of an object existing in the image based on the updated candidate area score .

The object detection device provided in a further embodiment of the present invention shows a feature extraction means for extracting features from an image of the current video frame in a video and a specific positional relationship between two related objects belonging to a predetermined category. An acquisition means for acquiring the spatial relationship between the extracted features based on a pre-generated model in which the spatial relationship is learned, and detection for detecting a candidate region of an object from the image based on the extracted features. A score of a candidate region indicating the probability that an object of a predetermined category exists in the region of the image based on the means and the spatial relationship between the features included in the two candidate regions of the detected candidate regions. It is characterized by having an update means for updating the image and a determination means for determining an area of an object existing in the image based on the score of the updated candidate area .

In yet another embodiment of the invention, a storage medium for storing instructions that can be executed by a processor is provided to enable the object detection method described above.

In the present invention, just as the spatial relationship between individual feature points regulates the detection of an area of an object, that spatial relationship is used when detecting the area of an object. This makes it possible to better detect related objects. Since related objects are generally more useful for monitoring people during the monitoring process, not only the recall of object detection, but also the effect of monitoring people can be improved according to the present invention.

Further other features and effects of the present invention will be apparent from the embodiments described below with reference to the accompanying drawings.

The accompanying drawings are incorporated herein by reference and constitute a portion thereof, and illustrate embodiments of the present invention in order to explain the principles of the present invention.
It is a block diagram which shows the hardware structure which can carry out the technique which concerns on embodiment of this invention. It is a block diagram which shows the structure of the object detection apparatus which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the detection part 230 shown in FIG. It is a figure which shows the flowchart of the object detection which concerns on 1st Embodiment of this invention. It is a schematic block diagram of the pre-generative model used in the flowchart of FIG. It is a figure which shows the flowchart of the detection step shown in FIG. 4 which concerns on 1st Embodiment of this invention. It is a figure which shows the flowchart of the rank determination step shown in FIG. 6 which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of the object detection apparatus which concerns on 2nd Embodiment of this invention. It is a figure which shows the flowchart of the generation method which generates the model applicable to this invention.

Embodiments suitable for the present invention will be described in detail below with reference to the accompanying drawings. It should be noted that the following description is merely exemplary and exemplary in nature and is not intended to limit the invention and its uses or uses. The relative arrangement of components and steps, numerical expressions, numerical values, and the like described in the embodiments do not limit the scope of the present invention unless otherwise specified. Moreover, techniques, methods, and devices known to those of skill in the art may not be discussed in detail, but should be part of the specification where appropriate.

In the drawings, the same reference numbers and characters indicate the same items. Therefore, it should be noted that once an item is defined in one figure, it does not need to be discussed in the figures below.

The present inventor has found that in the actual scene of object detection, there are generally several objects having a specific relationship (particularly spatial relationship) with each other, and these objects are generally referred to as related objects. .. Moreover, on the other hand, cases of interaction (eg, being blocked from each other) are likely to occur between related objects. On the other hand, in the monitoring process, related objects are useful for monitoring people. Thus, in the object detection process, the inventor presents this particular relationship (especially spatial relationship) that exists between related objects, even if some related objects are shielded, for example. We have found that it is possible to constrain the detection of the area of the object. As a result, the recall rate of object detection can be improved, and the effect of monitoring a person can be enhanced. In the present invention, a person (for example, a subject) and a person are wearing / holding / using, such as a woman and a suitcase pulled by the woman, a man and a wheelchair on which the man is sitting, etc. Objects that are present can be considered as related objects. A woman and her child, as well as two overlapping objects located in front and behind, can also be considered as related objects. Adjacent objects (eg, the target object and other objects next to it), such as a suitcase and a bag placed on it, a shadow of a person and a bag partially or entirely covered by that shadow, etc. Can be considered a related object. However, it is clear that it does not have to be limited to this.

In the present invention, the spatial relationship existing between related objects represents a spatial constraint between those related objects. For example, in two related objects, the spatial relationship between them (eg, the spatial constraints between the regions corresponding to the two related objects) includes at least some of the following restrictions:
-Relative positional relationship between two objects (for example, directional relationship, distance relationship, etc.)
For example, in the case of a computer placed on a desk, the directional relationship between the computer and the desk is "on the desk". For example, in the case of a person / animal on the lawn, the directional relationship between the person / animal and the lawn is “on the grass”. For example, in the case of a woman who holds the child's hand and guides her to walk, the distance relationship between the woman and the child is “adjacent to, close to”. However, it is clear that it does not have to be limited to this.
-Topological relationship between two objects (for example, overlapping relationship, inclusion relationship, adjacency relationship, etc.)
For example, in the case of a man sitting in a wheelchair, the phase relationship between the man and the wheelchair is "overlapping". For example, in the case of a woman who has a child, the phase relationship between the woman and the child is an "inclusion relationship". In the case of a woman with a parasol, the phase relationship between the woman and the parasol is "adjacent". However, it is clear that it does not have to be limited to this.
-Relative shape relationship between two objects
For example, like a man sitting in a wheelchair, the spatial regulation between a person and a wheelchair also becomes a "relative shape relationship". However, it is clear that it does not have to be limited to this.

The present invention has been proposed in view of the above discussion results, and will be described in detail below with reference to the accompanying drawings.

[Hardware configuration]
First, a hardware configuration capable of realizing the techniques described below will be described with reference to FIG.

The hardware configuration 100 includes, for example, a central processing unit (CPU) 110, a random access memory (RAM) 120, a read-only memory (ROM) 130, a hard disk 140, an input device 150, an output device 160, a network interface 170, and a system. Includes bus 180. Further, the hardware configuration 100 may be mounted on, for example, a camera, a video camera, a personal digital assistant (PDA), a smartphone, a tablet, a laptop, a desktop computer, or any other suitable electronic device.

The object detection process of the present invention in one implementation method is configured by hardware or firmware and functions as a module or component of hardware configuration 100. For example, the object detection device 200 described in detail below with reference to FIG. 2 and the object detection device 800 described in detail below with reference to FIG. 8 are used as modules or components of the hardware configuration 100. The object. The object detection process of the present invention in another mounting method is composed of software stored in the ROM 130 or the hard disk 140 and executed by the CPU 110. For example, the process 400 described in detail with reference to FIG. 4 and the process 900 described in detail with reference to FIG. 90 are used as programs stored in the ROM 130 or the hard disk 140.

The CPU 110 is any suitable programmable control device, such as a processor, that can execute various functions described below by executing various application programs stored in the ROM 130 or the hard disk 140 (memory, etc.). it can. The RAM 120 temporarily stores programs and data loaded from the ROM 130 or the hard disk 140, and also performs various processes executed by the CPU 110 (for example, the techniques of FIGS. 3 to 7 and 9 described later) and other uses. It is also used as a space to perform possible functions. The hard disk 140 stores various information such as an operating system (OS), various applications, control programs, videos, images, pre-generated models, and predefined data (for example, threshold groups (THs)). Store.

In one embodiment, the input device 150 allows the user to exchange information with the hardware configuration 100. In one embodiment, the user can input images / videos / data through the input device 150. In another embodiment, the user can activate the corresponding process of the invention through the input device 150. Further, the input device 150 can adopt various forms such as a button, a keyboard, or a touch screen. In another embodiment, the input device 150 is utilized to receive images / videos output from specialized electronic devices such as digital cameras, video cameras, and / or network cameras.

In one embodiment, the output device 160 is used to display detection results (positions, scores, features, etc. of detected areas of an object) to the user. Further, the output device 160 can adopt various forms such as a CRT (Cathode Ray Tube) and a liquid crystal display. In another embodiment, the output device 160 uses the detection result for subsequent processing such as person recognition processing (eg, person attribute recognition, person matching, person image retrieval, and person behavior / behavior recognition or analysis). Used to output.

The network interface 170 provides an interface for connecting the hardware configuration 100 to the network. For example, the hardware configuration 100 can perform data communication with other electronic devices connected via the network via the network interface 170. Alternatively, the hardware configuration 100 may include a wireless interface for performing wireless data communication. The system bus 180 provides a data transfer path for data transfer between the CPU 110, the RAM 120, the ROM 130, the hard disk 140, the input device 150, the output device 160, the network interface 170, and the like. The system bus 180 is called a bus, but is not limited to a specific data transmission technique.

The above hardware configuration 100 is merely an example and is not intended to limit the present invention and its use or use. Further, for brevity, only one hardware configuration is shown in FIG. However, multiple hardware configurations can be used as needed.

[Object detection]
Next, the object detection in the present invention will be described with reference to FIGS. 2 to 8.

FIG. 2 is a block diagram showing the configuration of the object detection device 200 according to the first embodiment of the present invention. Here, some or all of the modules shown in FIG. 2 may be realized by dedicated hardware. As shown in FIG. 2, the object detection device 200 includes an extraction unit 210, a determination unit 220, and a detection unit 230.

First, the input device 150 shown in FIG. 1 receives an image output from a specific electronic device (for example, a video camera or the like). Next, the input device 150 transfers the received image to the object detection device 200 via the system bus 180.

Then, as shown in FIG. 2, the extraction unit 210 extracts features from the received image. Here, the extraction unit 210 uses an existing feature extraction algorithm, for example, a local binary pattern (LBP) algorithm, a Gabor algorithm, a scale-invariant feature conversion (SIFT) algorithm, a neural network (NN) algorithm, or the like from an image. Features can be extracted. Here, the extracted features may be, for example, gradient features, edge features, apparent features, semantic features, and the like in the image.

The determination unit 220 determines the spatial relationship between the individual feature points in the image based on the extracted features. Here, the feature points are the extracted feature points. As for the spatial relationship between any two feature points, depending on whether the two feature points belong to the same region or different regions, the spatial correlation between the feature points is "positional correlation within the same region (" That is, it is categorized into "positional relationship of intra-category)" and "positional relationship between different regions (that is, positional relationship of inter-category)". Further, the spatial relationship between the determined feature points has a corresponding spatial relationship value. Here, in the case of a spatial relationship between two feature points, the corresponding spatial relationship value represents the probability that the two feature points belong to the spatial relationship.

In one embodiment, the determination unit 220 may determine the spatial relationship between feature points based on the extracted features according to a predefined rule. The predefined rules here may be stored in a corresponding recording device, such as the storage device 240 shown in FIG. Here, the storage device 240 may be the ROM 130 or the hard disk 140 shown in FIG. 1, or may be a server or an external storage device connected to the object detection device 200 via a network (not shown). As described above, in the embodiment, the determination unit 220 first acquires a predefined rule from the recording device 240, and then performs the corresponding determination process of the spatial relationship.

In another embodiment, in order to conveniently determine the spatial relationships in various scenes, the model used to determine the spatial relationships between feature points (ie, the pre-generated model) is spatial. The relationship is to be trained / pre-generated according to the labeled training sample and stored in the corresponding storage device (eg, storage device 240). Here, the method of generating the pre-generated model will be described in detail later with reference to FIG. On the other hand, the determination unit 220 determines the spatial relationship between the feature points based on the extracted features by using the pre-generated model.

Further, in order to improve the processing speed of object detection, the above pre-generated model is generated as referred to in FIG. Here, the pre-generative model includes a part for feature extraction and a part for determining a spatial relationship. Alternatively, the extraction unit 210 may extract features from the image using a pre-generated model. In this case, on the one hand, the extraction unit 210 acquires a pre-generated model from the storage device 20, and on the other hand, the extraction unit 210 extracts features from the image using the pre-generated model. Further, in this case, the determination unit 220 may use the pre-generated model acquired from the storage device 240 as it is by the extraction unit 210 without particularly acquiring the corresponding pre-generated model from the storage unit 240.

Returning to the description of FIG. After the spatial relationship between the individual feature points is determined, the detection unit 230 detects the area of the object in the image based on the determined spatial relationship. Here, the object is preferably a related object in the image, and the area of the detected object includes, for example, the position of the area, the score of the area, and the features included by the area. Here, the score of one area indicates the probability that the area belongs to an object of a certain category, and the features included in one area are the features extracted by the extraction unit 210 belonging to the area. Is.

In one embodiment, the detection unit 230 may detect the area of the object from the image by directly utilizing the determined spatial relationship. Specifically, first, the detection unit 230 clusters individual feature points based on the determined spatial relationship. Here, the clustering result can be regarded as one area, and the spatial relationship between the feature points in each clustering result belongs to the “intra-category spatial relationship” described above. Then, the detection unit 230 uses the corresponding area as the final detection area of the object based on the spatial relationship between the feature points belonging to different clustering results (that is, the above-mentioned “intercategory spatial relationship”). judge. Here, the clustering result in which the distance between each other is less than a predetermined threshold value (for example, TH1) can be regarded as the final detection area of the object, and the clustering result overlapping with each other can be regarded as the final detection, for example. ..

In another embodiment, the detection unit 230 can preferentially output the region of the more relevant object and make the position of the region of the detected object more accurate, as shown in FIG. , Candidate region detection unit 231 and ranking determination unit 232 may be included.

As shown in FIG. 3, the candidate area detection unit 231 detects the candidate area of the object from the image based on the features detected by the detection unit 210. Here, the candidate area detection unit 231 uses an existing area detection algorithm, for example, a selective search algorithm, an EdgeBoxes algorithm, an objectness algorithm, or the like to make a candidate from an image. The area can be detected. Further, as described above, the pre-generated model shown above may include a part for extracting features and a part for determining spatial relationships, and in order to further increase the processing speed of object detection, the pre-generated model is shown in the figure. It may include a portion to be detected in the candidate area of the object when it is generated as shown in 9. Therefore, the candidate area detection unit 231 may detect the candidate area of the object from the image based on the features extracted by using the pre-generated model. In this case, the candidate area detection unit 231 may detect the candidate area of the object from the image by using the pre-generated model acquired by the detection unit 210 from the storage device 240. Here, the detected candidate area of the object also includes, for example, the position of the candidate area, the score of the candidate area, and the feature including the candidate area. Here, the score of one candidate area indicates the probability that the candidate area belongs to an object of a certain category. For example, the score of the candidate area may be obtained by categorizing the candidate areas.

Next, as shown in FIG. 3, after the candidate region is detected from the image, the rank determination unit 232 determines the rank of the detected candidate region based on the spatial relationship determined by the determination unit 220. The candidate area after the ranking determination is used as the object detection area.

Further, the pre-generated model described above may include a portion that directly detects the region of the object when the object is created as shown in FIG. 9, in addition to the portion that detects the candidate region of the object. .. Therefore, in other embodiments, the detector 230 may detect a region of an object in the image based on a spatial relationship determined directly using a pre-generated model. In this case, the detection unit 230 may detect the area of the object from the image by using the pre-generated model acquired by the extraction unit 210 from the storage device 240.

Returning to FIG. 2, after the area of the object is detected from the image, the detection unit 230 sets the area of the object having a score equal to or higher than the predefined value as the final detection result, or ranks the top N objects. The region having the ranking score is taken as the final detection result, and the final detection result is transferred to the output device 160 via the system bus 180 shown in FIG. 1, and the final detection result of the object (for example, the region) is transferred. Position, score, feature) is displayed to the user, or subsequent processing such as person recognition processing (for example, person attribute recognition, person matching, person image search, recognition or analysis of person behavior / behavior, etc.) The detection area of the object is output to.

The flowchart 400 shown in FIG. 4 is a corresponding process of the object detection device 200 shown in FIG. The following is a description performed by the extraction unit 210, the determination unit 220, and the detection unit 230 that perform the corresponding processing using the pre-generated model. Here, the schematic configuration of the pre-generated model used in the processing is as shown in FIG. 5, for example. However, it is clear that there is no need to be limited to this.

As shown in FIG. 4, in the extraction step S410, the extraction unit 210 acquires a pre-generated model from the storage device 240, and receives an image using the acquired pre-generated model (particularly, a portion in which the feature is extracted). Features are extracted from.

In the determination step S420, the determination unit 220 determines the spatial relationship between the feature points based on the extracted features by using the pre-generated model (particularly the portion for determining the spatial relationship in the model).

In the detection step S430, the detection unit 230 detects the area of the object from the image by using the acquired pre-generated model. The object here is preferably an associated object in the image. As described above, in order to preferentially output the region of the more relevant object and to make the position of the detected region of the object more accurate, in one execution phase, the detection unit 230 According to 6, the area of the object in the image is detected.

As shown in FIG. 6, in the candidate region detection step S431, the candidate region detection unit 231 was extracted by the extraction unit 210 using a pre-generated model (particularly, a portion for detecting the candidate region in the candidate region). Detects a candidate area for an object from an image based on its characteristics. In the rank determination step S432, the rank determination unit 232 determines the rank of the candidate area based on the spatial relationship determined by the determination unit 220, and the candidate area after the rank determination is used as the final detection area of the object. To do. In one embodiment, the ranking determination unit 232 determines the ranking of the candidate region according to FIG.

As shown in FIG. 7, in step S4321, the ranking determination unit 232 determines the spatial relationship between the candidate regions detected by the candidate region detection unit 231. Specifically, for any two candidate regions, the ranking determination unit 232 determines the spatial relationship between the two candidate regions based on the mutual spatial relationship between the feature points included in the two candidate regions. judge. Here, as long as there are two corresponding feature points having a specific spatial relationship within the two candidate regions, the two candidate regions can be considered to have a specific spatial relationship.

In one embodiment, for any two candidate regions, the ranking determination unit 232 determines the spatial relationship between any two feature points between the two candidate regions and the spatial relationship between the two candidate regions. Judge as. Preferably, for example, the spatial relationship between the two feature points at the central position of these two candidate regions may be determined as the spatial relationship between the two candidate regions. Here, the spatial relationship value of the spatial relationship between the two candidate regions is regarded as the spatial relationship value of the spatial relationship between the two candidate regions. For example, the spatial relationship between two feature points having the largest spatial relationship value is determined as the spatial relationship between the two candidate regions. Here, the maximum spatial relationship value is regarded as the spatial relationship value of the spatial relationship between the two candidates.

In another embodiment, the ranking determination unit 232 uses all the spatial relationships between the feature points existing between the two candidate regions for any two candidate regions, and uses the space between the two candidate regions. Determine the relationship. Preferably, for example, on the one hand, the spatial relationship between the feature points is voted on, and the spatial relationship with the most votes is determined as the spatial relationship between the two candidate regions. On the other hand, all the spatial relationship values belonging to the spatial relationship with the largest number of votes are averaged, weighted, or maximized, and the obtained value is the spatial relationship value of the spatial relationship between the two candidate regions. Is considered.

Return to FIG. In step S4322, the ranking determination unit 232 updates the score of the candidate area based on the spatial relationship value of the spatial relationship after the determination between the candidate areas. In one embodiment, the rank determination unit 232 may update the score of the candidate region by a calculation operation between matrices. Specifically, for example, a matrix consisting of the spatial relation values of the determined spatial relations between the candidate regions and a matrix consisting of the scores of the candidate regions are mathematically operated (for example, matrix multiplication). Then, the result obtained after the calculation is used as the score after updating the candidate area. In another embodiment, when the target object to be detected (for example, the target person) is specified, the ranking determination unit 232 is a candidate area of an object (for example, an accessory of the target person) related to the target object. All you have to do is update your score. In particular. For example, one related object having the maximum spatial relationship value is first determined from the related object having a spatial relationship to the target object, and the maximum spatial relationship value is determined to update the score of the candidate area. It is superimposed on the score of the candidate area of the related object.

Further, in order to narrow the determination range of the spatial relationship between the candidate regions in order to improve the processing speed, step S4320 determines the spatial relationship between the candidate regions before S4321 as shown in FIG. 7. It may be included in the previous). As shown in FIG. 7, in step S4320, the ranking determination unit 232 acquires the corresponding auxiliary information. This auxiliary information is, for example, information about a specific detection task, information about a specific detection scene, and the like.

As the specific detection task, a target object to be detected (for example, a target person) is generally specified, that is, position information and category information of the target object are generally given. Furthermore, in general, it is desirable that the preferentially detected object is another object related to the target object (for example, an accessory around the target person).

Therefore, with respect to the specific detection task, the auxiliary information acquired by the ranking determination unit 232 is, for example, the position information and the category information of at least one target object. Further, on the other hand, since the category information of the target object is known, the rank determination unit 232 may clearly determine the type of spatial relationship existing between the target object and other objects. For example, when the target object is a target person, the spatial relationships between the target object and other objects are "spatial relationship between one person and another person" and "between one person and another object". It is only the "spatial relationship between" and not the "spatial relationship between an object and an object". On the other hand, since the position information of the target object is known, the ranking determination unit 232 roughly determines which candidate area should be determined without determining the spatial relationship between all the candidate areas. Can be defined in. Therefore, when the ranking determination unit 232 determines the spatial relationship between the candidate regions in step S4321, only the spatial relationship between the specific candidate regions needs to be determined, thereby improving the processing speed. Can be done.

Further, regarding the specific detection task, when the target object is the target person, the auxiliary information obtained by the ranking determination unit 232 is, for example, the joint point information of at least one target object (that is, the target person). is there. Here, the joint point information of the target person can be acquired by using manual labeling or a joint point detection method. Further, the ranking determination unit 232 may acquire an operation corresponding to the spatial relationship between the target person and the person / object associated with the target person by categorizing or recognizing the joint points of the target person. For example, when the target person is pulling the suitcase, the action corresponding to the spatial relationship between the target person and the suitcase is "pulling". Therefore, when the ranking determination unit 232 determines a specific spatial relationship between the candidate regions in S4321, only the specific spatial relationship between the specific candidate regions is determined, and further, the identification corresponding to the specific movement is specified. Only the spatial relationship of is determined, which can further improve the processing speed.

For a particular detection scene, there is generally a particular spatial relationship between the scene and the objects in the scene (eg, people, animals, etc.). For example, in grass / prairie, flying animals (such as birds) are generally flying in the air and are unlikely to be walking on the ground. Also, humans or walking animals (such as sheep) generally walk on the ground and rarely fly in the air. Therefore, for the specific detection scene, the auxiliary information acquired by the ranking determination unit 232 is, for example, scene information (that is, background information of the input image). Further, the ranking determination unit 232 may specifically determine a specific spatial relationship between a specific object and the scene according to the scene information. Therefore, when the ranking determination unit 232 determines the spatial relationship between the candidate regions in S4321, it is possible that only a specific spatial relationship is determined without determining all the spatial relationships. As a result, the processing speed can be improved.

Return to FIG. When the area of the object is detected from the image, the detection unit 230 sets the detection area of the object having a score equal to or higher than the threshold value or assigned to the top N as the final detection result, and is shown in FIG. The detection result is sent to the output device 160 via the system bus 180, and the final object detection result (for example, the position, score, and feature of the area) is displayed to the user, or the person recognition process (for example, person attribute recognition). , Person matching, person image search, recognition or analysis of person behavior / behavior, etc.), and output the object detection area for subsequent processing. For example, regarding the recognition or analysis of the behavior / behavior of a person, the area of the object detected by the object detection device 200 shown in FIG. 2 is preferably the target person and / worn by the target person. The area of the gripped / used appendage and another person in close proximity to the target person, whereby the behavior / behavior between the target person and the accessory or adjacent person is due to the spatial relationship between the areas. Can be directly recognized or analyzed. Also, for example, in the case of a target person and another person adjacent to the target person, if the spatial relationship between the regions is an "inclusion relationship", the behavior of the target person is, for example, "Hugging". It can be inferred as "holding". Also, regarding the target person and the attachments of the target person, if the spatial relationship between the areas is an "adjacency relationship", it is inferred that the behavior of the target person is, for example, "grasping". it can. Also, in a person image search within a segment of a video, the spatial relationship between the target person and the attachments of the target person generally does not change much. Therefore, it may only be determined whether the target persons in the video segment having the spatial relationship between the detected regions are similar. For example, it is only determined if the target person pulling the suitcase in the video segment is similar.

According to the first embodiment of the present invention, since the spatial relationship between the individual feature points in the image is utilized when the object area is detected, these spatial relationships regulate the area detection of the object. Therefore, it makes it possible to better detect related objects. Since related objects are generally useful for monitoring people during the monitoring process, not only the recall of object detection but also the effect of monitoring people can be improved according to the present invention.

In the first embodiment of the present invention, the object detection operation is performed in one image. Since the spatial relationships between objects generally do not change significantly within a short duration, the present invention can also be used to perform object detection within a segment of video. FIG. 8 is a block diagram showing the configuration of the object detection device 800 according to the second embodiment of the present invention. Here, some or all the modules shown in FIG. 8 may be realized by dedicated hardware. As shown in FIG. 8, the object detection device 800 includes a feature extraction unit 810, a candidate area detection unit 820, a spatial relationship determination unit 830, and a ranking determination unit 840.

First, the input device 150 shown in FIG. 1 receives a segment of video output or input by a particular electronic device (eg, a video camera, etc.). Next, the input device 150 transfers the received video to the object detection device 800 via the system bus 180.

Next, as shown in FIG. 8, the feature extraction unit 810 extracts features from the current video frame in the received video. Since the operation of the feature extraction unit 810 is the same as that of the extraction unit 210 shown in FIG. 2, the description thereof will not be repeated here.

The candidate area detection unit 820 detects the candidate area of the object from the current video frame based on the features extracted by the feature extraction unit 810. Since the operation of the candidate area detection unit 820 is the same as that of the candidate area detection unit 231 shown in FIG. 3, the description thereof will not be described here.

The spatial relationship determination unit 830 determines the spatial relationship between the candidate regions detected by the candidate region detection unit 820 based on the detection result of the previous frame with respect to the current video frame. Here, the detection result of the frame before that with respect to the current video frame may be obtained according to the first embodiment of the present invention. In one embodiment, for example, the spatial relationship between the regions of an object detected from any of the previous video frames is the spatial relationship between the candidate regions within the current video frame. In other embodiments, for example, a wide range of results of spatial relationships between regions of objects detected from the N video frames of the previous video frame (eg, obtained by performing mathematical operations such as weighting or averaging). ) Is the spatial relationship between the candidate areas in the current video frame.

The rank determination unit 840 determines the rank of the candidate area detected by the candidate area detection unit 820 based on the spatial relationship between the candidate areas determined by the spatial relationship determination unit 830.
The candidate area after the ranking determination is set as the object area.

When the area of the object is detected from the current video frame, the ranking determination unit 840 sets the area of the object having a score equal to or higher than the predefined threshold as the final detection result, or ranks the top N objects. The determined area is taken as the final detection result, the final detection result is transferred to the output device 160 via the system bus 180 of FIG. 1, and the area of the finally detected object of the current video frame (for example, the area). (Position, score, feature) is displayed to the user, or an object is used for subsequent processing such as person recognition processing (for example, person attribute recognition, person matching, person image search, recognition or analysis of person behavior / behavior, etc.) Output the detection area of.

As an application example of the second embodiment of the present invention, the object detection device 800 shown in FIG. 8 may be used to track a person in a video. Specifically, in the current video frame in the video, if the person in the current video frame can be successfully tracked using a commonly used person tracking device, the person in the current video frame is generally used. Detected using a person tracking device. Also, if the person in the current video frame cannot be successfully tracked using a commonly used person tracking device, the person in the current video frame may be detected using the object detection device 800 shown in FIG. .. This achieves tracking of the person throughout the video.

[Model generation]
As described in the first embodiment of the present invention, the model applicable to the present invention (ie, the pre-generated model) is pre-learned / pre-generated according to the learning of the spatially related sample. To. Here, as described above, in order to improve the processing speed of the present invention, for example, as shown in FIG. 5, the pre-generative model applied to the present invention determines, for example, a part for feature extraction and a spatial relationship. A part and a part for detecting a region / candidate region are included. In the present invention, the pre-learning model may be generated based on training of a sample labeled with spatial relationships using a deep learning method (eg, neural network method). Here, each part of the pre-generated model in the present invention is composed of a network of a plurality of layers. For example, a part for extracting features is composed of an N layer network, and a part for determining a spatial relationship is composed of an M layer network. The portion for detecting the region / candidate region may be configured by a T-layer network. Here, N, M, and T are natural numbers, and the values they indicate may be the same or different.

In one embodiment, in order to shorten the time required to generate the pre-generated model, the part of the model for extracting features, the part for determining the spatial relationship, and the part for detecting the region / candidate region are backed up. Updated at the same time by propagation means. FIG. 9 is a flowchart 900 schematically showing a generation method for generating a model applicable to the present invention. In the flowchart 900 shown in FIG. 9, an example of using a neural network for generating a model applicable to the present invention will be described. However, it is clear that it does not have to be limited to this. Here, the generation method according to FIG. 9 can also be executed by the hardware configuration 100 shown in FIG.

As shown in FIG. 9, first, the CPU 110 shown in FIG. 1 acquires an initial neural network and a plurality of training samples preset by the input device 150. Here, each training sample is labeled by spatial relationship, region position, and object category. The spatial relationships labeled in the training sample are, for example, "with / without spatial relationship", "which category the spatial relationship belongs to", and the like.

Next, in step S910, on the other hand, the CPU 110 uses the training sample as a part of the current neural network for extracting features (for example, an initial neural network) and a part of the current neural network for determining the spatial relationship. Pass it through (initial neural network) to get the spatial relationships that exist in the training sample. On the other hand, the CPU 110 determines the loss between the obtained spatial relationship and the sample spatial relationship (for example, the first loss Loss1). Here, the sample spatial relationship may be obtained according to the spatial relationship labeled in the training sample. The first loss Loss1 represents the error between the spatial relationship value of the predicted spatial relationship obtained by using the current neural network and the spatial relationship value of the sample spatial relationship (that is, the real spatial relationship value). The error is measured, for example, by distance. For example, the first loss Loss 1 can be obtained by the following equation (1).

Here, j represents the number of the spatial relationship category to which the object in the training sample belongs, C represents the maximum number of the spatial relationship categories, and y _j represents the real spatial relationship value of the object of the spatial relationship category j. , P _j indicate the predicted spatial category value of the object of the spatial relationship category j.

In step S920, on the other hand, the CPU 110 passes the training sample through all current neural networks (eg, the initial neural network) to obtain the area / weather area position of the object and the object category of the object. That is, the CPU 110 passes the training sample through the current neural network of the part for extracting features, the current neural network of the part for determining the spatial relationship, and the neural network of the part for detecting the area / candidate area of the object. To obtain the area / candidate area position of the object and the object category of the object. On the other hand, with respect to the area / candidate area position of the obtained object, the CPU 110 determines a loss between the position of the area / candidate area of the obtained object and the sample area position (for example, a second loss Loss2). Here, the sample region location can be obtained according to the location of the region labeled on the training sample. Here, the second loss Loss2 represents an error between the predicted region / candidate region position and the sample region position obtained by using the current neural network, and the error is measured by the distance. For example, the second loss Loss 2 is obtained by the following equations (2) and (3).

Here, smartphone _L1 (x) represents the difference between the area / candidate area position and the real area position of the object, x represents the horizontal coordinates of the upper left corner of the position of the object area / candidate area, and y represents the object. Represents the vertical coordinates of the upper left corner of the position of the area / candidate area, w represents the width of the object area / candidate area, h represents the height of the object area / candidate area, and t ⁿ _i represents the object category. Represents the area / candidate area position of an object of ⁿ , and v ⁿ _i represents the real area position of an object of object category n.

For the object category of the obtained object, the CPU 119 determines a loss between the object category of the obtained object and the sample object category (for example, a second loss Loss 3). Here, the sample object category can be obtained according to the object category labeled in the training sample. Further, the third loss Loss3 represents an error between the predicted object category obtained by using the current neural network and the sample object category (that is, the actual object category), and this error can be measured by, for example, a distance. For example, the third loss Loss 3 is obtained by the following equation (4).

Here, m represents the number of the object category of the object to which the training sample belongs, M represents the maximum number of object categories of the object to which the training sample belongs, y _m represents the actual object category of the object of the object category m, and p. _m represents the predicted object category of the object of the object category m.

Returning to FIG. 9, in step S930, the CPU 110 determines the total loss obtained by the determination of all the current neural networks (that is, the first loss Loss1, the second loss Loss2, and the third loss Loss3). Judge whether the condition of is satisfied. For example, if the sum of the three losses is compared to a threshold (eg TH2) and the sum of the three losses / weighted sum is less than or equal to TH2, then all current neural networks satisfy a given condition. It is determined and output as the final neural network (ie, pre-generated model). Here, the final neural network is output to the storage device 240 shown in FIG. 2, for example, for object detection described with reference to FIGS. 2 to 8. If the sum of the three losses / the sum of the weights is greater than TH2, it is determined that all the current neural networks do not meet the predetermined conditions, and the generation process proceeds to step S940.

In step S940, the CPU 110 updates the parameters of each layer of the current neural network of the portion for determining the spatial radius based on the first loss Loss 1. Here, the parameter of each layer is, for example, the weight of each convolution layer of the current neural network. In one example, the parameters of each layer are updated based on the first loss Loss 1, for example by using a stochastic gradient descent method.

In step S950, the CPU 110 updates the parameters of each layer in the current neural network of the portion that detects the area / candidate area of the object based on the second loss Loss2 and the third loss Loss3. The parameters of each layer here are also, for example, the weights of the convolution layers in the current neural network. In one embodiment, the parameters of each layer are updated based on a second loss Loss 2 and a third loss Loss 3, using, for example, stochastic gradient descent.

In step S960, the CPU 110 updates the parameters of each layer of the current neural network of the feature extraction portion based on the first loss Loss1, the second loss Loss2, and the third loss Loss3. Here, the parameter of each layer is also a weight in each convolution layer in the current neural network, for example. In one example, the parameters of each layer also use the stochastic gradient descent method to determine the first loss Loss 1. It is updated based on the second loss Loss2 and the third loss Loss3. After that, the generation process proceeds to step S910 again.

In the flowchart shown in FIG. 9, the condition of whether or not the sum / weight total of the three losses of the first loss Loss1, the second loss Loss2, and the third loss Loss3 satisfies a predetermined condition is the current neural network. It was set as a condition to stop the update of. However, it is clear that it does not have to be limited to this. For example, although step S930 is omitted, the corresponding update operation is stopped after the number of updates to the current neural network reaches a predetermined number.

All of the above units are exemplary and / or preferred modules for performing the processes described in the present disclosure. These units can be hardware units such as field programmable gate arrays (FPGAs), digital signal processors, application-specific integrated circuits, and / or software modules such as computer-readable programs. The units for performing each step are not described in detail above. However, when there are steps to perform a particular process, it may be the corresponding functional module or unit (implemented by hardware and / or software) to perform the same process. The technical solutions for all combinations of the steps described and the units corresponding to these steps are included in the disclosure of this application as long as the technical solutions they constitute are complete and applicable.

The methods and devices of the present invention can be implemented in multiple ways. For example, the methods and devices of the invention may be implemented by software, hardware, firmware, or any combination thereof. The above order of steps in the method is intended to be merely exemplary, and the steps in the method of the invention are not limited to the order specifically described above, unless otherwise stated. Further, in some embodiments, the invention can also be implemented as a program recorded on a recording medium containing machine-readable instructions for carrying out the methods according to the invention. Therefore, the present invention also includes a recording medium on which a program for carrying out the method according to the present invention is recorded.

Although some specific embodiments of the present invention have been detailed by way of example, those skilled in the art should understand that the above embodiments are merely exemplary and do not limit the scope of the invention. As a matter of course, those skilled in the art can modify the above embodiments without departing from the scope and spirit of the present invention. The scope of the present invention is defined by the accompanying claims.

Claims (13)

An object detector
Extraction means to extract features from images and
An acquisition means for acquiring the spatial relationship between the extracted features based on a pre-generative model that has learned the spatial relationship indicating a specific positional relationship between two related objects belonging to a predetermined category.
A detection means for detecting a candidate area of an object from the image based on the extracted features, and
Based on the spatial relationship between the features included in the two candidate regions of the detected candidate regions, the score of the candidate region indicating the probability that an object of a predetermined category exists in the region of the image is updated. Update method and
A determinant that determines the area of an object present in the image based on the score of the updated candidate area .
An object detection device characterized by having. The object detection according to claim 1, wherein the pre-generated model is a model that learns the probability that two related objects belong to the specific positional relationship based on the extracted features. apparatus. The acquisition means has the spatial relationship .
Position information and category information of at least one of the related objects ,
Connection point information of at least one of the related objects ,
Background information of the image,
The object detection device according to claim 1 or 2 , wherein the object detection device is obtained based on at least one of the above. The acquisition means acquires the spatial relationship representing the spatial constraint between the two related objects .
The spatial constraint, the relative positional relationship between two of the related objects, the phase relationship between two of the associated object, characterized in that it comprises at least one of corresponding shape relationship between two of the related objects The object detection device according to any one of claims 1 to 3 . The pre-generative model has at least a portion for extracting a feature from an image, a portion for determining the spatial relationship from the feature, and a portion for detecting an area of an object from the feature and the spatial relationship. Has three parts,
The object detection device according to any one of claims 1 to 4, wherein the three parts are simultaneously learned by means of backpropagation. The extraction means extracts features from the image based on the pre-generated model.
It said detecting means, on the basis of the pre-generated model, the object detection apparatus according to claim 5, characterized that you detecting the candidate region of the object from the image. The object detection device according to claim 5 or 6 , wherein the pre-generated model is generated based on a training sample labeled with the spatial relationship using a deep learning method. It is an object detection method
Extraction steps to extract features from images and
An acquisition step of acquiring the spatial relationship between the extracted features based on a pre-generative model that has learned the spatial relationship indicating a specific positional relationship between two related objects belonging to a predetermined category.
A detection step of detecting a candidate area of an object from the image based on the extracted features,
Based on the spatial relationship between the features included in the two candidate regions of the detected candidate regions, the score of the candidate region indicating the probability that an object of a predetermined category exists in the region of the image is updated. Update steps and
A determination step of determining the area of an object present in the image based on the score of the updated candidate area .
An object detection method characterized by having. The object detection according to claim 8, wherein the pre-generated model is a model that learns the probability that two related objects belong to the specific positional relationship based on the extracted features. Method. The acquisition step acquires the spatial relationship that represents the spatial constraint between the two related objects .
The spatial constraint, the relative positional relationship between two of the related objects, the phase relationship between two of the associated object, characterized in that it comprises at least one of corresponding shape relationship between two of the related objects The object detection method according to claim 8 or 9. The pre-generative model has at least a portion for extracting a feature from an image, a portion for determining the spatial relationship from the feature, and a portion for detecting an area of an object from the feature and the spatial relationship. Has three parts,
By means of backpropagation, the above three parts are learned at the same time.
In the extraction step, the features are extracted from the image using the pre-generated model.
In the detection step, a candidate region of the object is detected from the image using the pre-generated model.
The object detection method according to any one of claims 8 to 10, wherein the object detection method is characterized. A feature extraction method that extracts features from the image of the current video frame in the video,
An acquisition means for acquiring the spatial relationship between the extracted features based on a pre-generative model that has learned the spatial relationship indicating a specific positional relationship between two related objects belonging to a predetermined category.
A detection means for detecting a candidate area of an object from the image based on the extracted features, and
Based on the spatial relationship between the features included in the two candidate regions of the detected candidate regions, the score of the candidate region indicating the probability that an object of a predetermined category exists in the region of the image is updated. Update method and
A determinant that determines the area of an object present in the image based on the score of the updated candidate area .
An object detection device characterized by having. A storage medium that stores an instruction that executes the object detection method according to any one of claims 8 to 10 when executed by a computer.