JP7345355B2 - object identification device - Google Patents

Description

The present invention relates to an object identification device that identifies objects based on imaging information, and particularly to an object identification device that individually identifies a plurality of objects (including a plurality of objects having the same shape) arranged in a target area.

Technology for detecting objects using deep learning (a machine learning method using multilayer neural networks) is being researched. For example, there are many end-to-end methods such as SSD (Single Shot MultiBox Detector) and YOLO (You Only Look Once) that simultaneously detect the position and category of a target object based on imaging information. Proposed. These methods are based on multi-task learning that simultaneously performs learning using a multilayer neural network to detect the position of an object and learning using a multilayer neural network to determine the category of the object.
An object detection technique using SSD is disclosed in Non-Patent Document 1, for example, and an object detection technique using YOLO is disclosed in Non-Patent Document 2, for example.

“SSD: Single Shot MulitiBox Detector”, Wei Liu, Dragomir Anguelov. Domitru Erhan, Christian Szegedy, Scott reed, Cheng-Yang Fu and Alexsander C. berg (2015), https://arxiv.org/pdf/1512.02325.pdf “You Only Look Once Unified, Real-Time Object Detection”, Joseph Redmon, Santosh Divvala, Ross Girshick and Ali Farhadi (2016), https://pjreddie,com/media/files/papers/yolo.pdf

The object detection techniques disclosed in Non-Patent Document 1 and Non-Patent Document 2 can detect the positions and categories of multiple objects placed in the target area, but Objects, especially multiple objects with the same shape (including "substantially the same shape") cannot be individually identified.
For this reason, conventionally, placement information (positions and IDs of multiple objects) indicating the placement state of multiple objects placed in the target area is created (registered) as a two-dimensional map, and the information is output from the imaging means. Object detection information (positions and categories of multiple objects placed in the target area in the imaging region) detected based on the imaging information obtained by Multiple objects were individually identified by matching them.
However, it is necessary to manually create a two-dimensional map each time an object is registered, and creating a two-dimensional map requires effort and time.
The present invention was devised in view of these points, and provides a technology that can easily individually identify multiple objects (including multiple objects with the same shape) placed in a target area. The purpose is to provide.

The object identification device of the present invention is used to individually identify a plurality of objects arranged in a predetermined target area, particularly a plurality of objects including a plurality of objects having the same shape (including "substantially the same shape"). It will be done.
The present invention includes an imaging means, a target area determination means, an object detection means, an object placement information generation means, an object identification means, and a storage means.
The imaging means outputs imaging information indicating an imaging screen that captures an imaging area that includes a target area and a plurality of objects (including a plurality of objects having the same shape) arranged in the target area. As the imaging means, a digital camera using CCD or CMOS is preferably used.
The target area determination means determines the target area on the image capture screen based on the imaging information from the image capture unit.
The object detection means detects the categories and positions of the plurality of objects arranged in the target area on the imaging screen based on the imaging information from the imaging means and the target area on the imaging screen determined by the target area determining means. do. Preferably, the object detection means outputs object detection information indicating the categories and positions of the plurality of objects placed in the target area. Various known methods can be used as the method for determining the target region by the target region determining means and the method for detecting the category and position of the object by the object detecting means. For example, detection methods based on deep learning such as SSD and YOLO can be used.
The object placement information generating means determines the placement state of the plurality of objects in the target area based on the categories and positions (object detection information) of the plurality of objects placed in the target area on the imaging screen, which are detected by the object detection means. The object arrangement information shown is generated and stored in the storage means. The object placement information includes identification information (ID) that identifies the object and position information that indicates the position of the object in the target area. Identification information (ID) is given to each object by an appropriate method. Preferably, identification information (ID) including category information indicating the category of the object is used.
The object identification means uses object detection information (categories and positions of a plurality of objects placed in the target area on the imaging screen) output from the object detection means and object placement information (categories and positions of multiple objects placed in the target area on the imaging screen) stored in the storage means. The plurality of objects arranged in the target area on the imaging screen are individually identified based on the identification information and positions of the plurality of arranged objects. For example, the correspondence relationship between a plurality of objects arranged in a target area on an imaging screen and a plurality of objects indicated by object arrangement information is determined.
The present invention is configured to be able to be set to an object placement information generation mode and an object identification mode. When set to the object placement information generation mode, object placement information is generated by the target area determination means, object detection means, and object placement information generation means, and is stored in the storage means. When the object identification mode is set, a plurality of objects arranged in the object area on the image capturing screen are individually identified by the object area determining means, the object detecting means, and the object identifying means. Various methods can be used to configure the device to be set to object placement information generation mode or object identification mode. For example, the object placement information generation mode is set by inputting object placement information generation mode setting information from an input means, etc., and the object identification mode is set by inputting object identification mode setting information from an input means etc. It can be configured as follows. Alternatively, by inputting object identification start information from an input means or the like, the object arrangement information generation mode is first set, and then the object identification mode can be set.
In the present invention, in the object placement information generation mode, object placement information indicating the placement state of a plurality of objects in the target area is generated, and in the object identification mode, the object placement information is referred to to identify the plurality of objects placed in the target area. Since objects are individually identified, multiple objects placed in the target area can be easily identified individually.
In a different embodiment of the present invention, a viewing angle change amount determining means is provided. The viewing angle change amount determining means distinguishes between the imaging screen (M) used to generate the object arrangement information stored in the storage means and the imaging screen (P) indicated by the imaging information output from the imaging means. It is determined whether the amount of change in the angle of view between the two exceeds a predetermined range. As a method for determining whether the amount of change in the angle of view between the imaging screen (M) and the imaging screen (P) exceeds a predetermined range, it is preferable to A method can be used to determine whether or not the degree of similarity is low. For example, the feature points of the imaging screen (M) and the imaging screen (P) are detected, and the similarity of the feature vectors including the feature points with the same correspondence among the feature points of both screens (for example, the cosine distance of both vectors) is calculated. ) is greater than a predetermined value, it is determined that the amount of change in the angle of view does not exceed a predetermined range, and if it is less than a predetermined value, it is determined that the amount of change in the angle of view exceeds the predetermined range. Of course, other methods may be used to determine whether the amount of change in the angle of view exceeds the predetermined range. Note that the amount of change in the angle of view between the target area (M) on the imaging screen (M) and the target area (P) on the imaging screen (P) is the amount of change in the angle of view between the imaging screen (M) and the imaging screen (P). This is equivalent to the amount of change in angle of view.
When the object identification mode is set and the angle of view change determination means determines that the amount of change in the angle of view exceeds a predetermined range, the target area determination means, the object detection means, and the object placement information are generated. The object arrangement information is generated by the means and stored in the storage means.
When the imaging position or imaging angle (imaging direction) of the imaging means is changed, the arrangement state of multiple objects on the imaging screen and the imaging screen used to generate the object arrangement information stored in the storage means are changed. There is a risk that the deviation from the arrangement state of the plurality of objects (especially their position on the imaging screen) will become large, and the accuracy of individual identification of the plurality of objects based on the comparison between the object detection information and the object arrangement information by the object identification means will decrease. be. In this embodiment, the amount of change in the angle of view between the imaging screen used to generate the object placement information stored in the storage means and the imaging screen indicated by the imaging information output from the imaging means is within a predetermined range. If the object location information is exceeded, the object placement information is regenerated and stored in the storage means. can be reliably identified.
In a different form of the present invention, the object identifying means is configured to identify the categories and positions of the plurality of objects located in the target area on the imaging screen detected by the object detecting means, and the object arrangement information stored in the storage means. A plurality of objects are individually identified by comparing identification information and positions of the plurality of objects while allowing a predetermined error range. Preferably, the method of verifying the position of the object is based on the coordinates of the four corner points of the bounding box (the object's existing region candidate estimated by the object detection means), or the width and height of the bounding box, and the position of the center of gravity. A method of matching coordinates is used.
In this embodiment, individual identification of a plurality of objects can be easily performed.
In a different form of the present invention, the object detecting means detects the categories of the plurality of objects and their relative positions in the target area, and the object placement information generating means detects the object placement information indicating the identification information of the plurality of objects and their relative positions in the target area. Generate information. As the relative position, for example, coordinates in a coordinate system set on the imaging screen are used.
The object identification means uses the relative position of the object to check the object detection information and the object placement information.
In this embodiment, matching processing between object detection information and object arrangement information can be easily performed.
In a different form of the present invention, the target area determining means and the object detecting means are configured by a multilayer neural network.
In this embodiment, it is possible to easily perform a process of determining a target area and a process of detecting categories and positions of a plurality of objects arranged in a target area.

According to the present invention, it is possible to easily individually identify a plurality of objects arranged in a target area.

FIG. 1 is a block diagram of an embodiment of an object identification device of the present invention. It is a diagram showing the neural network configuration of SSD. It is a figure showing an example of a captured image. 3 is a diagram showing a target area in the captured image shown in FIG. 2. FIG. 3 is a diagram showing an example of an object detection result based on the captured image shown in FIG. 2. FIG. 3 is a diagram showing an example of object detection information based on the captured image shown in FIG. 2. FIG. 6 is a diagram showing an example of object placement information based on the object detection information shown in FIG. 5. FIG. FIG. 6 is a diagram illustrating an operation of individually identifying a plurality of objects based on object detection information and object placement information. 3 is a flowchart illustrating an embodiment of operations in object placement information generation mode. 3 is a flowchart illustrating an embodiment of operation in object identification mode. FIG. 6 is a diagram illustrating a method for determining the amount of change in the angle of view based on a feature vector of a captured image. 12 is a flowchart illustrating another embodiment of operation in object identification mode.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram of an embodiment of an object identification device of the present invention.
The object identification device of this embodiment includes an imaging means 10, a processing means 20, a storage means 30, an input means 40, a display means 50, and the like.

The imaging means 10 is preferably constituted by a digital camera using a CCD or CMOS. The imaging means 10 is arranged so as to be able to image an imaging area including a target area and a plurality of objects placed in the target area. In the following description of the embodiment, it is assumed that the plurality of objects includes at least one type of plurality of objects having the same shape.
In object detection processing, which will be described later, the category (type) of the object is determined based on the "shape of the object." The expression "same shape" is used to include "substantially the same shape" that is determined by object detection processing to be objects of the same category.

The processing means 20 is composed of a CPU or the like. The processing means 20 includes a management means 21 , an imaging information input means 22 , a target area determination means 23 , an object detection means 24 , an object placement information generation means 25 , an angle of view change amount determination means 26 , and an object identification means 27 . In this embodiment, each of the means 21 to 27 is provided in a CPU that constitutes the processing means 20. Of course, at least one of the means 21 to 27 may be configured with a CPU different from the CPU constituting the processing means 20.

The processing of each means 21 to 27 will be explained with reference to FIGS. 2 to 8.
In the following, as shown in FIG. 3, the operation panel 110 of the operation panel 100 has three volumes 120a to 120c of the same shape, two dials 130a and 130b of the same shape, and 11 dials of the same shape. A case where buttons 140a to 140k are arranged will be explained.
The operation panel 110 is surrounded by a rectangular edge 111 formed by an upper edge 111a, a left edge 111b, a lower edge 111c, and a right edge 111d.
Note that reference numeral 11 indicates an image capturing screen in which an image capturing area including the operation panel 110, volumes 120a to 120c, dials 130a, 130b, and buttons 140a to 140k is imaged by the image capturing means 10.
The operation panel 110 corresponds to the "target area" of the present invention, and the edge 111 (upper edge 111a to right edge 111d) of the operation panel 110 corresponds to the "target area edge (target area upper edge)" of the present invention. The volumes 120a to 120c, the dials 130a and 130b, and the buttons 140a to 140k correspond to the "multiple objects placed in the target area" of the present invention, and the volumes 120a to 120c correspond to 120c, dials 130a, 130b, or buttons 140a to 140k correspond to "a plurality of objects having the same shape" of the present invention. Further, the volume, dial, and button correspond to the "object category" of the present invention.

The management means 21 manages the processing of each means 22-27.
The imaging information input means 22 inputs imaging information from the imaging means 10.

The target area determining means 23 determines the target area based on the imaging information from the imaging means 10 inputted by the imaging information inputting means 22 .

Various methods can be used to determine the target area by the target area determining means 23. For example, the SSD disclosed in Non-Patent Document 1 can be used.
As shown in Figure 2, SSD is based on a multi-layer CNN (Convolutional neural network), which includes a layer that estimates object region candidates and a layer that estimates the object category within the object region candidates. It is composed of layers to be determined. In the layer that estimates the object's existing area candidate, image information is divided into multiple rectangular areas (default boxes) of a predetermined size, and the object's existing area candidate (bounding box) is estimated while taking into account the deviation of the rectangular areas. . In the layer that determines the category of an object within the existence region candidate, a separately trained CNN is used to determine the category of the object within the existence region candidate.
The SSD used in the target area determination means 23 is composed of a layer for estimating an existence area candidate of a target area in which a plurality of objects are arranged, and a layer for determining a target area within the existence area candidate. The target area determining means 23, which is constituted by an SSD, determines the target area and its position on the image capture screen where the image capture area is captured.
In this embodiment, as shown in FIG. 4, the operation panel 110 surrounded by the edge 111 is determined as the target area 110.

The object detection means 24 is arranged (displayed) in the target area on the imaging screen based on the imaging information from the imaging means 10 inputted by the imaging information input means 22 and the target area determined by the target area determination means 23. Detect the categories and locations of multiple objects in your environment.
Various methods can be used for the object detection means 24 to detect the categories and positions of a plurality of objects placed in the target area. For example, similar to the target area determination means 23, the SSD shown in FIG. 2 can be used.
The SSD used in the object detection means 24 is composed of a layer that estimates the existence region candidates (bounding boxes) of each of a plurality of objects in the target region, and a layer that detects the category of the object in each existence region candidate. .
In this embodiment, the bounding boxes indicated by broken lines in FIG. 5 and the volumes 120a to 120c, dials 130a and 130b, and buttons 140a to 140k within each bounding box are detected.

Then, the object detection means 24 outputs object detection information indicating the categories and positions of a plurality of objects placed in the target area. As the position of the object, the coordinates of the four corner points of the bounding box, or the coordinates of the width, height, and center of gravity of the bounding box are preferably used. As the coordinates, xy coordinates in an xy coordinate system set in the target area are preferably used. The "xy coordinates in the xy coordinate system set in the target area" corresponds to the "relative position in the target area" of the present invention.
In this embodiment, as shown in FIG. 6, three volumes V, two dials D, and 11 switches S are arranged at the bounding box indicated by the broken line. Object detection information indicating .

The object placement information generation means 25 creates object placement information based on the object detection information (categories and positions of a plurality of objects placed in the target area) output from the object detection means 24, and stores it in the storage means 30. do. The object arrangement information includes identification information (ID) that identifies each object and position information that indicates the position. The method of assigning identification information (ID) to each object can be set as appropriate. The identification information (ID) includes category information indicating the category of the object. Preferably, objects having the same shape are given identification information including category information (common information) indicating the type of object and a number (individual information). For example, the numbers are assigned in increasing order along the direction in which the dispersion of the center of gravity positions of the objects is large.
As the position of the object indicated by the object placement information, the position in the target area is preferably used, and more preferably the relative position in the target area is used.
In FIG. 7, an xy coordinate system is set with the upper edge 111a and the left edge 111b (the upper edge of the target area and the left edge of the target area) as the x and y axes, respectively, of the operation panel 110. The xy coordinates are used as relative positions.

For example, as shown in FIG. 7, the object placement information generation means 25 assigns identification information [V1] to [V3] to the three volumes [V] in order from the left side along the x-axis. Then, identification information [D1] and [D2] are assigned to the two dials [D] in order from the left side along the x-axis, and the upper seven buttons [S] of the 11 buttons [S] are assigned identification information [D1] and [D2] in order from the left side along the S] are given identification information [S1] to [S7] in order from the left side along the x-axis, and identification information [S8] to [S7] are given in order from the top to the four buttons [S] on the right side. S11].
In Fig. 7, the positions of the volumes [V1] to [V3], dials [D1], [D2], and buttons [S1] to [S11] are the width, height, and center of gravity of their respective bounding boxes. It is indicated by G. The position of point G is expressed by xy coordinates in the xy coordinate system.
The configuration of "representing the position of the center of gravity by xy coordinates in an xy coordinate system set in the imaging screen or the target area of the imaging screen" is included in the configuration of "representing the position of the object by a relative position" of the present invention.

Note that the view angle change amount determining means 26 will be described later.

The object identification means 27 includes object detection information (categories and positions of a plurality of objects arranged in the target area) outputted from the object detection means 24 and object arrangement information (identification of each object) stored in the storage means 30. A plurality of objects placed in the target area are individually identified based on the information and position).
Various methods can be used as the object identification method by the object identification means 27. For example, the positions of objects with the same shape are extracted from object detection information and object placement information. Then, the position extracted from the object detection information and the position extracted from the object placement information are compared to determine whether the two positions match. At this time, a determination as to whether the two positions match is made while allowing a predetermined error range. That is, if the difference between the two positions is within a predetermined error range, it is determined that the two positions match, and if the difference exceeds the predetermined error range, it is determined that the two positions do not match. If it is determined that both positions match, the object placed at the position extracted from the object detection information is the object indicated by the identification information given to the object placed at the position extracted from the object placement information. .

Specifically, the object identification means 27 compares the object detection information output from the object detection means 24 with the object arrangement information stored in the storage means 30, as shown in FIG.
For example, the position of the volume [V] (1) (the width and height of the bounding box and the xy coordinates of the center of gravity) included in the object detection information and the volume [V] included in the object placement information Check the positions of volumes [V1] to [V3] of the same shape (same category) as in (1) (the width and height of the bounding box and the xy coordinates of the center of gravity), and check whether the difference between the two is within the specified range. Decide whether or not. At this time, the difference between the position of volume [V] (1) and the positions of volumes [V2] and [V3] exceeds the predetermined range, but the difference between the position of volume [V] (1) and volume [V1] is If the difference in position is within a predetermined range, the volume [V] (1) included in the object detection information is identified as the volume [V1] included in the object placement information (individual identification). do.

The storage means 30 is constituted by ROM, RAM, etc., and stores programs for executing the processes of each means 21 to 27 and various data. The object placement information created by the object placement information generation means 25 is stored in the object placement information storage section 31 of the storage means 30.
The input means 40 is constituted by a keyboard or the like, and inputs various information.
The display means 50 is constituted by a liquid crystal display device, an organic EL display device, etc., and displays various information. In addition, when the display means 50 is a display means in which information can be input by touching the display section displayed on the display screen, the input means 40 can be omitted.

Next, the operation of the object identification device of this embodiment will be explained. The object identification device of this embodiment is configured to be able to be set to object placement information generation mode or object identification mode.
For example, by inputting object placement information generation mode setting information or object identification mode setting information from the input means 40, it is configured to be set to object placement information generation mode or object identification mode. Alternatively, by inputting object identification start information from the input means 40, the object arrangement information generation mode is first set, and after the object arrangement information is generated and stored in the storage means 30, the object identification mode is set. It is configured as follows.

The operation when the object information generation mode is set will be explained with reference to the flowchart shown in FIG.
In step A1, image information captured by the image capturing means 10 is input. The process of step A1 is executed by the imaging information input means 22.
In step A2, a target area is determined based on the input imaging information. The process of step A2 is executed by the target area determining means 23.
In step A3, the categories and positions of a plurality of objects placed in the target area are detected based on the input imaging information and the target area determined in step A2, and object detection information is output. The process of step A3 is executed by the object detection means 24.
In step A4, object placement information including identification information for identifying each object and the position of each object is generated based on the object detection information detected in step A3 and indicating the category and position of the object placed in the target area. do. The process of step A4 is executed by the object placement information generating means 25.
In step A5, the object placement information generated in step A4 is stored in the storage means 30. The process of step A5 is executed by the object placement information generating means 25.

The operation when the object identification mode is set will be explained with reference to the flowchart shown in FIG.
In step B1, image information captured by the image capturing means 10 is input. The process of step B1 is executed by the imaging information input means 22.
In step B2, a target area is determined based on the input imaging information. The process of step B2 is executed by the target area determining means 23.
In step B3, the categories and positions of a plurality of objects placed in the target area are detected based on the input imaging information and the target area determined in step B2, and object detection information is output. The process of step B3 is executed by the object detection means 24.
In step B4, the object detection information indicating the category and position of the object placed in the target area detected in step B3 and stored in the storage means 30 (generated in step A4 in the object placement information generation mode) The object arrangement information (identification information and position of each object) is compared to individually identify the plurality of objects located in the target area detected in step B3. The process of step B4 is executed by the object identifying means 27.
After completing the process in step B4, the process returns to step B1. Note that the processes of steps B1 to B4 may be repeatedly executed continuously, or may be executed at predetermined time intervals.

Next, different embodiments of the present invention will be described.
When the imaging position and imaging angle (imaging direction) of the imaging means 10 are changed, the positions (xy coordinates) of the plurality of objects arranged in the target area on the imaging screen change.
If the positions of the plurality of objects placed in the target area on the imaging screen change significantly, the accuracy of matching the object detection information and the object placement information, that is, the accuracy of identifying each object individually, may decrease.
For this reason, when the imaging position or imaging angle (imaging direction) of the imaging means 10 is changed, it is preferable to regenerate the object arrangement information to prevent a decrease in the accuracy of individual identification of each object.
Whether the imaging position or imaging angle (imaging direction) of the imaging means 10 has been changed can be determined by the amount of change in the angle of view of the imaging screen, as will be described later.

This embodiment includes a viewing angle change amount determining means 26 shown in FIG.
The viewing angle change amount determining means 26 distinguishes between the imaging screen (M) used when generating the object arrangement information stored in the storage means 30 and the imaging screen (P) indicated by the imaging information output from the imaging means 10. ) is beyond a predetermined range. The angle of view changes depending on the imaging position and imaging angle (imaging direction) of the imaging means 10.
Various methods can be used as a method for determining whether or not the amount of change in the angle of view exceeds a predetermined range (a method for determining the amount of change in the angle of view) by the angle of view change amount determining means 26.
For example, a method may be used that determines whether the degree of similarity between the captured screen (M) and the captured screen (P) is lower than a predetermined value. The degree of similarity between the image capture screen (M) and the image capture screen (P) can be determined based on the degree of similarity between the feature points of the image capture screen (M) and the feature points of the image capture screen (P). Here, each feature point of the imaging screen (M) and the imaging screen (P) can be detected using a method based on local feature amounts such as SIFT. As described later, by using the feature points of the image capture screen (M) and the image capture screen (P), it is possible to determine the degree of similarity between the image capture screen (M) and the image capture screen (P).

Here, the amount of change in the angle of view between the target area (M) on the imaging screen (M) and the target area (P) on the imaging screen (P) is the difference between the imaging screen (M) and the imaging screen (P). is equivalent to the amount of change in angle of view.
In this embodiment, as shown in FIG. By determining whether the degree of similarity between the feature points of (M) and the feature points of the target area (P) on the imaging screen (P) indicated by the imaging information output from the imaging means 10 is lower than a predetermined value. , it is determined whether the amount of change in the angle of view between the imaging screen (M) and the imaging screen (P) exceeds a predetermined range.
As the feature points of the target area, for example, the locations illustrated by broken lines in FIG. 11 are detected. Note that preferably a plurality of feature points are detected.
Then, feature vectors for both target regions are generated from a set of feature points that are determined to have matching correspondence relationships based on the degree of similarity between the respective feature points based on the local feature amount among the feature points of both target regions. Then, it is determined whether the degree of similarity between the feature vectors of both target regions is greater than or equal to a predetermined value.
For example, if the feature vector of the target region (M) is <M> and the feature vector of the target region (P) is <P>, the similarity of the feature vectors is expressed by the cosine distance of both vectors shown below. .
Sim(〈M〉,〈P〉) = (〈M〉・〈P〉)/(｜〈M〉｜｜〈P〉｜)
If the cosine distance is greater than or equal to a predetermined value, it is determined that the similarity of the feature vectors is high and the amount of change in the angle of view does not exceed a predetermined range, and if the cosine distance is less than a predetermined value, the similarity of the feature vectors is It is determined that the amount of change in the angle of view is low and the amount of change in the angle of view exceeds a predetermined range.
Note that various methods other than the above methods can be used as the feature point detection, feature vector calculation method, and similarity determination method.
Furthermore, as a method for detecting the amount of change in the angle of view, various methods other than the above method can be used, such as a method of calculating the amount of change due to pixel movement as a velocity vector using optical flow.
The feature points of the imaging screen (M) and the imaging screen (P) are not limited to the feature points of the target area (M) on the imaging screen (M) and the target area (P) on the imaging screen (P).

The operation of the object identification device of this embodiment will be explained. The object identification device of this embodiment is configured to be able to be set to object placement information generation mode or object identification mode.
The operation when the object placement information generation mode is set is the same as the operation shown in FIG. 9, so the explanation will be omitted.

The operation when the object identification mode is set will be explained with reference to the flowchart shown in FIG.
The operations of steps C1 and C2 are the same as the operations of steps B1 and B2 shown in FIG.
In step C3, it is determined whether the amount of change in the angle of view exceeds a predetermined range. If the amount of change in angle of view exceeds the predetermined range, the process proceeds to step C4. On the other hand, if the amount of change in the angle of view does not exceed the predetermined range, the process proceeds to step C7. The process of step C3 is executed by the viewing angle change amount determining means 26.
In steps C4 to C6, object placement information is generated (regenerated) and stored in the storage means 30 in the same way as steps A3 to A5 shown in FIG.
In steps C7 and C8, similar to steps B3 and B4 shown in FIG. 10, a plurality of objects placed in the target area are individually identified.

A preferred form of use of the present invention is assumed to be a monitoring device that monitors the status of a plurality of operation sections (including a plurality of operation sections having the same shape) arranged on an operation panel of an operation panel or the like.
When the SSD disclosed in Non-Patent Document 1 or the YOLO object detection means disclosed in Non-Patent Document 2 is used in such a monitoring device, as described above, objects of the same shape existing in the target area are detected. Unable to identify multiple objects individually.
For this reason, conventionally, arrangement information (position and ID of each object) indicating the arrangement state of a plurality of objects arranged in a target area is created (registered) as a two-dimensional map, and is output from an imaging means. Compare object detection information (positions and categories of multiple objects placed in the target area) detected based on imaging information with object placement information (position and ID of each object) shown on a two-dimensional map. This allows each object to be identified individually.
However, it is necessary to manually create a two-dimensional map each time an object is registered, and creating a two-dimensional map requires effort and time.
On the other hand, when using the object identification device of the present invention, object placement information is automatically generated, so there is no need to manually create a two-dimensional map showing the placement state of objects placed in the target area. .
In addition, each time imaging information is input, the object detection information and object placement information are compared to identify each of the multiple operation parts arranged on the operation panel. The state of the operating section can be easily identified.

The present invention is not limited to the configuration described in the embodiments, and various changes, additions, and deletions are possible.
Although the processing means 20 is configured by the management means 21 to the object identification means 27, the configuration of the processing means 20 is not limited to this. For example, unnecessary means can be deleted or multiple means can be integrated.
The configurations of the target area determination means 23, the object detection means 24, the object placement information generation means 25, the angle of view change amount determination means 26, and the object identification means 27 are not limited to those described in the embodiments, and the gist of the present invention is Various changes can be made within the same range.
The processes shown in FIGS. 9, 10, and 12 can be modified in various ways.

10 Imaging means 11 Imaging screen 20 Processing means 21 Management means 22 Imaging information input means 23 Target area determination means 24 Object detection means 25 Object arrangement information creation means 26 Object identification means 27 View angle change determination means 30 Storage means 31 Object arrangement information storage Section 40 Input means 50 Display means 100 Operation panel 110 Operation panel (target area)
111 Edge (target area edge)
111a Upper edge (upper edge of target area)
111b Left side edge (left side edge of target area)
111c Lower edge (lower edge of target area)
111d Right edge (right edge of target area)
120a~120c Volume 130a, 130b Dial 140a~140k Button

Claims (5)

An object identification device that individually identifies a plurality of objects, including a plurality of objects of the same shape, arranged in a target area,
comprising an imaging means, a target area determination means, an object detection means, an object placement information generation means, an object identification means, and a storage means,
The imaging means outputs imaging information indicating an imaging screen that captures an imaging area including the target area and the plurality of objects arranged in the target area,
The target area determining means determines the target area on the imaging screen based on the imaging information output from the imaging unit,
The object detecting means is arranged in the target area on the imaging screen based on the imaging information output from the imaging means and the target area on the imaging screen determined by the target area determining means. detecting categories and positions of the plurality of objects;
The object placement information generation means generates identification information and positions of the plurality of objects based on categories and positions of the plurality of objects arranged in the target area on the imaging screen detected by the object detection means. generating object placement information shown and storing it in the storage means,
The object identification means is configured to identify categories and positions of the plurality of objects located in the target area on the imaging screen detected by the object detection means, and the object arrangement information stored in the storage means. individually identifying the plurality of objects arranged in the target area on the imaging screen based on identification information and positions of the plurality of objects;
It can be set to object placement information generation mode and object identification mode,
When the object placement information generation mode is set, the object placement information is generated by the target area determination means, the object detection means, and the object placement information generation means, and is stored in the storage means;
When the object identification mode is set, the object area determining means, the object detecting means, and the object identifying means individually identify the plurality of objects arranged in the object area on the imaging screen. An object identification device comprising: The object identification device according to claim 1,
Equipped with means for determining the amount of change in angle of view,
The viewing angle change amount determining means determines the difference between the imaging screen used to generate the object arrangement information stored in the storage means and the imaging screen indicated by the imaging information output from the imaging means. Determine whether the amount of change in the angle of view exceeds a predetermined range,
When the object identification mode is set and the angle of view change amount determination means determines that the amount of change in the angle of view exceeds the predetermined range, the object area determination means, the object detection means and An object identification device characterized in that the object arrangement information is configured to be generated by the object arrangement information generating means and stored in the storage means. The object identification device according to claim 1 or 2,
The object identification means detects the category and position of the object located in the target area on the imaging screen detected by the object detection means, and the object placement information stored in the storage means. An object identification device characterized in that the plurality of objects are individually identified by comparing identification information and positions while allowing a predetermined error range. The object identification device according to any one of claims 1 to 3,
The object detection means detects categories and relative positions of the plurality of objects in the target area,
The object identification device is characterized in that the object placement information generating means generates object placement information indicating identification information of the plurality of objects and relative positions in the target area. The object identification device according to any one of claims 1 to 4,
An object identification device characterized in that the target area discriminating means and the object detecting means are constituted by a multilayer neural network.

Images