JP7761658B2 - Annotation Device - Google Patents

Description

The present disclosure relates to an annotation device.

At waste disposal sites, large amounts of waste flow on conveyor belts and are processed every day. At the waste disposal sites, the waste is sorted by hand. While sorting waste is a simple task, it places a heavy burden on the workers who sort the waste (hereinafter referred to as "sorters"). Therefore, devices that automatically sort waste (hereinafter referred to as "waste sorting devices") have been developed.

Japanese Patent Application Laid-Open No. 2019-049780

If a waste sorting device were to take over the tasks previously performed by a sorter, the waste sorting device could recognize each piece of waste flowing on the conveyor belt and use a robotic hand to pick up the desired waste based on the recognition results. Therefore, when multiple types of waste are mixed together on the conveyor belt, the waste sorting device needs to be able to identify the type of waste. For the waste sorting device to recognize various types of waste, it is effective to use a trained model generated by machine learning.

However, machine learning requires a huge amount of training data, and when generating training data from image data of various types of waste, if the annotation work on the image data is done manually by visually inspecting the images, generating the training data would be extremely labor-intensive.

Therefore, this disclosure proposes technology that can improve the efficiency of annotation work.

The annotation device disclosed herein includes an acquisition unit, a first determination unit, an annotation unit, and a second determination unit. The acquisition unit acquires a first image captured at a first timing before an operation of selecting a desired object from a group of objects on a transport path along which the group of objects is transported is performed, at a first position upstream of the position where the operation is performed, and a second image captured at a second timing after the operation is performed, at a second position downstream of the position where the operation is performed. The first determination unit compares the first image with the second image and determines a difference region, which is an area where there is a difference between the first image and the second image. The annotation unit assigns contour information, which is information indicating the contour of the difference region, to the difference region in the first image, and generates candidate data including the first image and the contour information. The second determination unit determines whether to use the candidate data as training data for machine learning.

The disclosed technology can improve the efficiency of annotation work.

FIG. 1 is a diagram illustrating a configuration example of an annotation system according to a first embodiment of the present disclosure. FIG. 2 is a diagram illustrating a configuration example of an annotation device according to a first embodiment of the present disclosure. FIG. 3 is a flowchart illustrating an example of a processing procedure in the annotation device according to the first embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of the operation of the annotation device according to the first embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of the operation of the annotation device according to the first embodiment of the present disclosure. FIG. 6 is a diagram illustrating an example of the operation of the annotation device according to the first embodiment of the present disclosure. FIG. 7 is a diagram illustrating a configuration example of an annotation device according to a second embodiment of the present disclosure. FIG. 8 is a flowchart illustrating an example of a processing procedure in the annotation device according to the second embodiment of the present disclosure. FIG. 9 is a diagram illustrating an example of the operation of the annotation device according to the second embodiment of the present disclosure. FIG. 10 is a diagram illustrating an example of the operation of the annotation device according to the third embodiment of the present disclosure.

Embodiments of the present disclosure will be described below with reference to the drawings. In the following embodiments, identical configurations and steps that perform identical processing will be designated by the same reference numerals.

[Example 1]
<Configuration of the annotation system>
FIG. 1 is a diagram illustrating a configuration example of an annotation system according to a first embodiment of the present disclosure.

In FIG. 1, the annotation system 1 has an annotation device 10, an upstream camera 20, a downstream camera 30, and a belt conveyor 40. The annotation device 10 is connected to the upstream camera 20 and the downstream camera 30 via a network. The upstream camera 20 and the downstream camera 30 are positioned above the belt conveyor 40 and capture images of the top surface of the belt conveyor 40 from above the belt conveyor 40.

The following describes an example in which the annotation system 1 shown in Figure 1 is installed in a waste disposal site where waste materials flow on a belt conveyor 40. However, the annotation system 1 may also be installed in an assembly plant or the like where parts flow on a belt conveyor 40. In other words, the objects that can be targeted by the annotation system 1 are not limited to waste materials, and the annotation system 1 can be used for a variety of objects.

The belt conveyor 40 transports the waste materials placed on it in the transport direction CD. In other words, the belt conveyor 40 forms a transport path along which the waste materials are transported in the transport direction CD.

The upstream camera 20 is positioned at a first position P1 on the upstream side of the belt conveyor 40, and the downstream camera 30 is positioned at a second position P2 on the downstream side of the belt conveyor 40. A sorter M is positioned at a third position P3 between the first position P1 and the second position P2, where the sorter M sorts desired waste from a mass of waste transported by the belt conveyor 40 (hereinafter referred to as "waste sorting work"). Thus, the upstream camera 20 photographs the waste on the belt conveyor 40 from above the belt conveyor 40 at a first position P1 upstream of the third position P3 where the sorter M performs the waste sorting work, and at a first timing T1 before the waste sorting work is performed. The downstream camera 30 photographs the waste on the belt conveyor 40 from above the belt conveyor 40 at a second position P2 downstream of the third position P3 where the sorter M performs the waste sorting work, and at a second timing T2 after the waste sorting work is performed. An image I1 captured by the upstream camera 20 (hereinafter sometimes referred to as an "upstream image") is transmitted from the upstream camera 20 to the annotation device 10, and an image I2 captured by the downstream camera 30 (hereinafter sometimes referred to as a "downstream image") is transmitted from the downstream camera 30 to the annotation device 10. The upstream image I1 and the downstream image I2 form a pair of images (hereinafter sometimes referred to as an "image set"). Note that, in order for the sorter M to perform the waste sorting work, the conveying speed of the belt conveyor 40 is preferably set to, for example, about 170 mm per second.

The annotation device 10 performs annotation using the upstream image I1 and the downstream image I2.

<Configuration of annotation device>
Fig. 2 is a diagram illustrating a configuration example of an annotation device according to a first embodiment of the present disclosure. The annotation device 10a illustrated in Fig. 2 corresponds to the annotation device 10 illustrated in Fig. 1. In Fig. 2, the annotation device 10a includes an image acquisition unit 11, an image storage unit 12, a differential region determination unit 13, an annotation unit 14, a candidate data storage unit 15, an adoption determination unit 16, and a teacher data storage unit 17.

The image acquisition unit 11 acquires the upstream image I1 transmitted from the upstream camera 20 and the downstream image I2 transmitted from the downstream camera 30. The image acquisition unit 11 stores the acquired upstream image I1 and downstream image I2 in the image storage unit 12.

The difference region determination unit 13 reads the upstream image I1 and downstream image I2 stored in the image storage unit 12 from the image storage unit 12, compares the upstream image I1 with the downstream image I2, and determines regions where there are differences between the upstream image I1 and the downstream image I2 (hereinafter, sometimes referred to as "difference regions"). The difference region determination unit 13 determines difference regions using matching methods such as SAD (Sum of Absolute Difference), SSD (Sum of Squared Difference), or NCC (Normalized Cross-Correlation). Because a difference region is determined as a region connecting multiple coordinate points, the difference region determination unit 13 outputs coordinate information indicating the difference region (hereinafter, sometimes referred to as "difference region information") to the annotation unit 14.

The annotation unit 14 reads the upstream image I1 stored in the image storage unit 12 from the image storage unit 12. Then, based on the differential region information, the annotation unit 14 assigns information indicating the contours of the differential regions in the upstream image I1 (hereinafter sometimes referred to as "contour information") to the differential regions in the upstream image I1, and generates candidate data including the upstream image I1 and the contour information. The annotation unit 14 stores the generated candidate data in the candidate data storage unit 15. The annotation unit 14 also outputs the contour information assigned to the differential regions in the upstream image I1 to the adoption determination unit 16.

The adoption determination unit 16 determines whether or not to adopt the candidate data stored in the candidate data storage unit 15 as training data for machine learning based on the contour information (hereinafter sometimes referred to as "training data adoption determination"). The adoption determination unit 16 reads the candidate data determined to be adopted as training data (hereinafter sometimes referred to as "adopted data") from the candidate data storage unit 15 and stores it in the training data storage unit 17, and deletes the adopted data from the candidate data storage unit 15. On the other hand, the adoption determination unit 16 deletes the rejected data from the candidate data storage unit 15 without storing the candidate data determined not to be adopted as training data (hereinafter sometimes referred to as "rejected data") in the training data storage unit 17.

The training data stored in the training data memory unit 17 is used for machine learning to generate a trained model for the waste sorting device to recognize various types of waste.

<Processing procedure in annotation device>
FIG. 3 is a flowchart illustrating an example of a processing procedure in the annotation device according to the first embodiment of the present disclosure.

As shown in Figure 3, in step S100, at a first timing T1 before the waste sorting work is performed, an upstream image I1 is captured by the upstream camera 20 and transmitted to the annotation device 10a.

Next, in step S105, waste sorting work is performed by sorter M.

Next, in step S110, at a second timing T2 after the waste sorting work has been performed, a downstream image I2 is captured by the downstream camera 30 and transmitted to the annotation device 10a.

Next, in step S115, the difference region determination unit 13 determines the difference region.

Next, in step S120, the annotation unit 14 performs annotation to add contour information to the difference region in the upstream image I1. In step S125, the annotation unit 14 generates candidate data including the upstream image I1 and the contour information. In step S130, the annotation unit 14 stores the candidate data in the candidate data storage unit 15.

Next, in step S135, the adoption determination unit 16 calculates the area of the difference region (hereinafter sometimes referred to as the "difference region area") based on the contour information, and determines whether to adopt the training data based on the calculated difference region area.

When the differential region area is less than the first threshold TH1 (step S135: Yes), the adoption determination unit 16 adopts the candidate data as training data and stores the candidate data as adopted data in the training data storage unit 17 (step S140).

On the other hand, when the differential region area is greater than or equal to the first threshold value TH1 (step S135: No), the adoption determination unit 16 does not adopt the candidate data as training data, and deletes the unadopted candidate data from the candidate data storage unit 15.

<Operation of the annotation device>
4, 5, and 6 are diagrams illustrating an example of the operation of the annotation device according to the first embodiment of the present disclosure. Hereinafter, it is assumed that empty bottles are waste materials, and that the waste sorting device sorts each empty bottle traveling on a belt conveyor into three types of bottles: brown empty bottles (hereinafter referred to as "brown bottles"), transparent empty bottles (hereinafter referred to as "transparent bottles"), and empty bottles of a color other than brown or transparent (hereinafter referred to as "other-colored bottles"). It is also assumed below that a sorter M sorts brown bottles as desired waste materials from a group of waste materials transported by the belt conveyor 40 in order to generate training data for brown bottles.

As shown in FIG. 4, the first image set IS1 includes a first upstream image I1a and a first downstream image I2a. The first upstream image I1a also includes an image of a brown bottle (hereinafter sometimes referred to as the "brown bottle image") B1 and an image of bottles of other colors (hereinafter sometimes referred to as the "other color bottle image") B2. After the first upstream image I1a is captured by the upstream camera 20, the brown bottles are sorted by sorter M. Then, after sorter M has sorted the brown bottles, the first downstream image I2a is captured by the downstream camera 30. Therefore, the first downstream image I2a includes the other color bottle image B2, but does not include the brown bottle image B1. Therefore, the differential region determination unit 13 determines the first differential region DRa by comparing the first upstream image I1a and the first downstream image I2a. The outline of the first differential region DRa in the first upstream image I1a corresponds to the outline of the brown bottle image B1 in the first upstream image I1a. Therefore, the differential region determination unit 13 outputs differential region information formed by a plurality of coordinate points indicating the first differential region DRa to the annotation unit 14 .

Next, as shown in FIG. 5, the annotation unit 14 assigns contour information INb to the first differential region DRa in the first upstream image I1a. Like the differential region information, the contour information INb is formed by multiple coordinate points [x1, y1], [x2, y2], .... Therefore, the contour information INb can be used to determine the position and area of the first differential region DRa. Furthermore, the annotation unit 14 assigns label information INa and image information INc to the first differential region DRa. The label information INa includes information indicating the color of the empty bottle, "Brown Bottle," and the image information INc includes the image file name "XXX.bmp" indicating the first upstream image I1a, and height information "1536" and width information "2048" indicating the pixel size of the first upstream image I1a. The label information INa, contour information INb, and image information INc then form annotation data AD1. The annotation unit 14 associates the first upstream image I1a with the annotation data AD1 and generates candidate data including the first upstream image I1a and the annotation data AD1. The annotation data AD1 including the label information INa, the contour information INb, and the image information INc is generated by the annotation unit 14 as a file in, for example, JSON format.

Here, waste materials transported by the belt conveyor 40 are often densely packed, for example, piled up. Therefore, when sorting waste materials, the sorter M may shuffle the waste materials while selecting the desired waste materials. If the waste materials are shuffled, the location of each waste material in the waste materials changes significantly before and after the waste sorting process, resulting in significant differences between the upstream and downstream images. For example, as shown in FIG. 6, the area of the second difference region DRb increases between the second upstream image I1b and the second downstream image I2b included in the second image set IS2. When the area of the second difference region DRb increases in this way, the contour of the second difference region DRb no longer represents the shape of an empty bottle, increasing the likelihood of errors occurring in the annotation data generated by the annotation unit 14.

Therefore, the adoption determination unit 16 adopts the candidate data as training data when the differential region area is less than the first threshold TH1, but does not adopt the candidate data as training data when the differential region area is equal to or greater than the first threshold TH1. This prevents an upstream image associated with erroneous annotation data from being adopted as training data.

In the above description of Example 1, the teacher data adoption determination was made based on the differential region area. However, instead of making the teacher data adoption determination based on the differential region area, the differential region determination unit 13 may calculate the differential region area, and the annotation unit 14 may perform annotation when the differential region area is less than the first threshold TH1, but may not perform annotation when the differential region area is equal to or greater than the first threshold TH1. Furthermore, when the differential region area is equal to or greater than the first threshold TH1, the differential region determination unit 13 may delete the image set used to calculate the differential region area (hereinafter sometimes referred to as the "calculation source image set") from the image storage unit 12, thereby preventing the annotation unit 14 from annotating the upstream image I1 included in the calculation source image set.

The above describes Example 1.

[Example 2]
The second embodiment differs from the first embodiment in that the determination of whether training data is adopted is made based on the image characteristics of the differential region. The differences from the first embodiment will be described below.

<Configuration of annotation device>
Fig. 7 is a diagram illustrating a configuration example of an annotation device according to a second embodiment of the present disclosure. The annotation device 10b illustrated in Fig. 2 corresponds to the annotation device 10 illustrated in Fig. 1. In Fig. 7, the annotation device 10b includes an image acquisition unit 11, an image storage unit 12, a differential region determination unit 13, an annotation unit 14, a candidate data storage unit 15, an adoption determination unit 16, a teacher data storage unit 17, and a feature detection unit 18.

The feature detection unit 18 reads the upstream image I1 and downstream image I2 stored in the image storage unit 12 from the image storage unit 12. In addition, the feature detection unit 18 receives differential region information from the differential region determination unit 13.

The feature detection unit 18 detects image features of the difference region in the upstream image I1 (hereinafter sometimes referred to as "upstream difference region features") from the upstream image I1. The feature detection unit 18 also detects image features of the difference region in the downstream image I2 (hereinafter sometimes referred to as "downstream difference region features") from the downstream image I2. The feature detection unit 18 then outputs information including information indicating the upstream difference region features and information indicating the downstream difference region features (hereinafter sometimes referred to as "feature information") to the adoption determination unit 16. The feature detection unit 18 detects the upstream difference region features and the downstream difference region features using, for example, a method such as SIFT (Scaled Invariance Feature Transform) or HOG (Histograms of Oriented Gradients).

The adoption determination unit 16 determines whether to adopt the training data based on the characteristic information.

<Processing procedure in annotation device>
8 is a flowchart showing an example of a processing procedure in the annotation device according to the second embodiment of the present disclosure. In FIG. 8, the processing of steps S100 to S115 and the processing of steps S120 to S130 are the same as those in the first embodiment (FIG. 3), and therefore a description thereof will be omitted.

As shown in FIG. 8, in step S200, the feature detection unit 18 detects upstream difference region features from the upstream image I1 and downstream difference region features from the downstream image I2.

In step S205, the adoption determination unit 16 calculates the degree of difference (hereinafter sometimes referred to as "feature difference") between the upstream differential area feature and the downstream differential area feature based on the feature information.

Next, in step S210, the adoption determination unit 16 determines whether to adopt the training data based on the feature dissimilarity.

When the feature dissimilarity is greater than or equal to the second threshold TH2 (step S210: Yes), the adoption determination unit 16 adopts the candidate data as training data and stores the candidate data as adopted data in the training data storage unit 17 (step S140).

On the other hand, when the feature dissimilarity is less than the second threshold TH2 (step S210: No), the adoption determination unit 16 does not adopt the candidate data as training data, and deletes the unadopted candidate data from the candidate data storage unit 15.

<Operation of the annotation device>
FIG. 9 is a diagram illustrating an example of the operation of the annotation device according to the second embodiment of the present disclosure.

During waste sorting, the sorter M's hands may touch waste other than the desired waste (hereinafter referred to as "undesired waste"), causing the undesired waste to shift in position relative to the belt conveyor 40. Examples of undesired waste that may shift in position during waste sorting include waste adjacent to the desired waste and waste in the path of the sorter M's hands. For example, if the desired waste in the waste sorting is brown bottles, and the sorter M's hands touch bottles of other colors while sorting the brown bottles, the position of the bottle image B2 of other colors may differ between the first upstream image I1a and the third downstream image I2c included in the third image set IS3, as shown in FIG. 9. If there is a difference in the position of the bottle image B2 of other colors between the first upstream image I1a and the third downstream image I2c, the difference region determination unit 13 will determine a third difference region DRc, and the annotation unit 14 will annotate the third difference region DRc in the first upstream image I1a. However, since the other color bottles are undesired waste, annotation should not be performed on the third difference region DRc.

In this regard, in FIG. 9, the feature detection unit 18 detects from the first upstream image I1a the image features of the first differential region DRa (hereinafter sometimes referred to as "first upstream differential region features") in the first upstream image I1a and the image features of the third differential region DRc (hereinafter sometimes referred to as "second upstream differential region features") in the first upstream image I1a. Also, in FIG. 9, the feature detection unit 18 detects from the third downstream image I2c the image features of the first differential region DRa (hereinafter sometimes referred to as "first downstream differential region features") and the image features of the third differential region DRc (hereinafter sometimes referred to as "second downstream differential region features") in the third downstream image I2c.

Next, the adoption determination unit 16 calculates the dissimilarity FD1 between the first upstream differential region feature and the first downstream differential region feature (hereinafter sometimes referred to as the "first feature dissimilarity") and the dissimilarity FD2 between the second upstream differential region feature and the second downstream differential region feature (hereinafter sometimes referred to as the "second feature dissimilarity"). Then, if both the first feature dissimilarity FD1 and the second feature dissimilarity FD2 are equal to or greater than the second threshold TH2, the adoption determination unit 16 adopts the candidate data as training data. On the other hand, if either the first feature dissimilarity FD1 or the second feature dissimilarity FD2 is less than the second threshold TH2, the adoption determination unit 16 does not adopt the candidate data as training data. This prevents an upstream image associated with erroneous annotation data from being adopted as training data.

In the above description of Example 2, the decision to adopt training data was made based on the feature dissimilarity. However, instead of making the decision to adopt training data based on the feature dissimilarity, the annotation unit 14 may annotate difference regions whose feature dissimilarity is equal to or greater than the second threshold value TH2, while not annotating difference regions whose feature dissimilarity is less than the second threshold value TH2.

The above describes Example 2.

[Example 3]
In Example 3, the determination of whether training data is adopted is made based on the image features of the difference region, as in Example 2, but the difference from Example 2 is that the feature dissimilarity between two different image sets is calculated. The differences from Example 2 will be described below.

<Configuration of annotation device>
The configuration example of the annotation device of the third embodiment is the same as that of the second embodiment (FIG. 7), and therefore the description thereof will be omitted.

<Processing procedure in annotation device>
In the third embodiment, the processes of steps S100 to S110 in Fig. 8 are repeated to acquire two image sets, and the processes of steps S115, S200, and S120 to S130 in Fig. 8 are performed on the two consecutive image sets, after which the processes of step S205 and subsequent steps are performed. Hereinafter, of the two consecutive image sets, the former set may be referred to as the "former image set" and the latter set may be referred to as the "latter image set."

By repeating the processing of steps S100 to S110, the image acquisition unit 11 acquires the former image set and then the latter image set. Therefore, in step S130, the candidate data generated based on the former image set (hereinafter sometimes referred to as "former candidate data") and the candidate data generated based on the latter image set (hereinafter sometimes referred to as "latter candidate data") are stored in the candidate data storage unit 15.

In step S200, the feature detection unit 18 detects upstream difference region features from the upstream image I1 included in the former image set, and detects downstream difference region features from the downstream image I2 included in the latter image set.

In step S205, the adoption determination unit 16 calculates the feature dissimilarity between the upstream difference area features in the former image set and the downstream difference area features in the latter image set.

In step S210, the adoption determination unit 16 determines whether to adopt the training data based on the feature dissimilarity. If the feature dissimilarity is greater than or equal to the threshold value TH2 (step S210: Yes), processing proceeds to step S140. If the feature dissimilarity is less than the threshold value TH2 (step S210: No), processing proceeds to step S145.

In step S140, the adoption determination unit 16 adopts both the former candidate data and the latter candidate data as training data, and stores the adopted data, the former candidate data and the latter candidate data, in the training data storage unit 17.

On the other hand, in step S145, the adoption determination unit 16 does not adopt either the former candidate data or the latter candidate data as training data, and deletes the former candidate data and the latter candidate data, which are unadopted data, from the candidate data storage unit 15.

<Operation of the annotation device>
FIG. 10 is a diagram illustrating an example of the operation of the annotation device according to the third embodiment of the present disclosure.

When sorting waste, sorter M may accidentally pick up undesired waste. Furthermore, sorter M may realize a few seconds (e.g., three seconds) after picking up the undesired waste and return the undesired waste to the belt conveyor 40. For example, if the conveying speed of the belt conveyor 40 is 170 mm per second, the belt conveyor 40 will move approximately 50 cm in three seconds. Therefore, if the angle of view of the downstream camera 30 is small, downstream image I2 of the former image set may not include an image of the undesired waste (hereinafter sometimes referred to as a "undesired waste image"), while downstream image I2 of the latter image set may include an undesired waste image.

For example, suppose brown bottles are desired waste, and sorter M mistakenly picks up bottles of other colors, which are undesired waste, and then returns them to the belt conveyor 40.

In this case, if sorter M accidentally picks up a bottle of a different color, as shown in FIG. 10, of the third upstream image I1c and fourth downstream image I2d included in the former image set IS4, the third upstream image I1c includes the bottle image B3 of the different color and the brown bottle image B4, while the fourth downstream image I2d includes the brown bottle image B4 but not the bottle image B3 of the different color. In other words, in the former image set IS4, the bottle image B3 of the different color is included in the third upstream image I1c but not in the fourth downstream image I2d. As a result, the difference region determination unit 13 determines the fourth difference region DRd, and the annotation unit 14 annotates the fourth difference region DRd in the third upstream image I1c.

Furthermore, when sorter M returns the other-colored bottle that he accidentally picked up back onto the conveyor belt 40, as shown in FIG. 10, of the fourth upstream image I1d and fifth downstream image I2e included in the latter image set IS5, the fourth upstream image I1d includes the other-colored bottle image B5 and the transparent bottle image (hereinafter sometimes referred to as the "transparent bottle image") B6, but does not include the other-colored bottle image B7, while the fifth downstream image I2e includes the other-colored bottle image B5, the transparent bottle image B6, and the other-colored bottle image B7. In other words, in the latter image set IS5, the other-colored bottle image B7 is not included in the fourth upstream image I1d but is included in the fifth downstream image I2e. As a result, the difference region determination unit 13 determines the fifth difference region DRe, and the annotation unit 14 annotates the fifth difference region DRe in the fourth upstream image I1d.

However, since the other colored bottles are undesired waste that sorter M mistakenly picked up and then returned to the belt conveyor 40, annotation should not be performed on the fifth difference region DRe.

In this regard, in Figure 10, the feature detection unit 18 detects the image features of the fourth difference region DRd in the third upstream image I1c of the former image set IS4 (hereinafter, sometimes referred to as the "third upstream difference region feature") from the third upstream image I1c. Also, in Figure 10, the feature detection unit 18 detects the image features of the fifth difference region DRe in the fifth downstream image I2e of the latter image set IS5 (hereinafter, sometimes referred to as the "third downstream difference region feature") from the fifth downstream image I2e.

Next, the adoption determination unit 16 calculates the dissimilarity FD3 between the third upstream differential region feature and the third downstream differential region feature (hereinafter sometimes referred to as the "third feature dissimilarity") FD3. Then, when the third feature dissimilarity FD3 is equal to or greater than the second threshold TH2, the adoption determination unit 16 adopts both the former candidate data and the latter candidate data as training data. On the other hand, when the third feature dissimilarity FD3 is less than the second threshold TH2, the adoption determination unit 16 does not adopt the former candidate data or the latter candidate data as training data. This prevents an upstream image associated with erroneous annotation data from being adopted as training data.

In the above description of Example 3, the decision to adopt training data was made based on the feature dissimilarity. However, instead of making the decision to adopt training data based on the feature dissimilarity, the annotation unit 14 may annotate difference regions whose feature dissimilarity is equal to or greater than the second threshold value TH2, while not annotating difference regions whose feature dissimilarity is less than the second threshold value TH2.

In addition, the threshold value TH2 used in Example 3 may be set to a value greater than the threshold value TH2 used in Example 2. By doing so, in Example 3, the proportion of candidate data adopted as training data can be reduced compared to Example 2.

The above describes Example 3.

In addition, the above Examples 1, 2, and 3 can also be combined as appropriate and implemented in parallel.

[Example 4]
The image acquisition unit 11 is realized as hardware, for example, by a network interface module. The image storage unit 12, the candidate data storage unit 15, and the teacher data storage unit 17 are realized as hardware, for example, by memory or storage. The difference region determination unit 13, the annotation unit 14, the adoption determination unit 16, and the feature detection unit 18 are realized as hardware, for example, by a processor such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or an ASIC (Application Specific Integrated Circuit).

The above describes Example 4.

As described above, the annotation device disclosed herein (annotation devices 10, 10a, and 10b in the examples) includes an acquisition unit (image acquisition unit 11 in the examples), a first determination unit (difference region determination unit 13 in the examples), an annotation unit (annotation unit 14 in the examples), and a second determination unit (adoption determination unit 16 in the examples). The acquisition unit acquires a first image (upstream image in the examples) captured at a first timing before an operation to select a desired object from a group of objects on a transport path along which the group of objects is transported is performed, at a first position upstream of the position where the operation is performed, and a second image (downstream image in the examples) captured at a second timing after the operation is performed, at a second position downstream of the position where the operation is performed. The first determination unit compares the first image with the second image and determines a difference region, which is an area where there is a difference between the first image and the second image. The annotation unit assigns contour information, which is information indicating the contour of the difference region, to the difference region in the first image, and generates candidate data including the first image and the contour information. The second determination unit determines whether or not the candidate data is to be adopted as training data for machine learning.

For example, the second judgment unit adopts the candidate data as training data when the area of the difference region is less than a threshold, and does not adopt the candidate data as training data when the area of the difference region is greater than or equal to the threshold.

For example, the second judgment unit adopts the candidate data as training data when the degree of difference between the features of the first region image, which is an image of the difference region in the first image, and the features of the second region image, which is an image of the difference region in the second image, is greater than or equal to a threshold, and does not adopt the candidate data as training data when the degree of difference between the features is less than the threshold.

For example, the acquisition unit acquires a first set of a first image and a second image, and then acquires a second set of the first image and the second image. When the degree of difference between the features of a first region image, which is an image of a difference region in a first image included in the first set, and the features of a second region image, which is an image of a difference region in a second image included in the second set, is equal to or greater than a threshold, the second determination unit adopts, as training data, first candidate data, which is candidate data generated based on the first set, and second candidate data, which is candidate data generated based on the second set. On the other hand, when the degree of difference between the features is less than the threshold, the second determination unit does not adopt the first candidate data or the second candidate data as training data.

This allows annotation to be performed automatically through object sorting, which requires less effort than manual annotation based on visual inspection of images, thereby improving the efficiency of annotation work. Furthermore, when annotation is performed manually based on visual inspection of images, the accuracy of the annotation varies depending on the ability of the worker. In contrast, automatic annotation through the simple task of object sorting, which does not result in differences in the quality of the work, allows for highly accurate annotation regardless of the ability of the worker performing the object sorting work. Furthermore, it prevents images with erroneous annotations from being used as training data. In particular, when annotation is performed automatically through the task of selecting desired objects from a densely packed group of objects, highly accurate annotation can be performed, preventing the generation of erroneous training data.

1 Annotation systems 10, 10a, 10b Annotation device 20 Upstream camera 30 Downstream camera 40 Belt conveyor 11 Image acquisition unit 12 Image storage unit 13 Difference region determination unit 14 Annotation unit 15 Candidate data storage unit 16 Adoption determination unit 17 Teacher data storage unit 18 Feature detection unit

Claims (4)

an acquisition unit that acquires a first image taken at a first position upstream of a position where a desired object is selected from a group of objects on a conveyance path along which the group of objects is conveyed, at a first timing before the operation is performed, and a second image taken at a second position downstream of the position where the operation is performed, at a second timing after the operation is performed;
a first determination unit that compares the first image with the second image and determines a difference region that is a region where there is a difference between the first image and the second image;
an annotation unit that assigns contour information, which is information indicating a contour of the difference region, to the difference region in the first image and generates candidate data including the first image and the contour information;
a second determination unit that determines whether or not to adopt the candidate data as training data for machine learning;
An annotation device comprising: The second determination unit
When the area of the difference region is less than a threshold, the candidate data is adopted as the training data;
When the area is equal to or larger than the threshold, the candidate data is not adopted as the training data.
The annotation device according to claim 1 . The second determination unit
adopting the candidate data as the training data when a degree of difference between a feature of a first region image that is an image of the difference region in the first image and a feature of a second region image that is an image of the difference region in the second image is equal to or greater than a threshold;
When the degree of difference is less than the threshold, the candidate data is not adopted as the training data.
The annotation device according to claim 1 . the acquisition unit acquires a second set of the first image and the second image after acquiring a first set of the first image and the second image;
The second determination unit
When a degree of difference between a feature of a first region image, which is an image of the difference region in the first image included in the first set, and a feature of a second region image, which is an image of the difference region in the second image included in the second set, is equal to or greater than a threshold, first candidate data, which is the candidate data generated based on the first set, and second candidate data, which is the candidate data generated based on the second set, are adopted as the teacher data;
When the degree of difference is less than the threshold, the first candidate data and the second candidate data are not adopted as the training data.
The annotation device according to claim 1 .