JP7776129B2 - Foreign object detection teacher data generation device, trained model manufacturing device, foreign object detection device, foreign object detection teacher data generation method, and program - Google Patents

Description

The present invention relates to a device for generating training data for foreign object detection, a trained model production device, a foreign object detection device, a method for generating training data for foreign object detection, and a program.

A known method for detecting and classifying foreign objects contained in a detection target is a foreign object inspection method that processes image data using supervised machine learning (Patent Document 1).

Japanese Patent Application Laid-Open No. 2008-14652

However, supervised classification requires a large amount of training data. Preparing a large amount of training data requires labeling the location and type of foreign objects in images containing them, which poses the problem of requiring a great deal of time and effort to do manually.

The present invention therefore aims to provide a device for generating training data for foreign object detection that can generate training data for foreign object detection without requiring labeling work.

In order to achieve the above object, the foreign object detection training data generating device of the present invention comprises:
The apparatus includes a detection object image acquisition unit, a foreign substance image acquisition unit, an image synthesis unit, and a training data output unit,
the detection target image acquisition unit acquires a detection target image obtained by capturing an image of the foreign object detection target after the foreign object has been removed,
the foreign substance image acquisition unit acquires a foreign substance image by capturing an image of the foreign substance;
the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image;
The teacher data output unit
A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

The trained model production device of the present invention comprises:
The system includes a training data acquisition unit and a trained model generation unit,
the teacher data acquisition unit acquires, as the teacher data for foreign object detection, the teacher data for foreign object detection output by the teacher data generation device for foreign object detection of the present invention;
The trained model generation unit generates, through machine learning using the foreign object detection training data, a foreign object detection model that, when inputted, outputs whether or not the foreign object detection target contains a foreign object, as a trained model.

The foreign object detection device of the present invention comprises:
a foreign object detection target image acquisition unit and a foreign object detection unit,
the foreign object detection target image acquisition unit acquires a foreign object detection target image by capturing an image of the foreign object detection target,
the foreign object detection unit inputs the foreign object detection target image into a foreign object detection model and detects whether or not the foreign object detection target includes a foreign object;
The foreign object detection model is a trained model generated by machine learning using training data generated by the foreign object detection training data generation device of the present invention, so that when an image of the foreign object detection target is input, it outputs whether or not the foreign object detection target contains a foreign object, or it is a trained model manufactured by the trained model manufacturing device of the present invention.

The method for generating training data for foreign object detection according to the present invention includes the steps of:
The method includes a detection object image acquisition step, a foreign substance image acquisition step, an image synthesis step, and a training data output step,
the detection target image acquisition step acquires a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed;
the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance;
the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image;
The teacher data output step includes:
A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

The trained model production method of the present invention includes:
The method includes a training data acquisition step and a trained model generation step.
the teacher data acquisition step acquires, as the teacher data for foreign object detection, the teacher data for foreign object detection output by the method for generating teacher data for foreign object detection of the present invention;
The trained model generation process uses machine learning using the foreign object detection training data to generate a trained model of a foreign object detection model that, when an image of a foreign object detection target is input, outputs whether or not the foreign object detection target contains a foreign object.

The foreign object detection method of the present invention comprises:
The method includes a foreign object detection image acquisition step and a foreign object detection step,
the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target,
the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model and detecting whether or not a foreign object is included in the foreign object detection target;
The foreign object detection model is a trained model generated by machine learning using training data generated by the foreign object detection training data generation method of the present invention, so that when a foreign object detection target image of the foreign object detection target is input, it outputs whether or not the foreign object detection target contains a foreign object, or is a trained model produced by the trained model production method of the present invention.

The first program of the present invention includes a detection object image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure,
the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed;
the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance;
the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image;
The teacher data output procedure includes:
outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance;
This is a program for causing a computer to execute each of the above procedures.

A second program of the present invention includes a training data acquisition procedure and a trained model generation procedure,
the teacher data acquisition step acquires teacher data for foreign object detection output by the first program of the present invention as teacher data for foreign object detection;
the trained model generation step generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, by machine learning using the foreign object detection training data;
This is a program for causing a computer to execute each of the above procedures.

A third program of the present invention includes a foreign object detection image acquisition procedure and a foreign object detection procedure,
the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target;
the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object;
the foreign object detection model is a trained model generated by machine learning using training data generated by the first program of the present invention so as to output whether or not a foreign object is included in a foreign object detection target when a foreign object detection target image obtained by capturing an image of the foreign object detection target is input, or is a trained model produced by the second program of the present invention;
This is a program for causing a computer to execute each of the above procedures.

This invention makes it possible to generate training data for foreign object detection without the need for labeling.

FIG. 1 is a block diagram showing an example of the configuration of a foreign object detection training data generation device according to the first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of the foreign object detection training data generation device according to the first embodiment. FIG. 3 is a flowchart showing an example of processing in the foreign object detection training data generation device of the first embodiment. FIG. 4A is a schematic diagram showing a foreign substance composite image, and FIG. 4B is a schematic diagram for explaining the composite position of a foreign substance in the foreign substance composite image. FIG. 5 is a block diagram showing an example of the configuration of a foreign object detection training data generating device according to the second embodiment. FIG. 6 is a flowchart showing an example of processing in the foreign object detection training data generation device of the second embodiment. FIG. 7 is a block diagram showing an example of the configuration of a foreign object detection training data generating device according to the third embodiment. FIG. 8 is a flowchart showing an example of processing in the foreign object detection training data generation device of the third embodiment. FIG. 9 is a block diagram showing an example of the configuration of a trained model production device according to the fourth embodiment. FIG. 10 is a block diagram showing an example of the hardware configuration of a trained model production device according to the fourth embodiment. FIG. 11 is a flowchart showing an example of processing in the trained model production device of the fourth embodiment. FIG. 12 is a block diagram showing an example of the configuration of a foreign object detection device according to the fifth embodiment. FIG. 13 is a block diagram illustrating an example of the hardware configuration of a foreign object detection device according to the fifth embodiment. FIG. 14 is a flowchart showing an example of processing in the foreign object detection device of the fifth embodiment.

Next, embodiments of the present invention will be described using the drawings. The present invention is not limited to the following embodiments. In each of the following drawings, identical parts are designated by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
The foreign object detection training data generation device of this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of a foreign object detection training data generation device 10 of this embodiment. As shown in Fig. 1, the foreign object detection training data generation device 10 (hereinafter also referred to as "the device 10") includes a detection object image acquisition unit 11, a foreign object image acquisition unit 12, an image synthesis unit 13, and a training data output unit 14. Although not shown, the device 10 may also include, for example, a storage unit.

The device 10 may be, for example, a single device including each of the above-mentioned components, or a device in which each of the above-mentioned components can be connected via a communication network. The device 10 can also be connected to external devices (described later) via a communication network. The communication network is not particularly limited, and any known network can be used, and may be wired or wireless. Examples of communication networks include the Internet, the World Wide Web (WWW), telephone lines, LANs (Local Area Networks), SANs (Storage Area Networks), DTNs (Delay Tolerant Networking), LPWAs (Low Power Wide Area Networks), and L5Gs (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be in the form of direct communication between devices (ad hoc communication), infrastructure communication, or indirect communication via an access point. The device 10 may be incorporated into a server as a system. The device 10 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 10 may be in the form of cloud computing or edge computing, in which at least one of the components is located on a server and the other components are located on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a CPU 101, memory 102, a bus 103, a storage device 104, an input device 106, an output device 107, and a communication device (communication unit) 108. Each unit of the device 10 is connected to each other via the bus 103 and their respective interfaces (I/F).

The CPU 101 operates in conjunction with other components, for example, via a controller (system controller, I/O controller, etc.), and is responsible for overall control of the device 10. In the device 10, the CPU 101 executes, for example, the program 105 of the present invention and other programs, and also reads and writes various types of information. Specifically, the CPU 101 functions, for example, as a detection target image acquisition unit 11, a foreign substance image acquisition unit 12, an image synthesis unit 13, and a training data output unit 14. The device 10 includes a CPU as a computing device, but may also include other computing devices such as a GPU (Graphics Processing Unit) or APU (Accelerated Processing Unit), or may include a combination of the CPU and these.

The bus 103 can also be connected to, for example, external devices. Examples of such external devices include a trained model production device (described below), a foreign object detection device, an external storage device (external database, etc.), a printer, an external input device, an external output device, an audio output device such as a speaker, an external imaging device such as a camera, and various sensors such as an acceleration sensor, a geomagnetic sensor, and a direction sensor. The device 10 can be connected to an external network (the above-mentioned communication line network) via, for example, a communication device 108 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the CPU 101 performs processing, the memory 102 reads various operating programs, such as the program 105 of the present invention, stored in the storage device 104 (described below), and the CPU 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read-only memory).

The storage device 104 is also referred to as an auxiliary storage device, in contrast to the main memory. As described above, the storage device 104 stores an operating program 105 including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive for reading and writing from and to the recording medium. The recording medium is not particularly limited and may be internal or external, and examples include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, and memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or solid state drive (SSD) that integrates the recording medium and drive. If the device 10 includes, for example, the storage unit, the storage device 104 functions as the storage unit. The storage device 104 may store, for example, a foreign object detection model, as described below.

In the device 10, the memory 102 and storage device 104 can also store various information such as log information, information obtained from an external database (not shown) or external device, information generated by the device 10, and information used by the device 10 when executing processing. Note that at least some of the information may be stored, for example, on an external server other than the memory 102 and storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology, etc.

The device 10 further includes, for example, an input device 106 and an output device 107. Examples of the input device 106 include pointing devices such as a touch panel, track pad, or mouse; a keyboard; imaging means such as a camera or scanner; card readers such as IC card readers or magnetic card readers; and audio input means such as a microphone. Examples of the output device 107 include display devices such as LED displays or liquid crystal displays; audio output devices such as speakers; and a printer. In this embodiment 1, the input device 106 and the output device 107 are configured separately, but the input device 106 and the output device 107 may also be configured as an integrated device, such as a touch panel display.

Next, an example of the method for generating training data for foreign object detection according to this embodiment will be described based on the flowchart in Figure 3. The method for generating training data for foreign object detection according to this embodiment can be implemented, for example, as follows, using the device for generating training data for foreign object detection 10 shown in Figure 1 or Figure 2. Note that the method for generating training data for foreign object detection according to this embodiment is not limited to use with the device for generating training data for foreign object detection 10 shown in Figure 1 or Figure 2. The object to be detected for foreign objects in this invention is not particularly limited as long as it is an object that can be detected for foreign objects, and examples include food waste; products on a production line; agricultural products, etc. In the following description, the object to be detected for foreign objects will be described using food waste from a biogas plant as an example, but the present invention is not limited in any way to the following example.

First, the detection target image acquisition unit 11 acquires a detection target image capturing an image of the foreign object detection target after the foreign object has been removed (S1, detection target image acquisition process). The detection target image may be, for example, an image of the foreign object detection target from which the foreign object has been completely removed, or an image of the foreign object detection target from which the foreign object still remains. However, an image of the foreign object detection target from which the foreign object has been completely removed is preferable. The detection target image may be, for example, a still image, a video, or a still image extracted from a video. The detection target image acquisition unit 11 may, for example, acquire images continuously or intermittently. In the latter case, images may be acquired every predetermined time period or at any timing. The detection target image acquisition unit 11 may, for example, acquire the detection target image by capturing an image of the foreign object detection target using the imaging device serving as the input device 106. Alternatively, the detection target image may be acquired from an external imaging device via the communication network using the communication device 108. As a specific example of the latter, the detection target image can be acquired by capturing an image taken by a camera capturing an image of the process of removing foreign matter from food waste at a food waste sorting area in a biogas plant. The detection target image acquisition unit 11 may, for example, store the acquired detection target image in memory 102 or storage device 104.

The detection target image acquisition unit 11 may also include, for example, a detection target imaging unit and a difference target image extraction unit. In this case, the detection target imaging unit captures an image of the detection target, for example, at predetermined time intervals. The difference target image extraction unit then extracts the difference between, for example, a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, to generate a difference target image. The second target image of the detection target captured the (n-1)th time may be stored, for example, in the storage device 104, or in an external storage device outside the device. In the latter case, for example, the difference target image extraction unit acquires the second target image from the external storage device and generates the difference target image. The extraction of the difference can be performed, for example, by a difference extraction method in a known image analysis technique. The difference target image extraction unit then acquires the first target image as the detection target image, for example, when the area of the difference target image is equal to or greater than a threshold. The threshold is not particularly limited and may be, for example, the size of the difference target image or the ratio of the area of the difference target image to the area of the first target image. For example, if the area is equal to or greater than the threshold, the difference target image extraction unit may store the first target image in storage device 104 or memory 102 as the detection target image, and if the area is less than the threshold, discard the first target image without storing it. This configuration makes it possible to acquire, for example, only images with large differences, i.e., images in which the detection target has changed, thereby generating more diverse training data and, for example, conserving device storage capacity.

Next, the foreign object image acquisition unit 12 acquires a foreign object image (S2, foreign object image acquisition process). The foreign object image is, for example, an image of a foreign object removed from the detection target, and may be an image of a single type of foreign object or an image containing multiple types of foreign objects. The foreign object image may also be, for example, an image of a foreign object actually removed from the detection target, or may include images of foreign objects expected to be contained in the detection target. However, an image of a foreign object actually removed from the detection target is preferable. The foreign object image may, for example, be a still image, a video, or a still image extracted from a video. The foreign object image acquisition unit 12 may, for example, acquire images continuously or intermittently. In the latter case, images may be acquired at predetermined time intervals or at any timing. The foreign object image acquisition unit 11 may acquire the foreign object image by, for example, capturing an image of the foreign object using the imaging device that is the input device 106, or may acquire the foreign object image from an external imaging device via the communication network using the communication device 108. As a specific example of the latter, the foreign object image may be acquired by capturing an image taken by a camera that captures images of foreign objects removed from food waste at a food waste sorting area in a biogas plant. The foreign object image acquisition unit 12 may store the acquired foreign object image in the memory 102 or the storage device 104, for example.

The foreign substance image acquisition unit may include, for example, a foreign substance imaging unit, a differential foreign substance image extraction unit, and a foreign substance determination unit. In this case, for example, the foreign substance imaging unit images the foreign substance at predetermined time intervals. The differential foreign substance image extraction unit generates a differential foreign substance image by extracting the difference between a first foreign substance image captured the nth time and a second foreign substance image captured the n-1th time. The difference can be extracted, for example, using a difference extraction method in a well-known image analysis technique. For example, if the area of the differential foreign substance image is within a threshold range, the foreign substance determination unit determines that the differential foreign substance image is an image of a foreign substance and acquires the differential foreign substance image as the foreign substance image. The threshold range is not particularly limited and may, for example, be the range of the size of the differential foreign substance image or the range of the ratio of the area of the differential foreign substance image to the area of the first foreign substance image. The threshold range can be set appropriately, for example, depending on the size and type of foreign substances that may be contained in the detection target. For example, if the area is within a threshold range, the foreign object determination unit may store the difference image as the foreign object image in storage device 104 or memory 102, and further store the first foreign object image in storage device 104 or memory 102 as a second foreign object image for subsequent determinations. Alternatively, if the area is outside the threshold range, the foreign object determination unit may discard the first foreign object image and the difference threshold image rather than store them. If the detection target is food waste and the foreign object image is an image of a foreign object removed from the food waste captured at a food waste sorting area, if the area of the difference foreign object image is below a certain level, the image captured by the foreign object image capture unit is likely to not contain a new foreign object. Furthermore, if the area of the difference foreign object image is above a certain level, the image captured by the foreign object image capture unit is likely to contain an object other than a foreign object (e.g., a part of the body of a worker removing the foreign object). According to this aspect, for example, such images can be removed from the acquired foreign object images, thereby generating higher-quality training data.

The camera that captures the detection target and the camera that captures the foreign object may be, for example, the same camera or different cameras.

Next, the image composition unit 13 combines the foreign object image with the detection target image to generate a foreign object composite image (S3, image composition process). Specifically, the image composition unit 13 can generate the foreign object composite image by, for example, randomly selecting the detection target image and the foreign object image and combining the selected detection target image with the foreign object image. The image composition unit 13 can combine, for example, one foreign object image with one detection target image, or two or more foreign object images. For example, the position (combining position) at which the foreign object image is combined in the detection target image is not particularly limited, and the combination can be performed at any position. The image composition unit 13 can generate multiple foreign object composite images by changing the combining position for a pair of the detection target image and the foreign object image. The image composition unit 13 can also perform processing such as changing the angle or size of the foreign object image or flipping it, and combine the processed foreign object image with the detection target image.

The composition position can be identified, for example, by dividing the foreign substance composite image into multiple pixel elements and then specifying a two-dimensional matrix with the same number and arrangement as the pixel elements. Figure 4 shows a specific example of the foreign substance composite image 131 and its composition position. Figure 4(A) is a schematic diagram of the foreign substance composite image 131, showing that a foreign substance image 131b is composed in the lower right corner of the detection target image 131a. As shown in Figure 4(B), the composition position can be identified by dividing the foreign substance composite image 131 into multiple pixel elements 131c and then specifying a two-dimensional matrix with the same number and arrangement as the pixel elements 131c, by recording "0" for matrix elements 131d corresponding to pixel elements 131c where no foreign substance image 131b is present and "1" for matrix elements 131d corresponding to pixel elements 131c where a foreign substance image 131b is present. For ease of explanation, in Figure 4, matrix elements corresponding to pixel elements where no foreign matter is present are shown as "0" and matrix elements corresponding to pixel elements where a foreign matter is present are shown as "1". However, the present invention is not limited to this, and it is sufficient if the matrix elements corresponding to positions where a foreign matter is present and positions where no foreign matter is present have different numerical values.

Then, the training data output unit 14 outputs a pair of the composite position of the foreign substance image in the foreign substance composite image as training data for the presence of a foreign substance (S4, training data output step). Alternatively, the training data output unit 14 may output, for example, the detection target image as training data for the absence of a foreign substance. The output may be, for example, output (storage) to the memory 102 or storage device 104 of the device 10, or output to an external device via a communications network. The external device may be, for example, an external storage device or a device that uses the training data generated by the device 10, specifically, a trained model production device or foreign substance detection device of the present invention, which will be described later.

In the method for generating training data for foreign object detection of this embodiment, an example has been described in which S1 and S2 are executed, followed by S3 and S4, but the present invention is not limited to this. Specifically, in the present invention, S1 and S2 may be executed in a process upstream of S3, and S1 and S2 may be executed simultaneously or separately, for example. In the latter case, the order of execution is not particularly limited and is arbitrary.

The foreign object detection training data generation device of this embodiment separately acquires a detection target image of the detection target and a foreign object image of the foreign object, and then combines these to generate foreign object detection training data that includes a large number of pairs of foreign object composite images and foreign object composite positions, i.e., the positions of foreign objects in the foreign object composite images. This allows the foreign object detection training data generation device of this embodiment to generate a large number of training data for foreign object detection without manual labeling.

[Embodiment 2]
The second embodiment is another example of the foreign object detection training data generating device of the present invention.

The foreign object detection training data generation device of this embodiment is similar to the foreign object detection training data generation device 10 of the first embodiment, except that it includes a change information acquisition unit in addition to the configuration of the foreign object detection training data generation device 10 of the first embodiment, and the description therefor can be used. The foreign object detection training data generation device 10A of this embodiment includes, for example, a change information acquisition unit that acquires change information detecting a change in at least one of the detection target and the foreign object, the detection target image acquisition unit acquires the detection target image when it acquires the change information of the detection target, and the foreign object image acquisition unit acquires the foreign object image when it acquires the change information of the foreign object.

Figure 5 is a block diagram showing an example configuration of a foreign object detection training data generation device 10A according to this embodiment. As shown in Figure 5, the foreign object detection training data generation device 10A includes a change information acquisition unit 15 in addition to the configuration of the foreign object detection training data generation device 10 according to embodiment 1. The hardware configuration of the foreign object detection training data generation device 10A is the same as that of the foreign object detection training data generation device 10 of Figure 2, except that the CPU 101 includes the configuration of the foreign object detection training data generation device 10A of Figure 5 instead of the configuration of the foreign object detection training data generation device 10 of Figure 1.

Next, the method for generating training data for foreign object detection of this embodiment will be described using the flowchart in Figure 6. The method for generating training data for foreign object detection of this embodiment can be implemented, for example, using the device for generating training data for foreign object detection 10A of this embodiment shown in Figure 5. Note that the method for generating training data for foreign object detection of the present invention is not limited to use with the device for generating training data for foreign object detection 10A.

First, the change information acquisition unit 15 acquires change information that detects a change in at least one of the detection target and the foreign object (S6, change information acquisition process). The change information is information indicating a change in the state of the detection target or foreign object. Specific examples of the change information include weight change information of the detection target or foreign object, distance change information of the detection target or foreign object, volume change information of the detection target or foreign object, and acceleration information of the container containing the detection target or foreign object. However, the change information is not particularly limited as long as it indicates a change in the state of the detection target or foreign object. The change information can be acquired by a sensor that continuously measures the state of the detection target or foreign object, such as a weigh scale, distance sensor, volume measuring device, or acceleration sensor. Below, we will explain the case of acquiring weight change information as a specific example of the change information, but the present invention is not limited to this. The change information acquisition unit 15 may, for example, measure the weight of the detection object or the foreign object over time, and if there is a change in the measurement value, acquire the changed measurement value as the change information, or may acquire a signal notifying that there has been a change in weight of at least one of the detection object and the foreign object from a wired or wirelessly connected weighing scale external to the device as the change information. As a specific example, at a food waste sorting area in a biogas plant, weighing scales may be installed in both trash bins for disposing of food waste (detection object) and trash bins for disposing of foreign objects sorted from the food waste, and change information on the detection object and the foreign object can be acquired by communicating with the weighing scales to acquire weight information measured by the weighing scales.

When the detection target image acquisition unit 11 acquires change information about the detection target, it acquires the detection target image in the same manner as S1 in the first embodiment. Furthermore, an image captured immediately after a change in the weight of the detection target is likely to capture, for example, the cause of the change in the state of the detection target (for example, a part of the body of the person who disposed of the food waste). Therefore, the detection target image acquisition unit 11 may acquire the detection target image, for example, a predetermined time after acquiring the change information.

When the foreign object image acquisition unit 12 acquires the change information of the foreign object, it acquires the foreign object image in the same manner as S2 in the first embodiment. Furthermore, an image captured immediately after a change in the weight of the foreign object is likely to capture, for example, the cause of the change in the state of the foreign object (e.g., a part of the body of the person who discarded the foreign object). Therefore, the foreign object image acquisition unit 12 may acquire the foreign object image, for example, a predetermined time after acquiring the change information.

Then, steps S3 and S4 are carried out in the same manner as steps S3 and S4 in embodiment 1, and the process ends (END).

In the foreign object detection training data generation device of this embodiment, for example, a change information acquisition unit acquires change information that detects changes in at least one of the detection target and the foreign object, and the detection target image acquisition unit acquires the detection target image when it acquires the change information of the detection target, and the foreign object image acquisition unit acquires the foreign object image when it acquires the change information of the foreign object. Therefore, with the foreign object detection training data generation device of this embodiment, for example, detection target images and foreign object images for generating training data can be acquired more efficiently.

[Embodiment 3]
The third embodiment is another example of the foreign object detection training data generating device of the present invention.

The foreign object detection training data generation device of this embodiment is similar to the foreign object detection training data generation device 10 of the first embodiment, except that it includes an image processing unit in addition to the configuration of the foreign object detection training data generation device 10 of the first embodiment, and the description thereof can be used. The foreign object detection training data generation device 10B of this embodiment includes, for example, an image processing unit that processes the detection target image to generate a processed detection target image and a processed foreign object image by processing the foreign object image, and the image synthesis unit generates a foreign object composite image by synthesizing at least one of the processed detection target image and the processed foreign object image.

Figure 7 is a block diagram showing an example configuration of a foreign object detection training data generation device 10B of this embodiment. As shown in Figure 7, foreign object detection training data generation device 10B includes an image processing unit 16 in addition to the configuration of foreign object detection training data generation device 10 of embodiment 1. The hardware configuration of foreign object detection training data generation device 10B is the same as the hardware configuration of foreign object detection training data generation device 10 of Figure 2, except that the CPU 101 includes the configuration of foreign object detection training data generation device 10B of Figure 7 instead of the configuration of foreign object detection training data generation device 10 of Figure 1.

Next, the method for generating training data for foreign object detection of this embodiment will be described using the flowchart in Figure 8. The method for generating training data for foreign object detection of this embodiment can be implemented, for example, using the device for generating training data for foreign object detection 10B of this embodiment shown in Figure 7. Note that the method for generating training data for foreign object detection of the present invention is not limited to use with the device for generating training data for foreign object detection 10B.

First, steps S1 and S2 are performed in the same manner as in embodiment 1, and an image of the detection target and an image of the foreign substance are obtained.

The image processing unit 16 processes the detection target image to generate a processed detection target image and the foreign substance image to generate a processed foreign substance image. The processing can be performed using, for example, an image data extension method used in creating training data using general image recognition. Specific examples include changing the color, size, tilt, perspective, etc. of the image, horizontal shifting, random shifting, horizontal flipping, vertical flipping, shearing, RGB channel conversion, background cutting, etc.
When the detection target is food waste and the foreign matter is a foreign matter removed from the food waste, the image processing unit 16 preferably generates a processed detection target image and a processed foreign matter image by performing color adjustment on the detection target image and the foreign matter image using a filter with a color tone specified for each time period. Furthermore, the image processing unit 16 may generate a processed foreign matter image by performing oblique deformation on the foreign matter image using a shear transformation process.

Next, the image composition unit 13 generates a foreign substance composite image by combining, for example, at least one of the processed detection target image and the processed foreign substance image (S3, image composition process). The composition can be performed in the same manner as S3 in the first embodiment, except that an image including at least one of the processed target image and the processed foreign substance image is used in addition to the detection target image and the foreign substance image.

Then, S4 is carried out in the same manner as S4 in embodiment 1, and the process ends (END).

The foreign object detection training data generation device of this embodiment can, for example, use an image processing unit to generate a processed detection target image by processing the detection target image and a processed foreign object image by processing the foreign object image. Therefore, with the foreign object detection training data generation device of this embodiment, training data can be generated using not only actually acquired images but also images that have been expanded through processing for the detection target image and foreign object image used to generate training data, making it possible to generate training data for foreign object detection more efficiently.

[Embodiment 4]
Embodiment 4 is an example of a trained model production device of the present invention.

The trained model production device of this embodiment will be described using Figure 9. Figure 9 is a block diagram showing an example configuration of a trained model production device 20 of this embodiment. As shown in Figure 9, the trained model production device 20 includes a teacher data acquisition unit 21 and a trained model generation unit 22. Although not shown, the trained model production device 20 may also include, for example, a memory unit.

The trained model production device 20 may be, for example, a single device including each of the above-mentioned units, or a device to which each of the above-mentioned units can be connected via a communication network. The trained model production device 20 can also be connected to an external device (described later) via a communication network. The communication network is not particularly limited, and any known network can be used, and may be wired or wireless, for example. Examples of communication networks include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), and L5G (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (ad hoc communication), infrastructure communication, indirect communication via an access point, or the like. The trained model production device 20 may be incorporated into a server as a system. The trained model production device 20 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the trained model production device 20 may be, for example, a form of cloud computing or edge computing in which at least one of the above-mentioned components is located on a server and the other components are located on a terminal.

Figure 10 illustrates a block diagram of the hardware configuration of the trained model manufacturing device 20. As shown in Figure 10, the trained model manufacturing device 20 includes, for example, a CPU 201, memory 202, bus 203, storage device 204, input device 206, output device 207, and communication device 208. The description of each component of the trained model manufacturing device 20 can be made using the description of each component of the foreign object detection training data generation device 10. Each unit of the trained model manufacturing device 20 is connected via the bus 203 and its respective interface (I/F). In the trained model manufacturing device 20, the CPU 201 functions as a training data acquisition unit 21 and a trained model generation unit 22.

Next, an example of a method for manufacturing a trained model according to this embodiment will be described based on the flowchart in Figure 11. The method for manufacturing a trained model according to this embodiment is carried out as follows, for example, using the trained model manufacturing device 20 of Figures 9 and 10. Note that the method for manufacturing a trained model according to this embodiment is not limited to use of the trained model manufacturing device 20 of Figures 9 and 10.

First, the teacher data acquisition unit 21 acquires the foreign object detection teacher data output by the foreign object detection teacher data generation device of the present invention as foreign object detection teacher data (S21, teacher data acquisition step). The teacher data acquisition unit 21 may acquire the foreign object detection teacher data from the foreign object detection teacher data generation device of the present invention via the communication network, for example, or may acquire the foreign object detection teacher data from an external storage device in which the foreign object detection data is stored.

Next, the trained model generation unit 21 generates a trained foreign object detection model through machine learning using the foreign object detection training data. The trained model outputs whether or not a foreign object is contained in a foreign object detection target when a foreign object detection target image is input (S22, training process). The machine learning is not particularly limited, and may be, for example, machine learning using a neural network such as a convolutional neural network (CNN), a support vector machine (SVM), a Bayesian network, a regression tree, or the like. The machine learning is preferably, for example, machine learning using a convolutional neural network, particularly semantic segmentation. Furthermore, the trained model generation unit 21 may, for example, generate a retrained trained model (derived model) using the foreign object detection training data and a previously generated trained model. Furthermore, the trained model generation unit 21 may generate a trained model obtained by transfer learning using a trained model generated using the training data for foreign object detection, or may generate the trained model by model compression of a trained model generated using the training data for foreign object detection.

The trained model generated by this embodiment is used, for example, in the foreign object detection device described below. This makes it possible to detect foreign objects within a foreign object detection target using a foreign object detection target image captured of the foreign object detection target.

[Embodiment 5]
The fifth embodiment is an example of a foreign object detection device of the present invention.

The foreign object detection device of this embodiment will be described using FIG. 12. FIG. 12 is a block diagram showing an example configuration of a foreign object detection device 30 of this embodiment. As shown in FIG. 12, the foreign object detection device 30 includes a foreign object detection target image acquisition unit 31 and a foreign object detection unit 32. Although not shown, the foreign object detection device 30 may also include, for example, a memory unit.

For example, the foreign object detection device 30 may be a single device including each of the above-mentioned components, or a device in which each of the above-mentioned components can be connected via a communication network. Furthermore, the foreign object detection device 30 can be connected to an external device (described later) via the communication network. The communication network is not particularly limited, and any known network can be used, and may be wired or wireless, for example. Examples of communication networks include the Internet, the World Wide Web (WWW), telephone lines, LANs (Local Area Networks), SANs (Storage Area Networks), DTNs (Delay Tolerant Networking), LPWAs (Low Power Wide Area Networks), and L5Gs (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (ad hoc communication), infrastructure communication, or indirect communication via an access point. The foreign object detection device 30 may be incorporated into a server as a system. The foreign object detection device 30 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the foreign object detection device 30 may be, for example, a cloud computing or edge computing system in which at least one of the above components is located on a server and the other components are located on a terminal.

Figure 13 shows an example block diagram of the hardware configuration of the foreign object detection device 30. As shown in Figure 13, the foreign object detection device 30 includes, for example, a CPU 301, memory 302, a bus 303, a storage device 304, an input device 306, an output device 307, and a communication device 308. The explanation of each component of the foreign object detection device 30 can be made using the explanation of each component of the foreign object detection training data generation device 10. Each unit of the foreign object detection device 30 is connected via the bus 303 and its respective interface (I/F). In the foreign object detection device 30, the CPU 301 functions as the foreign object detection target image acquisition unit 31 and the foreign object detection unit 32.

Next, an example of a method for manufacturing a trained model of this embodiment will be described based on the flowchart in Figure 14. The method for manufacturing a trained model of this embodiment is carried out as follows, for example, using the foreign object detection device 30 of Figures 12 and 13. Note that the method for manufacturing a trained model of this embodiment is not limited to using the foreign object detection device 30 of Figures 12 and 13.

First, the foreign object detection target image acquisition unit 31 acquires a foreign object detection target image by capturing an image of the foreign object detection target (S31, foreign object detection target image acquisition process). The foreign object detection target may, for example, contain a foreign object, may not contain a foreign object, or it may be unclear whether the foreign object is present. The foreign object detection target image may, for example, be a still image, a video, or a still image extracted from a video. The foreign object detection target image acquisition unit 31 may, for example, acquire images continuously or intermittently. In the latter case, images may be acquired every predetermined time period or at any timing. The foreign object detection target image acquisition unit 31 may, for example, acquire the foreign object detection target image by capturing an image of the foreign object detection target using the imaging device that is the input device 306. Alternatively, the foreign object detection target image may be acquired from an external imaging device via the communication network using the communication device 308. As a specific example of the latter, the foreign object detection target image can be acquired by capturing an image of food waste taken by a camera at a food waste disposal site in a biogas plant. The foreign object detection target image acquisition unit 31 may, for example, store the acquired foreign object detection target image in memory 302 or storage device 304.

The foreign object detection unit 32 inputs the foreign object detection target image into a foreign object detection model and detects whether the foreign object detection target contains a foreign object (S32, foreign object detection process). The foreign object detection model is a trained model generated, for example, by machine learning using training data generated by the foreign object detection training data generation device of the present invention, so that when the foreign object detection target image is input, it determines whether the foreign object detection target contains a foreign object, i.e., outputs a foreign object detection result. Note that the foreign object detection model may also be, for example, a trained model generated by the trained model generation device of embodiment 4.

The foreign object detection model may include, for example, an input layer that inputs a foreign object detection target image, an output layer that outputs the foreign object detection results, and at least one intermediate layer located between the input and output layers. The foreign object detection model may be a program module that is part of artificial intelligence software. Examples of the multi-layered network include neural networks. Examples of the neural network include convolutional neural networks (CNNs), but are not limited to CNNs and may include neural networks other than CNNs, as well as trained models constructed using other learning algorithms such as SVMs (Support Vector Machines), Bayesian networks, and regression trees.

The foreign object detection model can be generated, for example, by machine learning using the training data generated by the foreign object detection training data generation device of the present invention. The foreign object detection model may also be, for example, a trained model generated in advance. The trained model may also be a trained model (derived model) retrained using the foreign object detection training data and an already generated trained model. Furthermore, the trained model may be a trained model obtained by transfer learning using a trained model generated using foreign object detection training data, or a trained model generated by model compression of a trained model generated using foreign object detection training data.

The foreign object detection device 30 may include, for example, an output unit. In this case, the output unit may, for example, output the foreign object detection result. The output unit may, for example, output the foreign object detection result to a terminal external to the device via the communication network, or may output the foreign object detection result to the output device 307. The output foreign object detection result may also be stored, for example, in memory 302 or storage device 304.

In the method for generating training data for foreign object detection of this embodiment, steps S31 and S32 are executed sequentially, as an example, but the present invention is not limited to this. Specifically, in the present invention, steps S31 and S32 may be executed simultaneously or separately, and in the latter case, the order in which they are executed is not particularly limited and can be arbitrary.

The foreign object detection device of this embodiment makes it possible to detect foreign objects using, for example, a foreign object detection model generated by machine learning.

[Embodiment 6]
The first program of this embodiment is a program for causing a computer to execute each step of the above-described method for generating training data for foreign object detection. Specifically, the first program of this embodiment is a program for causing a computer to execute a detection object image acquisition procedure, a foreign object image acquisition procedure, an image synthesis procedure, and a training data output procedure.

the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed;
the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance;
the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image;
The teacher data output procedure includes:
A set of the foreign substance composite image and the composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance.

The first program of this embodiment can also be described as a program that causes a computer to function as a detection target image acquisition procedure, a foreign object image acquisition procedure, an image synthesis procedure, and a training data output procedure.

The first program of this embodiment can use the descriptions of the foreign object detection training data generation device and foreign object detection training data generation method of the present invention. For each of the steps, for example, "step" can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

[Embodiment 7]
The second program of this embodiment is a program for causing a computer to execute each step of the trained model production method described above. Specifically, the second program of this embodiment is a program for causing a computer to execute a training data acquisition procedure and a trained model generation procedure.

the teacher data acquisition step acquires, as teacher data for foreign object detection, teacher data for foreign object detection output by the first program;
The trained model generation procedure uses machine learning using the foreign object detection training data to generate a foreign object detection model as a trained model that, when an image of a foreign object detection target is input, outputs whether or not the foreign object detection target contains a foreign object.

The second program of this embodiment can also be described as a program that causes a computer to function as a training data acquisition procedure and a trained model generation procedure.

The second program of this embodiment can use the descriptions of the trained model production device and trained model production method of the present invention. For example, the "procedure" in each of the steps can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

[Embodiment 8]
The third program of this embodiment is a program for causing a computer to execute each step of the foreign object detection method described above. Specifically, the third program of this embodiment is a program for causing a computer to execute a foreign object detection target image acquisition procedure and a foreign object detection procedure.

the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target;
the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object;
The foreign object detection model is a trained model generated by machine learning using training data generated by the first program so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or is a trained model produced by the second program.

The second program of this embodiment can also be described as a program that causes a computer to function as a foreign object detection target image acquisition procedure and a foreign object detection procedure.

The third program of this embodiment can use the descriptions of the foreign object detection device and foreign object detection method of the present invention. For each of the steps, for example, "step" can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments can be described as, but not limited to, the following supplementary notes.
(Appendix 1)
The apparatus includes a detection object image acquisition unit, a foreign substance image acquisition unit, an image synthesis unit, and a training data output unit,
the detection target image acquisition unit acquires a detection target image obtained by capturing an image of the foreign object detection target after the foreign object has been removed,
the foreign substance image acquisition unit acquires a foreign substance image by capturing an image of the foreign substance;
the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image;
The teacher data output unit
a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance;
A device for generating training data for foreign object detection.
(Appendix 2)
2. The foreign object detection training data generating device according to claim 1, wherein the training data output unit outputs the detection target image as training data without a foreign object.
(Appendix 3)
the detection target image acquisition unit includes a detection target imaging unit and a difference target image extraction unit;
the detection target imaging unit images the detection target at predetermined time intervals;
The foreign object detection training data generation device described in Appendix 1 or 2, wherein the difference target image extraction unit extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, generates a difference target image, and acquires the first target image as the detection target image if the area of the difference target image is equal to or greater than a threshold.
(Appendix 4)
the foreign substance image acquisition unit includes a foreign substance imaging unit, a differential foreign substance image extraction unit, and a foreign substance determination unit;
the foreign object image capturing unit captures an image of the foreign object at predetermined time intervals;
the differential foreign substance image extracting unit extracts a difference between a first foreign substance image of the foreign substance captured the nth time and a second foreign substance image of the foreign substance captured the (n-1)th time to generate a differential foreign substance image;
The foreign substance detection training data generation device according to any one of appendixes 1 to 3, wherein the foreign substance determination unit determines that the differential foreign substance image is an image of a foreign substance when the area of the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image.
(Appendix 5)
A change information acquisition unit is included,
the change information acquisition unit acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter;
the detection target image acquisition unit acquires the detection target image when change information of the detection target is acquired;
5. The foreign substance detection training data generation device according to claim 1, wherein the foreign substance image acquisition unit acquires the foreign substance image when it acquires change information of the foreign substance.
(Appendix 6)
Includes an image processing section,
the image processing unit processes the detection target image to generate a processed detection target image and a processed foreign substance image to generate a processed foreign substance image,
The foreign matter detection training data generation device according to any one of appendices 1 to 5, wherein the image synthesis unit generates a foreign matter composite image by synthesizing at least one of the processed detection target image and the processed foreign matter image.
(Appendix 7)
The system includes a training data acquisition unit and a trained model generation unit,
the teacher data acquisition unit acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a teacher data generation device for foreign object detection described in any one of Supplementary Notes 1 to 6;
The trained model generation unit generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, through machine learning using the foreign object detection training data.
(Appendix 8)
a foreign object detection target image acquisition unit and a foreign object detection unit,
the foreign object detection target image acquisition unit acquires a foreign object detection target image by capturing an image of the foreign object detection target,
the foreign object detection unit inputs the foreign object detection target image into a foreign object detection model and detects whether or not the foreign object detection target includes a foreign object;
The foreign object detection model is a trained model generated by machine learning using training data generated by a foreign object detection training data generation device described in any one of Supplementary Notes 1 to 6, so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or is a trained model manufactured by a trained model manufacturing device described in Supplementary Note 7.
(Appendix 9)
The method includes a detection object image acquisition step, a foreign substance image acquisition step, an image synthesis step, and a training data output step,
the detection target image acquisition step acquires a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed;
the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance;
the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image;
The teacher data output step includes:
a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance;
A method for generating training data for foreign object detection.
(Appendix 10)
10. The foreign substance detection training data generating method according to claim 9, wherein the training data output step outputs the detection target image as training data without any foreign substance.
(Appendix 11)
the detection target image acquisition step includes a detection target imaging step and a difference target image extraction step;
the detection target imaging step images the detection target at predetermined time intervals;
The method for generating training data for foreign object detection described in Appendix 9 or 10, wherein the differential object image extraction step extracts the difference between a first object image of the detection object captured the nth time and a second object image of the detection object captured the (n-1)th time, generates a differential object image, and acquires the first object image as the detection object image if the area of the differential object image is equal to or greater than a threshold.
(Appendix 12)
the foreign substance image acquisition step includes a foreign substance imaging step, a differential foreign substance image extraction step, and a foreign substance determination step;
the foreign matter imaging step images the foreign matter at predetermined time intervals,
the differential impurity image extraction step extracts a difference between a first impurity image of the impurity captured the nth time and a second impurity image of the impurity captured the (n-1)th time to generate a differential impurity image;
12. The foreign substance detection training data generation method according to any one of appendices 9 to 11, wherein the foreign substance determination step determines that the differential foreign substance image is an image of a foreign substance if the area of the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image.
(Appendix 13)
A change information acquisition step is included,
the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection object and the foreign substance;
the detection target image acquisition step acquires the detection target image when change information of the detection target is acquired;
13. The foreign substance detection training data generation method according to any one of appendices 9 to 12, wherein the foreign substance image acquisition step acquires the foreign substance image when change information of the foreign substance is acquired.
(Appendix 14)
Including image processing process,
the image processing step processes the detection target image and the foreign substance image to generate a processed detection target image and a processed foreign substance image,
The foreign matter detection training data generation method according to any one of appendices 9 to 13, wherein the image synthesis step generates a foreign matter composite image by synthesizing at least one of the processed detection target image and the processed foreign matter image.
(Appendix 15)
The method includes a training data acquisition step and a trained model generation step.
the teacher data acquisition step acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a method for generating teacher data for foreign object detection described in any one of Supplementary Notes 9 to 14;
The trained model generation process is a trained model manufacturing method in which, through machine learning using the foreign object detection training data, a foreign object detection model is generated as a trained model that, when an image of a foreign object detection target captured as a foreign object detection target is input, outputs whether the foreign object detection target contains a foreign object.
(Appendix 16)
The method includes a foreign object detection image acquisition step and a foreign object detection step,
the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target,
the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model and detecting whether or not a foreign object is included in the foreign object detection target;
The foreign object detection method, wherein the foreign object detection model is a trained model generated by machine learning using training data generated by the foreign object detection training data generation method described in any one of Supplementary Notes 9 to 14, so as to output whether or not a foreign object is contained in a foreign object detection target when an image of the foreign object detection target is input, or is a trained model produced by the trained model production method described in Supplementary Note 15.
(Appendix 17)
The method includes a detection target image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure.
the detection target image acquisition step includes acquiring a detection target image by capturing an image of the foreign object detection target after the foreign object has been removed;
the foreign substance image acquisition step acquires a foreign substance image by capturing an image of the foreign substance;
the image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image;
The teacher data output procedure includes:
outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance;
A program for causing a computer to execute each of the above procedures.
(Appendix 18)
18. The program according to claim 17, wherein the teacher data output step outputs the detection target image as teacher data without foreign matter.
(Appendix 19)
the detection target image acquisition step includes a detection target image capture step and a difference target image extraction step;
the step of capturing an image of the detection target includes capturing an image of the detection target every predetermined time period;
The program described in Appendix 17 or 18, wherein the differential target image extraction procedure extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, generates a differential target image, and acquires the first target image as the detection target image if the area of the differential target image is equal to or greater than a threshold.
(Appendix 20)
the foreign substance image acquisition step includes a foreign substance imaging step, a differential foreign substance image extraction step, and a foreign substance determination step;
the foreign object imaging step images the foreign object at predetermined time intervals;
the step of extracting a difference impurity image includes extracting a difference between a first impurity image of the impurity captured the nth time and a second impurity image of the impurity captured the (n-1)th time to generate a difference impurity image;
20. The program according to any one of appendices 17 to 19, wherein the foreign substance determination step determines that the subtraction foreign substance image is an image of a foreign substance if an area of the subtraction foreign substance image is within a threshold range, and acquires the subtraction foreign substance image as the foreign substance image.
(Appendix 21)
A procedure for acquiring change information is included.
the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection object and the foreign substance;
the detection target image acquisition step includes acquiring the detection target image when change information of the detection target object is acquired;
21. The program according to any one of appendices 17 to 20, wherein the foreign substance image acquisition step acquires the foreign substance image when change information of the foreign substance is acquired.
(Appendix 22)
Includes image processing procedures,
the image processing step includes processing the detection target image to generate a processed detection target image and processing the foreign substance image to generate a processed foreign substance image,
22. The program according to claim 17, wherein the image synthesis step generates a foreign matter composite image by synthesizing at least one of the processed foreign matter detection image and the processed foreign matter image.
(Appendix 23)
This includes a procedure for acquiring training data and a procedure for generating a trained model.
the teacher data acquisition step acquires, as the teacher data for foreign object detection, teacher data for foreign object detection output by a program described in any one of Supplementary Notes 17 to 22;
the trained model generation step generates, as a trained model, a foreign object detection model that outputs whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, by machine learning using the foreign object detection training data;
A program for causing a computer to execute each of the above procedures.
(Appendix 24)
The method includes a foreign object detection image acquisition step and a foreign object detection step,
the foreign object detection target image acquisition step acquires a foreign object detection target image by capturing an image of the foreign object detection target;
the foreign object detection step includes inputting the foreign object detection target image into a foreign object detection model, and detecting whether or not the foreign object detection target includes a foreign object;
the foreign object detection model is a trained model generated by machine learning using training data generated by a program described in any one of Supplementary Notes 17 to 22, so as to output whether or not a foreign object is included in a foreign object detection target when an image of the foreign object detection target obtained by capturing the foreign object detection target is input, or is a trained model produced by a program described in Supplementary Note 23,
(Appendix 25)
A computer-readable recording medium having a program according to any one of appendices 17 to 24 recorded thereon.

The present invention makes it possible to generate training data for foreign object detection without the need for labeling. Therefore, the present invention is widely useful in fields related to foreign object detection.

10, 10A, 10B Foreign substance detection training data generating device 11 Detection target image acquisition unit 12 Foreign substance image acquisition unit 13 Image synthesis unit 14 Training data output unit 15 Change information acquisition unit 16 Image processing unit 101 CPU
102 Memory 103 Bus 104 Storage device 105 Program 106 Input device 107 Output device 108 Communication device 20 Trained model production device 21 Teacher data acquisition unit 22 Trained model generation unit 201 CPU
202 Memory 203 Bus 204 Storage device 205 Program 206 Input device 207 Output device 208 Communication device 30 Foreign object detection device 31 Foreign object detection target image acquisition unit 32 Foreign object detection unit 301 CPU
302 Memory 303 Bus 304 Storage device 305 Program 306 Input device 307 Output device 308 Communication device

Claims (7)

The apparatus includes a detection object imaging unit, a foreign object imaging unit, a change information acquisition unit, a detection object image acquisition unit, a foreign object image acquisition unit, an image synthesis unit, and a teacher data output unit,
the detection target imaging unit intermittently images the detection target after the foreign matter has been removed,
the foreign object image capturing unit intermittently captures images of the foreign object;
the change information acquisition unit acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter;
the detection target image acquisition unit acquires a detection target image when change information of the detection target is acquired ;
the foreign substance image acquisition unit acquires a foreign substance image when acquiring information about a change in the foreign substance ;
the image composition unit combines the foreign substance image with the detection target image to generate a foreign substance composite image;
The teacher data output unit
a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance;
A device for generating training data for foreign object detection. The foreign object detection training data generation device of claim 1, wherein the training data output unit outputs the detection target image as training data without foreign objects. the detection target image acquisition unit includes a difference target image extraction unit,
the detection target imaging unit images the detection target at predetermined time intervals;
The foreign object detection training data generation device of claim 1 or 2, wherein the difference target image extraction unit extracts the difference between a first target image of the detection target captured the nth time and a second target image of the detection target captured the (n-1)th time, generates a difference target image, and acquires the first target image as the detection target image if the area of the difference target image is equal to or greater than a threshold. the foreign substance image acquisition unit includes a differential foreign substance image extraction unit and a foreign substance determination unit,
the foreign object image capturing unit captures an image of the foreign object at predetermined time intervals;
the differential foreign substance image extracting unit extracts a difference between a first foreign substance image of the foreign substance captured the nth time and a second foreign substance image of the foreign substance captured the (n-1)th time to generate a differential foreign substance image;
4. The foreign substance detection training data generation device according to claim 1, wherein the foreign substance determination unit determines that the differential foreign substance image is an image of a foreign substance when an area of the differential foreign substance image is within a threshold range, and acquires the differential foreign substance image as the foreign substance image. Includes an image processing section,
the image processing unit processes the detection target image to generate a processed detection target image and a processed foreign substance image to generate a processed foreign substance image,
The foreign matter detection training data generating device according to claim 1 , wherein the image combining unit generates a foreign matter composite image by combining at least one of the processed detection object image and the processed foreign matter image. A method for generating training data for foreign substance detection, in which each step including a detection object imaging step, a foreign substance imaging step, a change information acquisition step, a detection object image acquisition step, a foreign substance image acquisition step, an image synthesis step, and a training data output step is executed by a computer,
the detection target imaging step intermittently images the detection target after the foreign matter has been removed,
the foreign matter imaging step intermittently images the foreign matter,
the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection target and the foreign matter;
the detection object image acquisition step acquires a detection object image when change information of the detection object is acquired ;
the foreign substance image acquisition step acquires a foreign substance image when the change information of the foreign substance is acquired ,
The image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesis image, and the training data output step
a pair of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image is output as training data indicating the presence of a foreign substance;
A method for generating training data for foreign object detection. The method includes a detection target imaging procedure, a foreign substance imaging procedure, a change information acquisition procedure, a detection target image acquisition procedure, a foreign substance image acquisition procedure, an image synthesis procedure, and a training data output procedure,
the step of capturing an image of the detection target includes intermittently capturing an image of the detection target after the foreign object has been removed;
The foreign object imaging step includes intermittently imaging the foreign object,
the change information acquisition step acquires change information obtained by detecting a change in at least one of the detection target and the foreign object;
the detection target image acquisition step includes acquiring a detection target image when change information of the detection target is acquired ;
the foreign substance image acquisition step includes acquiring a foreign substance image when the change information of the foreign substance is acquired ;
The image synthesis step synthesizes the foreign substance image with the detection target image to generate a foreign substance synthesized image, and the training data output step
outputting a set of the foreign substance composite image and a composite position of the foreign substance image in the foreign substance composite image as training data indicating the presence of a foreign substance;
A program for causing a computer to execute each of the above procedures.