JP7108283B2 - Waste Sorting System, Teacher Data Generating Device, Teacher Data Generating System, and Teacher Data Generating Program - Google Patents

Description

The present invention relates to a waste sorting system, a teacher data generation device, a teacher data generation system, and a teacher data generation program for generating teacher data used for machine learning performed in the waste sorting system.

In recent years, from the viewpoint of effective use of resources, there is a demand for reusing waste as recyclable resources. In order to promote reuse of waste, it is necessary to sort waste by material. For example, waste from dismantled houses and facilities, as well as rubble generated at the time of a disaster, contains a mixture of different materials such as wood, metal, glass, and plastic. Most of them are manually sorted.

The applicant of the present application discloses a sorting system for such waste in Patent Document 1. The sorting system disclosed in Patent Document 1 displays information such as the image of the waste conveyed by the conveyor and the weight of the waste on a monitor. The operator determines the material of the waste based on the information displayed on the monitor and operates the switch according to the material. The sorting system sorts the waste by driving the robot arm to grip the waste on the conveyor and ejecting the waste into the collection box of the material corresponding to the operated switch. Also, the image of the waste and the material information indicating the material thereof are linked and stored as training data.

JP 2017-109161 A

The sorting system disclosed in Patent Document 1 can generate an amount of training data corresponding to the amount of waste to be processed. However, this sorting system is not suitable for generating a large amount of training data because the wastes flow one by one on the conveyor, so the amount of wastes processed per unit time is relatively small.

For example, by adopting a configuration in which a plurality of pieces of waste flow on a conveyor, the amount of waste processed per unit time can be increased. However, when multiple wastes are displayed, multiple operations of selecting a waste that links material information and specifying the material of the selected waste are required. There was a risk that the operator's operation would be delayed due to the transport speed.

The present invention has been made in view of this point, and provides a waste sorting system suitable for generating a large amount of teacher data, a teacher data generation device, a teacher data generation system, and a teacher data generation program. intended to

In order to achieve the above object, a waste sorting system according to one aspect of the present invention includes a conveying device that conveys a plurality of wastes in one direction, and a photographing device that photographs the plurality of wastes conveyed by the conveying device. a moving device for moving the waste transported by the transporting device from the transporting device; and a control device for controlling the moving device. A waste sorting system for generating training data that links material information indicating the material of an object, wherein the control device includes an image display processing unit that displays an image captured by the imaging device on a display device; a first selection unit that selects one waste to which the material information is not linked from among a plurality of wastes contained in the a first selection mark display processing unit for displaying on the display device; a material information acquisition unit for acquiring material information indicating the material of the waste based on a material designation operation input to an operation input device; a teaching data storage processing unit for storing, in a storage device, teaching data linking the image of the waste selected by the unit and the material information acquired by the material information acquiring unit; a movement control unit configured to control the moving device to move the collected waste from the conveying device to a location corresponding to the material information acquired by the material information acquiring unit. do.

In the present invention, it is preferable that the first selection unit selects the waste whose upstream end position is closest to the downstream side among the wastes to which the material information is not linked.

In the present invention, a second selection for selecting one waste different from the waste selected by the first selection unit and not associated with the material information from among the plurality of wastes included in the image. and a second selection mark display processing unit that displays a second selection mark on the display device in association with the waste selected by the second selection unit.

In the present invention, it is preferable that the second selection unit selects the waste whose upstream end position is second counted from the downstream side among the wastes to which the material information is not linked.

The present invention further includes a first material estimation unit for estimating the material of the waste based on the image of the waste selected by the first selection unit, wherein the first selection mark display processing unit 1. The name of the material guessed by the material guessing unit is displayed on the display device together with the first selection mark .

In the present invention, the first material estimation unit may perform machine learning for estimating the material of the waste selected by the first selection unit, using the teacher data stored in the storage device. preferable.

The present invention further comprises a second material estimation unit for estimating the material of the waste based on the image of the waste selected by the second selection unit, wherein the second selection mark display processing unit It is preferable that the name of the material guessed by the second material guessing unit is displayed on the display device together with the second selection mark.

In the present invention, the second material estimation unit may perform machine learning for estimating the material of the waste selected by the second selection unit using the teacher data stored in the storage device. preferable.

In the present invention, it is preferable that the first selection mark is a frame graphic surrounding the waste selected by the first selection unit, or a balloon graphic connected to the waste.

In the present invention, it is preferable that the second selection mark is a frame figure surrounding the waste selected by the second selection section, or a balloon figure connected to the waste.

In order to achieve the above object, a training data generation device according to another aspect of the present invention is a training data generation device that generates training data by linking an image of waste with material information indicating the material of the waste. an image display processing unit for displaying, on a display device, an image in which a plurality of wastes are transported in one direction; a first selection unit for selecting one waste; a first selection mark display processing unit for displaying a first selection mark on the display device in association with the waste selected by the first selection unit; a material information acquisition unit that acquires material information indicating the material based on a material designation operation input to an operation input device; and an image of the waste selected by the first selection unit and the material information acquisition unit and a teaching data storage processing unit that stores teaching data associated with the material information that has been obtained in a storage device.

In order to achieve the above object, a training data generation system according to another aspect of the present invention is a training data generation system that generates training data by linking an image of waste with material information indicating the material of the waste. an image storage device in which images of a plurality of wastes being transported in one direction are stored; and displaying the image stored in the image storage device on the display device.

To achieve the above object, a training data generation program according to another aspect of the present invention is a training data generation program for generating training data in which an image of waste is linked with material information indicating the material of the waste. an image display processing unit configured to display, on a display device, a photographed image of a plurality of wastes being transported in one direction; a first selection unit that selects one waste that is not linked; a first selection mark display processing unit that displays a first selection mark on the display device in association with the waste selected by the first selection unit; a material information acquisition unit for acquiring material information indicating the material of the waste based on a material designation operation input to the operation input device; and an image of the waste selected by the first selection unit and the material information acquisition. and a teaching data storage processing unit that stores teaching data associated with the material information acquired by the unit in a storage device.

According to the present invention, an image obtained by photographing a state in which a plurality of wastes are conveyed in one direction is displayed on the display device. One waste (first waste) to which material information indicating the material is not associated is selected from among a plurality of wastes included in the image displayed on the display device. A first selection mark is displayed on the display in association with the first waste. Material information indicating the material of the waste is acquired based on the material designation operation input to the operation input device. Then, the teacher data that associates the image of the first waste with the acquired material information is stored in the storage device. With this arrangement, the operator visually recognizes the first waste associated with the first selection mark, identifies the material, and inputs a material designation operation for designating the material to the operation input device. Teacher data is generated. Therefore, it is possible to generate training data with a small number of operations without the need to select a waste whose material information is to be linked from among a plurality of wastes. Therefore, in a configuration in which a plurality of pieces of waste are conveyed, it is possible to prevent the operator's operation from being delayed by the speed of conveying the waste, and it is possible to generate a large amount of training data in a short time.

In addition, from among the plurality of wastes included in the image displayed on the display device, another waste (second waste) that is different from the first waste and to which the material information is not associated is selected. . A second selection mark is displayed on the display in association with the second waste. By doing so, the worker can recognize in advance that the second waste associated with the second selection mark is the waste whose material is specified next to the first waste. Therefore, it is possible to more effectively prevent the operator's operation from being delayed due to the transport speed of the waste, and it is possible to generate a large amount of teaching data in a short time.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure explaining schematic structure of the waste sorting system based on 1st Example of this invention. It is a figure explaining the screen display of the tablet terminal used in the waste sorting system of FIG. 2 is a diagram illustrating functional blocks of a control device of the waste sorting system of FIG. 1; FIG. 3 is a diagram for explaining a modification of screen display in the tablet terminal of FIG. 2; FIG. 2 is a flowchart illustrating an example of processing executed by a control device of the waste sorting system of FIG. 1; FIG. 10 is a diagram illustrating a schematic configuration of a training data generation system according to a second embodiment of the present invention; FIG. FIG. 7 is a diagram for explaining a screen display of a training data generation device included in the training data generation system of FIG. 6; FIG. 8 is a diagram illustrating functional blocks of the training data generation device of FIG. 7; FIG. 8 is a flowchart illustrating an example of processing executed by a control unit of the training data generation device of FIG. 7; FIG.

(First embodiment)
A waste sorting system according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 5. FIG.

FIG. 1 is a diagram illustrating a schematic configuration of a waste sorting system according to a first embodiment of the present invention. FIG. 2 is a diagram for explaining a screen display of a tablet terminal used in the waste sorting system of FIG. 1. FIG. FIG. 3 is a functional block configuration diagram of the control device of the waste sorting system of FIG. FIG. 4 is a diagram for explaining a modification of screen display on the tablet terminal in FIG. 5 is a flowchart illustrating an example of processing executed by the control device of the waste sorting system of FIG. 1. FIG.

A waste sorting system 1 according to the first embodiment of the present invention sorts out wastes contained in debris generated during demolishing of houses or the like, or rubble generated in the event of a disaster. These wastes are of a wide variety of materials such as wood, plastic, paper, siding, gypsum board, textiles, glass, ferrous metals and non-ferrous metals. In addition, the size and shape of these wastes are different and inconsistent.

As shown in FIG. 1, the waste sorting system 1 has a conveyor 10 as a transport device, a camera 20 as a photographing device, a robot arm 30 as a moving device, and a control device 40 .

The conveyor 10 conveys a plurality of wastes W in one direction (from left to right in FIG. 1). A sub-conveyor 11 is arranged upstream of the conveyor 10 , and a vibrating screen device 12 is arranged between the conveyor 10 and the sub-conveyor 11 . A plurality of wastes W conveyed by the sub-conveyor 11 are fed to the conveyor 10 after wastes Ws of a predetermined size (for example, 50 mm) or less are screened out by the vibrating sieve device 12 . In this embodiment, the conveyor 10 and the sub-conveyor 11 employ belt-type conveyors, but as long as a plurality of wastes W can be conveyed, various types of conveyors such as roller conveyors and bucket-type conveyors can be used. It is possible.

After sieving out the waste Ws of a predetermined size or smaller in the vibrating sieve device 12, ferromagnetic materials such as iron and titanium may be further removed from the plurality of wastes W using a magnetic force. Since ferromagnetic materials such as iron can be easily removed by magnetic force, the operator's burden of identifying materials can be reduced by removing the magnetic materials in advance and then sorting out other materials. On the other hand, if priority is given to generating teacher data corresponding to a wide variety of materials in the teacher data storage processing unit 46, a plurality of wastes passed through the vibrating sieve device 12 are not removed using magnetic force. W may be supplied to the conveyor 10 as it is. In this embodiment, the waste W is not removed using magnetic force.

The camera 20 is installed upstream of the conveyor 10 . Specifically, the camera 20 is arranged so as to photograph a part of the conveyor 10 closer to the upstream side from above. The camera 20 photographs a state in which a plurality of wastes W are conveyed by the conveyor 10. - 特許庁In this embodiment, the camera 20 employs an RGB camera that captures color moving images. In addition, as long as it does not contradict the purpose of the present invention, any image that can be used by the worker to identify the material of each waste can be used according to the system configuration, the type of waste, etc. Other types of imaging devices than cameras may be employed. For example, a still image camera that captures still images, an X-ray camera or near-infrared camera that captures images using X-rays or near-infrared rays, or a 3D camera that captures a subject three-dimensionally may be employed.

The robot arm 30 is installed downstream of the conveyor 10 . Specifically, the robot arm 30 is installed on the side of the conveyor 10 near the downstream side. The robot arm 30 picks up a plurality of wastes W on the conveyor 10 one by one. Then, the robot arm 30 moves the picked up waste W to a recovery box 70 provided for each material (paper, cardboard, wood, siding, concrete, hard plastic, plastic, vinyl, vinyl chloride). Any device other than the robot arm 30 may be used depending on the system configuration, type of waste, etc., as long as it does not violate the object of the present invention. type of moving device may be employed.

The control device 40 controls the operation of the entire waste sorting system 1 . In this embodiment, the control device 40 generates teacher data by linking material information indicating the material to the images of the plurality of wastes W conveyed by the conveyor 10 . In addition, the control device 40 controls the robot arm 30 to sequentially move (discharge) a plurality of wastes W from the conveyor 10 to collection boxes 70 provided for each material.

The control device 40 has a computer. The computer may be a personal computer or a large computer such as a workstation or server device. A computer has a central processing unit (CPU) and a program storage device such as a memory or a hard disk device that stores various programs executed by the CPU.

A tablet terminal 50 is connected to the control device 40 through a wireless communication line. As shown in FIG. 2 , the tablet terminal 50 has a display section 51 as a display device such as a liquid crystal display or an organic EL display, and a touch panel 52 superimposed on the display section 51 . The tablet terminal 50 also has an operation input unit 54 as an operation input device including a touch panel 52 , operation icons 53 displayed on the display unit 51 , and software switches combined with the touch panel 52 . The operation icon 53 is provided for each material of the waste W to be sorted.

In the tablet terminal 50, an operation of tapping (shortly touching once) the software switch of the operation input unit 54 is an operation of specifying a material corresponding to the operation icon 53 constituting the software switch (material specifying operation). When a material specifying operation is input to the operation input unit 54, the tablet terminal 50 transmits material information indicating a material corresponding to the material specifying operation to the control device 40 through a wireless communication line. For example, when the software switch having the operation icon 53 of "paper" is tapped, material information indicating "paper" is transmitted to the control device 40 assuming that "paper" is designated as the material. The same is true for other materials such as "cardboard" and "wood".

The display unit 51 of the tablet terminal 50 is provided with an image display area 55 for displaying an image transmitted from the control device 40 through the wireless communication line. The image display area 55 displays a plurality of wastes W flowing from left to right. Further, the display unit 51 is provided with a message area 56 for displaying messages to the operator and the like, and a graph display area 57 for displaying information on the number of clicks in the form of a bar graph.

In this embodiment, a tablet terminal 50 is used as a display device and an operation input device. However, instead of the tablet terminal 50 , a display device such as a monitor device having a liquid crystal display or an organic EL display, and an operation input device such as a keyboard and a mouse may be connected to the control device 40 . In such a configuration, the same display and operation as those of the tablet terminal 50 are performed on these display device and operation input device. Alternatively, a control unit (not shown) composed of a computer included in the tablet terminal 50 may be used as the control device 40 .

A storage device 60 such as an external hard disk device is connected to the control device 40 . The storage device 60 stores teacher data used for learning to guess the material of the waste.

As shown in FIG. 3, the computer of the control device 40 executes a program (teaching data generation program) stored in the program storage device to perform an image display processing section 41, a selection section 42, a material estimation section 43, a selection It functions as a mark display processing unit 44 , a material information acquisition unit 45 , a teacher data storage processing unit 46 and a movement control unit 47 .

The image display processing unit 41 transmits to the tablet terminal 50 an image of how a plurality of wastes W are transported in one direction, which is captured by the camera 20, and displays it in the image display area 55 of the display unit 51 of the tablet terminal 50. indicate. The image display processing unit 41 also sends the image sent to the tablet terminal 50 to the selection unit 42 .

The selection unit 42 selects the waste W ( A first waste W1) is selected. Further, the selection unit 42 selects the waste W (second waste W2) whose material information is to be linked next to the first waste W1.

Specifically, the selection unit 42 analyzes the image, extracts the image of each of the plurality of wastes W, and identifies each waste W individually. Then, any waste Wd associated with material information in past processing is excluded, and the position of the upstream end Wa of the waste W not associated with material information is closest to the downstream side (toward the right side in FIG. 2). is selected as the first waste W1. Furthermore, among the wastes W to which the material information is not associated, the wastes W whose upstream end Wa is located second from the downstream side is selected as the second waste W2. The selector 42 corresponds to a first selector and a second selector.

In addition to the selection method described above, the selection unit 42 may select, for example, the waste W not associated with material information that has the smallest displayed area as the first waste W1. The second waste W2 may be selected based on the shape of the waste W such as the area. Alternatively, among the wastes W to which the material information is not associated, the one closest to the upper part of the downstream side (that is, the upper right of the image display area 55) is selected as the first waste W1, or the same waste as the first waste W1 is selected. A waste W presumed to be the material may be selected as the second waste. A method of selecting the first waste W1 and the second waste W2 in the selection unit 42 is arbitrary as long as it does not contradict the purpose of the present invention.

The material estimation unit 43 estimates the materials of the first waste W1 and the second waste W2 selected by the selection unit 42 .

Specifically, an image of the first waste W1 is acquired from the selection unit 42, and pattern matching processing is performed on this image using recognition pattern data prepared for each material, thereby identifying the material of the first waste W1. Infer. Similarly, the material of the second waste W2 is also guessed. Of course, the material may be guessed by a processing method other than this. The recognition pattern data used for estimation is updated by machine learning using teacher data stored in the storage device 60 . The material guessing section 43 corresponds to the first material guessing section and the second material guessing section.

The material guessing unit 43, for example, periodically, during a period when the system is not performing the sorting process, or when new teacher data is added to the storage device 60, at a predetermined timing. , and performs machine learning to update the recognition pattern data. In this embodiment, the material estimation unit 43 is configured to perform machine learning using a convolutional neural network using an optimization technique based on error backpropagation. Of course, you may perform machine learning by the method other than this.

The selection mark display processing unit 44 displays the first selection mark M1 in the image display area 55 of the display unit 51 of the tablet terminal 50 in association with the first waste W1. Also, the second selection mark M2 is displayed in the image display area 55 of the display unit 51 of the tablet terminal 50 in association with the second waste W2. The first selection mark M1 informs the operator that the first waste W1 associated with it is selected as a movement target and a teacher data generation target. The second selection mark M2 informs the operator that the second waste W2 with which it is associated will be selected next as an object to be moved and an object to generate teacher data.

Specifically, the selection mark display processing unit 44 transmits display information regarding a red and thick frame graphic surrounding the first waste W1 to the tablet terminal 50, and displays the frame graphic as the first selection mark M1. 51. In addition, the display information about the blue and thick line frame graphic surrounding the second waste W2 is transmitted to the tablet terminal 50, and the frame graphic is displayed on the display unit 51 as the second selection mark M2. The selection mark display processing unit 44 may display at least the first selection mark M1, and preferably displays the second selection mark M2.

Then, the selection mark display processing unit 44 transmits display information regarding the name N1 (that is, material identification information) of the material candidate of the first waste W1 to the tablet terminal 50 based on the result of estimation by the material estimation unit 43, The name N1 is displayed within the first selection mark M1. Similarly, the display information regarding the name N2 of the material candidate for the second waste W2 is transmitted to the tablet terminal 50, and the name N2 is displayed within the second selection mark M2. In addition to the name of the material, an identification number, an identification symbol, an abbreviation, or the like that can be used by the worker to identify the material may be displayed as identification information of the material. In this way, by displaying the identification information of the estimated materials for the first waste W1 and the second waste W2, it is possible to assist the operator in identifying the materials. Note that the display of the identification information of the materials of the first waste W1 and the second waste W2 may be omitted. The selection mark display processing section 44 corresponds to the first selection mark display processing section and the second selection mark display processing section.

In addition, when there is a waste W to which the material information is not linked other than the first waste W1 and the second waste W2, the selection mark display processing unit 44 displays a blue thin-line frame figure surrounding the waste W. is transmitted to the tablet terminal 50 , and the frame figure is displayed on the display unit 51 as an unselected mark M. Note that the display of the unselected mark M may be omitted.

Further, when there is a waste Wd associated with the material information in the past processing, the selection mark display processing unit 44 transmits display information regarding a blue and dotted line frame figure surrounding the waste Wd to the tablet terminal 50, The frame graphic is displayed on the display unit 51 as the material determination mark Md. At this time, the selection mark display processing unit 44 transmits display information regarding the material name Nd indicated in the material information associated with the waste Wd to the tablet terminal 50, and uses the name Nd as the material name of the waste Wd. It is displayed within the confirmation mark Md. The display of the material determination mark Md or the material name Nd may be omitted.

It should be noted that the selection mark display processing unit 44 uses the first selection mark M1, the second selection mark M2, and the material determination mark Md instead of the frame figures as shown in FIG. The display unit 51 may display a balloon graphic connected to the object W2 and the waste Wd. As shown in FIG. 4, in the message area 56, a material confirmation mark Md, a first selection mark M1, and a second selection mark M2, which are balloon graphics, are displayed in order from right to left. In the balloon graphic, the name N1 of the material of the first waste W1, the name N2 of the material of the second waste W2, and the name Nd of the material of the waste Wd are displayed.

When a material designation operation is input to the operation input unit 54 while the first selection mark M1 is displayed on the display unit 51 of the tablet terminal 50, the material information acquisition unit 45 selects the tablet terminal based on the material designation operation. The material information transmitted from 50 is received (obtained).

When the material information is received by the material information acquisition unit 45, the teaching data storage processing unit 46 acquires the image of the first waste W1 from the selection unit 42, and associates this image with the received material information to obtain the teaching data. It is generated and stored in the storage device 60 . The teaching data generated by the teaching data storage processing unit 46 is accumulated in the storage device 60 .

When the material information acquisition unit 45 receives the material information, the movement control unit 47 controls the robot arm 30 to move the first waste W1 on the conveyor 10 to the collection box 70 corresponding to the received material information. As a result, the first waste W1 is moved to the collection box 70 and discharged from the conveyor 10. As shown in FIG.

The waste sorting system 1 is divided into a working room 81 and a control room 82 partitioned by an isolation wall 80 and installed. Specifically, a conveyor 10, a sub-conveyor 11, a vibrating sieve device 12, a camera 20, a robot arm 30, and a collection box 70 are installed in the work chamber 81. As shown in FIG. In a control room 82 having a relatively better work environment than the work room 81, a control device 40, a tablet terminal 50, and a storage device 60 are installed. Note that the work room 81 and the control room 82 may be located in remote locations away from each other. Also, the control device 40 and at least one of the storage device 60 and the tablet terminal 50 may be connected via the Internet, for example.

Next, referring to the flowchart of FIG. 5, an example of the processing (teaching data generation processing, waste sorting processing) according to the present invention executed by the computer of the control device 40 of the waste sorting system 1 of the embodiment described above. to explain.

The control device 40 displays the image captured by the camera 20 in the image display area 55 of the display section 51 of the tablet terminal 50 (S110). Further, the control device 40 analyzes the image transmitted to the tablet terminal 50 and identifies a plurality of wastes W included in the image (S120).

Next, if there is a plurality of wastes W to which material information is not linked (Y in S130), the control device 40 determines that the position of the upstream end Wa is closest to the downstream side. is selected as the first waste W1 (S140). The control device 40 guesses the material based on the image of the first waste W1 (S150). Then, the control device 40 displays the first selection mark M1, which is a frame graphic surrounding the first waste W1 (S160), and inserts the name N1 of the material candidate for the first waste W1 into the first selection mark M1. Display (S170).

Next, if the control device 40 finds that there is a waste W to which the material information is not linked among the plurality of wastes W (Y in S180), the position of the upstream end Wa among them is counted from the downstream side. The second one is selected as the second waste W2 (S190). The control device 40 guesses the material based on the image of the second waste W2 (S200). Then, the control device 40 displays the second selection mark M2, which is a frame graphic surrounding the second waste W2 (S210), and the name N2, which is the material candidate for the second waste W2, is displayed in the second selection mark M2. (S220).

Next, if there is another waste W to which material information is not linked among the plurality of wastes W (Y in S230), the control device 40 selects an unselected mark M which is a frame figure surrounding the waste W. is displayed (S240).

Next, the control device 40 displays the unselected mark M (S240) or when there is no waste W to which the material information is not linked among the plurality of wastes W (N in S130, S180 N in S230, N in S230), and if there is a waste Wd associated with material information among a plurality of wastes W (Y in S250), the material determination mark Md, which is a frame figure surrounding the waste Wd, is displayed. (S260), and the name Nd of the material of the waste Wd is displayed in the material determination mark Md (S270).

Next, if the control device 40 has selected the first waste W1 from among the plurality of wastes W in the previous process (Y in S280), the control device 40 waits until the material information is acquired from the tablet terminal 50. Wait (N at S290), and if the first waste W1 is not selected, the processing of this flow chart ends (N at S280). When the control device 40 acquires the material information from the tablet terminal 50 (Y in S290), it generates teaching data by linking the image of the first waste W1 and the acquired material information, and stores the generated teaching data in the storage device 60. (S300). Further, the control device 40 controls the robot arm 30 to move (select) the first waste W1 to the collection box 70 corresponding to the acquired material information and discharge it from the conveyor 10 (S310). End the process.

The control device 40 repeats the above-described processing to proceed with the sorting of the waste W and the generation of training data.

As described above, according to the waste sorting system 1 of the present embodiment, the display unit 51 displays an image in which a plurality of wastes W are transported in one direction. One waste W (first waste W1) not associated with material information indicating the material is selected from among a plurality of wastes W included in the image displayed on the display unit 51 . A first selection mark M1 is displayed on the display unit 51 in association with the first waste W1. Material information indicating the material of the waste W is acquired based on the material designation operation input to the operation input unit 54 . Then, the storage device 60 stores teaching data that associates the image of the first waste W1 with the acquired material information. As a result, the operator visually recognizes the first waste W1 associated with the first selection mark M1, identifies the material thereof, and performs a material designation operation for designating the identified material through the operation input unit 54. Teacher data is generated by inputting. Therefore, it is possible to generate training data with a small number of operations without the need to select a waste to which material information is to be linked from among the plurality of wastes W displayed on the display unit 51 . Therefore, in a configuration in which a plurality of pieces of waste W are conveyed, it is possible to prevent the operator's operation from being delayed by the conveying speed of the waste W, and to generate a large amount of teaching data in a short time.

In addition, among the plurality of wastes W included in the image displayed on the display unit 51, one waste W (second waste W2). A second selection mark M2 is displayed on the display device in association with the second waste W2. By doing so, the worker can recognize in advance that the second waste W2 associated with the second selection mark M2 is the waste W whose material is specified next to the first waste W1. Therefore, it is possible to more effectively prevent the operator's operation from being delayed due to the transport speed of the waste W, and to generate a large amount of teaching data in a short time.

In addition, among the wastes W to which the material information is not linked, the one whose upstream end Wa is located at the furthest downstream side is selected as the first waste W1. By doing so, the waste W that flows away (disappears earliest) from the display unit 51 is selected as the first waste W1. It is possible to suppress the W from flowing away.

In addition, among the wastes W to which the material information is not associated, the wastes W whose upstream end Wa is located second from the downstream side is selected as the second waste W2. By doing so, the waste W that flows away from the display section 51 second fastest is selected as the second waste W2, so that the waste W flows away from the display section 51 before the material information is linked. can be further suppressed.

Also, the material of the first waste W1 is guessed based on the image of the first waste W1, and the name of the guessed material is displayed on the display section 51 together with the first selection mark M1. Similarly, the material of the second waste W2 is guessed based on the image of the second waste W2, and the name of the guessed material is displayed on the display section 51 together with the second selection mark M2. By doing so, it is possible to effectively assist the worker in determining the material.

Also, machine learning is performed to estimate the material of the first waste W1 using the teaching data stored in the storage device 60 . Similarly, using the teacher data stored in the storage device 60, machine learning is performed to estimate the material of the second waste W2. By doing so, it is possible to further improve the accuracy of estimating the material.

Also, the first selection mark M1 is a frame figure surrounding the first waste W1 or a balloon figure connected to the first waste W1. Similarly, the second selection mark M2 is a frame figure surrounding the second waste W2, or a balloon figure connected to the second waste W2. By doing so, the first selection mark M1 is displayed in association with the first waste W1, and the second selection mark M2 is displayed in association with the second waste W2. At a glance, the first waste W1 and the second waste W2 can be visually recognized.

(Second embodiment)
A teaching data generation system according to a second embodiment of the present invention will be described below with reference to FIGS. 6 to 9. FIG.

FIG. 6 is a diagram for explaining the schematic configuration of a training data generation system according to the second embodiment of the present invention. FIG. 7 is a diagram for explaining a screen display in the training data generation device included in the training data generation system of FIG. FIG. 8 is a diagram for explaining functional blocks of the training data generation device of FIG. FIG. 9 is a flowchart illustrating an example of processing executed by a control unit of the training data generation device of FIG. 7;

The training data generation system 101 according to the second embodiment of the present invention is used for machine learning for estimating waste when sorting out waste contained in demolition waste such as houses or debris generated in the event of a disaster. This is to generate teacher data.

As shown in FIG. 6 , the training data generation system 101 has a training data generation device 110 , a training data storage device 160 and an image storage device 170 . The teacher data storage device 160 stores the teacher data generated by the teacher data generator 110. FIG. The image storage device 170 stores, for example, images of a plurality of wastes W flowing in one direction, which are captured by the camera 20 of the first embodiment described above.

Teacher data generation device 110 is connected to teacher data storage device 160 and image storage device 170 via mobile communication network 180 and Internet 190 . Note that the training data generation device 110, training data storage device 160 and image storage device 170 may be connected via a local area network.

The teacher data generation device 110 is configured by a tablet terminal. The teacher data generation device 110 has a control unit 140 , a display unit 151 as a display device, and a touch panel 152 .

As shown in FIG. 7, the display unit 151 is composed of a liquid crystal display, an organic EL display, or the like. The touch panel 152 is superimposed on the display section 151 . The teacher data generation device 110 also has an operation input unit 154 as an operation input device including a touch panel 152 , operation icons 153 displayed on the display unit 151 , and software switches combined with the touch panel 152 . The operation icon 153 is provided for each material of the waste W to be sorted. Operation inputs for the software switches are the same as in the first embodiment.

The control unit 140 has a computer including a central processing unit (CPU) and a program storage device such as a non-volatile memory that stores various programs executed by the CPU.

As shown in FIG. 8, the computer of the control unit 140 executes the teacher data generation program stored in the program storage device to perform the image display processing unit 141, the selection unit 142, the material estimation unit 143, the selection mark display process. It functions as a unit 144 , a material information acquisition unit 145 , and a teacher data storage processing unit 146 .

The image display processing unit 141 displays an image of a plurality of wastes W being transported in one direction, which is stored in the image storage device 170 , in the image display area 155 of the display unit 151 . Also, the image display processing unit 141 sends the image displayed on the display unit 151 to the selection unit 142 .

Similar to the selection unit 42 of the first embodiment, the selection unit 142 selects the waste W (first 1 Select waste W1). Further, the selection unit 142 selects the waste W (second waste W2) whose material information is to be linked next to the first waste W1. The selector 142 corresponds to a first selector and a second selector.

The material estimation unit 143 estimates the materials of the first waste W1 and the second waste W2 selected by the selection unit 142 in the same manner as the material estimation unit 43 of the first embodiment. The material guessing section 143 corresponds to the first material guessing section and the second material guessing section.

As in the first embodiment, the selection mark display processing unit 144 displays the first selection mark M1 in the image display area 155 of the display unit 151 in association with the first waste W1 and associates it with the second waste W2. to display the second selection mark M2 in the image display area 155 of the display unit 151 . Further, the selection mark display processing unit 144 displays the name N1 of the material candidate of the first waste W1 in the first selection mark M1 based on the result of estimation by the material estimation unit 143 in the same manner as in the first embodiment. Then, the name N2 of the material candidate for the second waste W2 is displayed within the second selection mark M2. The selection mark display processing unit 144 of this embodiment does not display the unselected mark M, the material confirmation mark Md, and the material name Nd. The selection mark display processing section corresponds to the first selection mark display processing section and the second selection mark display processing section.

When a material designation operation is input to the operation input unit 154 while the first selection mark M1 is displayed on the display unit 151, the material information acquisition unit 145 responds to the material designation operation based on the material designation operation. Acquire the material information to be used.

When the material information acquisition unit 145 acquires the material information, the teaching data storage processing unit 146 acquires the image of the first waste W1 from the selection unit 142, and associates this image with the acquired material information to obtain the teaching data. It is generated and stored in the teacher data storage device 160 . The teacher data storage device 160 stores the teacher data generated by the teacher data storage processor 146 .

Next, an example of processing according to the present invention (teaching data generation processing) executed in the control unit 140 of the teaching data generating apparatus of the embodiment described above will be described with reference to the flowchart of FIG.

The control unit 140 displays the image of a plurality of wastes W being transported in one direction, stored in the image storage device 170, in the image display area 155 of the display unit 151 (T110). Then, the control unit 140 analyzes the image displayed on the display unit 151 and identifies a plurality of wastes W included in the image (T120).

Next, if there is a waste W to which the material information is not linked among the plurality of wastes W (Y in T130), the control unit 140 determines that the position of the upstream end Wa is the most downstream from among the wastes W. is selected as the first waste W1 (T140). The control unit 140 guesses the material based on the image of the first waste W1 (T150). Then, the control unit 140 displays the first selection mark M1, which is a frame graphic surrounding the first waste W1 (T160), and inserts the name N1 of the material candidate for the first waste W1 into the first selection mark M1. Display (T170).

Next, if there is another waste W to which the material information is not linked among the plurality of wastes W (Y in T180), the control unit 140 counts the position of the upstream end Wa from the downstream side. The second one is selected as the second waste W2 (T190). The control unit 140 guesses the material based on the image of the second waste W2 (T200). Then, the control unit 140 displays the second selection mark M2, which is a frame graphic surrounding the second waste W2 (T210), and displays the name N2, which is a material candidate for the second waste W2, in the second selection mark M2. (T220).

Next, after the second selection mark M2 is displayed (T220), or when there is no waste W to which the material information is linked among the plurality of wastes W (N in T130, N at T180), if the first waste W1 has been selected from a plurality of wastes W in the previous process (Y at T280), wait until material information is acquired (N at T290), If the first waste W1 is not selected, the processing of this flow chart ends (N in T280). When the material information is acquired (Y in T290), the control unit 140 generates teacher data by linking the image of the first waste W1 and the acquired material information, and stores the generated teacher data in the teacher data storage device 160. (T300), and the processing of this flowchart ends.

Also in the second embodiment, the same effect as in the first embodiment is obtained.

Although the first embodiment 1 and the second embodiment of the present invention have been described in detail above, the present invention is not limited to the above embodiments, and various modifications are possible. For example, in each of the above-described embodiments, the application to a sorting system for dismantled garbage or rubble generated when a building is dismantled has been described as an example. You may apply it to the system which sorts a product. At that time, since the deep learning optimization method and network configuration depend on the learning object, it is sufficient to apply the best method for the type of waste to be sorted, such as general household garbage and home appliances.

(First embodiment)
DESCRIPTION OF SYMBOLS 1... Waste sorting system, 10... Conveyor, 11... Sub-conveyor, 12... Vibrating screen device, 20... Camera, 30... Robot arm, 40... Control device, 41... Image display processing part, 42... Selection part, 43... Material estimation unit 44 Selection mark display processing unit 45 Material information acquisition unit 46 Teacher data storage processing unit 47 Movement control unit 50 Tablet terminal 51 Display unit 52 Touch panel 53 Operation Icon 54 Operation input unit 55 Image display area 56 Message area 57 Graph display area 60 Storage device 70 Collection box W Waste W1 First waste W2 Second 2 wastes, Wd... Wastes to which material information is linked, M... Unselected mark, M1... First selection mark, M2... Second selection mark, Md... Material confirmation mark, N1... Material of first waste name of candidate, N2... name of second waste material candidate, Nd... name of waste material linked to material information (second embodiment)
DESCRIPTION OF SYMBOLS 101... Teacher data generation system 110... Teacher data generation apparatus 140... Control part 141... Image display process part 142... Selection part 143... Material estimation part 144... Selection mark display process part 145... Material information acquisition Unit 146 Teaching data storage processing unit 151 Display unit 152 Touch panel 153 Operation icon 154 Operation input unit 155 Image display area 156 Message area 157 Graph display area 160 Teacher Data storage device 170 Image storage device 180 Mobile communication network 190 Internet W Waste W1 First waste W2 Second waste M1 First selection mark M2 Second 2 selection marks, N1... name of first waste material candidate, N2... name of second waste material candidate

Claims (12)

A conveying device for conveying a plurality of wastes in one direction, a photographing device for photographing the plurality of wastes conveyed by the conveying device, and a moving device for moving the wastes conveyed by the conveying device from the conveying device. , and a control device for controlling the moving device, the waste sorting system for sorting waste and generating training data in which material information indicating the material of the waste is linked to an image of the waste. There is
The control device
an image display processing unit that displays an image captured by the imaging device on a display device;
a first selection unit that selects one waste to which the material information is not linked from among a plurality of wastes included in the image;
a first selection mark display processing unit that displays a first selection mark on the display device in association with the waste selected by the first selection unit;
a material information acquisition unit that acquires material information indicating the material of the waste based on a material designation operation input to an operation input device;
a teaching data storage processing unit for storing, in a storage device, teaching data linking the image of the waste selected by the first selection unit and the material information acquired by the material information acquiring unit;
a movement control unit that controls the movement device to move the waste selected by the first selection unit from the transport device to a location corresponding to the material information acquired by the material information acquisition unit;
a first material estimation unit for estimating the material of the waste based on the image of the waste selected by the first selection unit ;
The waste sorting system , wherein the first selection mark display processing section displays the name of the material guessed by the first material guessing section together with the first selection mark on the display device . 2. The waste sorting system according to claim 1, wherein the first selection unit selects the waste whose upstream end position is closest to the downstream from among the wastes to which the material information is not linked. a second selection unit that selects one waste that is different from the waste selected by the first selection unit and that is not associated with the material information from among the plurality of wastes included in the image;
A second selection mark display processing unit that displays a second selection mark on the display device in association with the waste selected by the second selection unit. Item 3. The waste sorting system according to item 2. 4. The disposal according to claim 3, wherein the second selection unit selects the waste having the second upstream end position counted from the downstream side among the wastes to which the material information is not linked. object sorting system. The first material estimation unit performs machine learning for estimating the material of the waste selected by the first selection unit, using the teacher data stored in the storage device. The waste sorting system according to any one of claims 1 to 4 . The control device further comprises a second material estimation unit for estimating the material of the waste based on the image of the waste selected by the second selection unit,
5. The display device according to claim 3, wherein the second selection mark display processing unit displays the name of the material guessed by the second material guessing unit together with the second selection mark on the display device. A waste sorting system as described. The first material estimation unit performs machine learning for estimating the material of the waste selected by the first selection unit using the teacher data stored in the storage device,
The second material estimation unit performs machine learning for estimating the material of the waste selected by the second selection unit, using the teacher data stored in the storage device. 7. A waste sorting system according to Item 6 . The first selection mark is a frame graphic surrounding the waste selected by the first selection unit, or a balloon graphic connected to the waste. A waste sorting system as described in paragraph 1. The first selection mark is a frame graphic surrounding the waste selected by the first selection unit, or a balloon graphic connected to the waste,
Claims 3, 4, and 6 , wherein the second selection mark is a frame figure surrounding the waste selected by the second selection unit, or a balloon figure connected to the waste. and a waste sorting system according to claim 7 . A training data generation device for generating training data in which material information indicating a material of the waste is linked to an image of the waste,
an image display processing unit that displays an image of a plurality of wastes being transported in one direction on a display device;
a first selection unit that selects one waste to which the material information is not linked from among a plurality of wastes included in the image;
a first selection mark display processing unit that displays a first selection mark on the display device in association with the waste selected by the first selection unit;
a material information acquisition unit that acquires material information indicating the material of the waste based on a material designation operation input to an operation input device;
a teaching data storage processing unit for storing, in a storage device, teaching data linking the image of the waste selected by the first selection unit and the material information acquired by the material information acquiring unit;
a first material estimation unit for estimating the material of the waste based on the image of the waste selected by the first selection unit ;
The teacher data generating apparatus , wherein the first selection mark display processing section displays the name of the material guessed by the first material guessing section together with the first selection mark on the display device. A training data generation system for generating training data in which material information indicating a material of the waste is linked to an image of the waste,
an image storage device storing an image of a plurality of wastes being transported in one direction;
and a training data generation device according to claim 10 ,
A teacher data generation system, wherein the image display processing unit of the teacher data generation device displays the image stored in the image storage device on the display device. A training data generation program for generating training data in which material information indicating a material of the waste is linked to an image of the waste,
the computer,
an image display processing unit that displays an image of a plurality of wastes being conveyed in one direction on a display device;
a first selection unit that selects one waste to which the material information is not linked from among a plurality of wastes included in the image;
a first selection mark display processing unit that displays a first selection mark on the display device in association with the waste selected by the first selection unit;
a material information acquisition unit that acquires material information indicating the material of the waste based on a material designation operation input to an operation input device ;
a teaching data storage processing unit for storing, in a storage device, teaching data linking the image of the waste selected by the first selection unit and the material information acquired by the material information acquiring unit ;
functioning as a first material estimation unit that estimates the material of the waste based on the image of the waste selected by the first selection unit ;
A teacher data generating program , wherein the first selection mark display processing unit displays the name of the material guessed by the first material guessing unit together with the first selection mark on the display device .

Images