JP7531412B2 - Machining surface determination device, machining surface determination program, machining surface determination method, machining system, inference device, and machine learning device - Google Patents

Description

The present invention relates to a machining surface determination device, a machining surface determination program, a machining surface determination method, a machining system, an inference device, and a machine learning device.

In recent years, in the manufacturing process of various products, instead of workers visually judging the quality of products, development has been progressing on devices that use various sensors to automatically judge the quality of products. For example, Patent Document 1 discloses an inspection device that inspects the external shape of an impeller by performing binarization processing on an image of the impeller, which is the object to be inspected.

JP 2008-51664 A

One of the indicators for judging the quality of a product is the state of the processed surface after various processing steps such as polishing, grinding, cutting, casting, etc. The state of the processed surface includes various evaluation items such as roughness, unevenness, waviness, warping, patterns, streaks, and waves.

However, the inspection device disclosed in Patent Document 1 inspects the external shape of the object being inspected, but is unable to determine the condition of the machined surface of the object being inspected. Furthermore, if an operator were to determine the condition of the machined surface, this would depend on the operator's skill and experience (including tacit knowledge), leading to large individual differences between operators and making it difficult to guarantee product quality.

In view of the above-mentioned problems, the present invention aims to provide a machined surface determination device, a machined surface determination program, a machined surface determination method, a machining system, an inference device, and a machine learning device that are capable of automatically determining the state of the machined surface of an object to be determined.

In order to achieve the above object, a machined surface judgment device according to one aspect of the present invention comprises:
A machined surface judgment device that judges a state of a machined surface of an object to be judged based on an image for judgment obtained by capturing an image of the machined surface, comprising:
a classification result acquisition unit that acquires, for each of a plurality of small image regions obtained by dividing a judgment image region of the judgment image, a classification result when classifying the state of the processed surface into one of a plurality of processing states for each of the small image regions;
The apparatus further includes a judgment result inference unit that infers the judgment result for the judgment image by inputting the classification results for the plurality of small image regions into a judgment learning model that has been machine-learned to determine the correlation between the classification results for a plurality of learning image regions corresponding to the plurality of small image regions and a judgment result when the state of the processed surface in the plurality of learning image regions is judged based on the classification results.

According to the machined surface judgment device of the present invention, the judgment result inference unit infers the judgment result for the judgment image by inputting the classification results for each small image region when the judgment image region of the judgment image is divided into a plurality of small image regions into the judgment learning model. Therefore, the state of the machined surface of the judgment target object can be automatically judged.

Other issues, configurations and advantages will be made clear in the detailed description of the invention described below.

1 is a schematic configuration diagram showing an example of a machining system 1 including a machining surface determination device 7 according to a first embodiment. FIG. 2 is a hardware configuration diagram showing an example of a computer 200 constituting the machine learning device 6 and the machining surface determination device 7. FIG. 2 is a block diagram showing an example of a machine learning device 6 according to the first embodiment. FIG. 11 is a data configuration diagram illustrating an example of first classification learning data. FIG. 2 is a data configuration diagram showing an example of judgment learning data. A schematic diagram showing an example of an inference model 20 applied to the first classification learning model 2A. A schematic diagram showing an example of an inference model 20 applied to the judgment learning model 2. 2 is a block diagram showing an example of a machining surface determining device 7 according to the first embodiment. FIG. 11 is a functional explanatory diagram showing an example of a classification result acquisition process performed by a classification result acquisition unit 70A. FIG. 11 is a functional explanatory diagram showing an example of a judgment result inference process performed by a judgment result inference unit 71. FIG. 4 is a flowchart showing an example of a machining surface determination method by the machining surface determination device 7 according to the first embodiment. FIG. 11 is a block diagram showing an example of a machine learning device 6 according to a second embodiment. FIG. 11 is a data configuration diagram illustrating an example of second classification learning data. A schematic diagram showing an example of an inference model 20B applied to the second classification learning model 2B. FIG. 11 is a block diagram showing an example of a machining surface determining device 7 according to a second embodiment. FIG. 13 is a functional explanatory diagram showing an example of a classification result acquisition process performed by a classification result acquisition unit 70B. 10 is a flowchart showing an example of a machining surface determination method by a machining surface determination device 7 according to a second embodiment.

The following describes an embodiment of the present invention with reference to the drawings. The following shows a schematic view of the scope of the description necessary to achieve the object of the present invention, and mainly describes the scope of the description necessary to explain the relevant parts of the present invention. The parts that are omitted are based on publicly known technology.

First Embodiment
FIG. 1 is a schematic configuration diagram showing an example of a machining system 1 including a machining surface determining device 7 according to a first embodiment.

The processing system 1 includes a processing unit 3 that processes an object 10 to be determined, an imaging unit 4 that images a processed surface 100 of the object 10 to be determined, a processed surface determination device 7 that determines the state of the processed surface 100 of the object 10 to be determined using a first classification learning model 2A and a determination learning model 2, and a control device 5 that controls the processing unit 3, the imaging unit 4, and the processed surface determination device 7. In addition, the processing system 1 includes, as an additional component, a machine learning device 6 that generates the first classification learning model 2A and the determination learning model 2.

The object to be evaluated 10 is any article that is formed from any material, such as metal, resin, ceramics, etc., and is to be processed by the processing unit 3. A specific example of the object to be evaluated 10 is a fluid machine or a fluid component that constitutes a fluid machine. Note that there are no particular limitations on the three-dimensional shape, surface properties, color, size, etc., of the object to be evaluated 10.

The processed surface 100 is, for example, the surface of the object to be evaluated 10 when the object to be evaluated 10 is processed by the processing unit 3. The processed surface 100 may be any surface of the object to be evaluated 10, and may be the entire surface of the object to be evaluated 10 or a part of it.

The processing unit 3 is composed of various robot manipulators and machining mechanism units of machine tools that operate using electricity, fluid pressure, etc. as a driving source. The processing unit 3 performs processing steps such as polishing, grinding, cutting, or casting based on control commands from the control device 5. The processing unit 3 may perform any processing step that processes or forms the surface of the evaluation target object 10, and may also perform a combination of multiple processing steps.

In the processing system 1 shown in FIG. 1, the processing unit 3 is composed of a robot manipulator with a replaceable grinding wheel attached to its tip, and performs the grinding process. The object 10 to be evaluated is an impeller having multiple blades as a fluid component that constitutes a pump, and the processed surface 100 is the surface of each blade that has been processed by the grinding process performed by the processing unit 3.

The imaging unit 4 is a camera that captures the machining surface 100, and is composed of an image sensor such as a CMOS sensor or a CCD sensor. The imaging unit 4 is attached to a predetermined position where it can capture an image of the machining surface 100. If the machining unit 3 is composed of, for example, a robot manipulator, the imaging unit 4 may be attached to the tip of the robot manipulator, or may be fixed above a mounting table (including movable types) on which the evaluation target object 10 is placed. Also, if the machining unit 3 is composed of, for example, a machining mechanism part of a machine tool, the imaging unit 4 may be attached inside the safety cover of the machine tool, or may be fixed above a work table separate from the machine tool.

The imaging unit 4 is attached at a predetermined position as described above, and its position and orientation are adjusted so that the machining surface 100 fits within the angle of view of the imaging unit 4. As shown in FIG. 1, the imaging unit 4 may be provided with an imaging unit 4 connected to the machine learning device 6 and an imaging unit 4 connected to the machining surface determination device 7 separately, or one imaging unit 4 may be connected to both the machine learning device 6 and the machining surface determination device 7 and used in common. The imaging unit 4 may also have pan/tilt/zoom functions. Furthermore, the imaging unit 4 is not limited to imaging the machining surface 100 with one camera, and may be imaging with multiple cameras.

The control device 5 includes a control panel 50 that is configured, for example, by a general-purpose or dedicated computer (see FIG. 2 below) or a microcontroller, and an operation display panel 51 that is configured by a touch panel display, switches, buttons, etc.

The control panel 50 is connected to the actuators and sensors (neither shown) of the machining unit 3, and controls the machining process by the machining unit 3 by sending control commands to the actuators according to machining operation parameters for performing the machining process and detection signals from the sensors. The control panel 50 sends an imaging command to the imaging unit 4, and as a result, receives a captured image captured by the imaging unit 4. The control panel 50 sends the captured image to the machining surface determination device 7 as a judgment image, and as a result, receives the state of the machining surface 100 judged by the machining surface determination device 7. The control panel 50 may send the captured image to the machine learning device 6.

The operation display panel 51 accepts operations by the operator and outputs various information by display and sound.

The machine learning device 6 operates as the subject of the learning phase in machine learning. The machine learning device 6 acquires learning data based on the captured image captured by the imaging unit 4, and generates a first classification learning model 2A and a judgment learning model 2 based on the learning data. The machine learning device 6 provides the trained first classification learning model 2A and judgment learning model 2 to the machining surface judgment device 7 via any communication network, recording medium, etc. Details of the machine learning device 6 will be described later.

The machining surface determination device 7 operates as the subject of the inference phase in machine learning. The machining surface determination device 7 uses the trained first classification learning model 2A and the judgment learning model 2 generated by the machine learning device 6 to determine the state of the machining surface 100 of the judgment target object 10 using the image of the machining surface 100 captured by the imaging unit 4 as a judgment image. Details of the machining surface determination device 7 will be described later.

The components of the machining system 1 may be incorporated into a single housing to form, for example, a single machine tool. In that case, at least one of the machine learning device 6 and the machining surface determination device 7 may be incorporated into the control device 5. The components of the machining system 1 may be composed of a machining device having a machining unit 3 and an inspection device having an imaging unit 4 and a machining surface determination device 7. In that case, the functions of the control device 5 may be distributed between the machining device and the inspection device. Furthermore, the components of the machining system 1 may be connected by a wireless or wired network, and at least one of the machine learning device 6 and the machining surface determination device 7 may be installed in a location away from the machining site where the machining unit 3 and the imaging unit 4 are installed. In that case, the control device 5 may be installed at the machining site or elsewhere.

Figure 2 is a hardware configuration diagram showing an example of a computer 200 that constitutes the machine learning device 6 and the machining surface determination device 7.

Each of the machine learning device 6 and the machining surface determination device 7 is configured by a general-purpose or dedicated computer 200. As shown in FIG. 2, the computer 200 has, as its main components, a bus 210, a processor 212, a memory 214, an input device 216, a display device 218, a storage device 220, a communication I/F (interface) unit 222, an external device I/F unit 224, an I/O (input/output) device I/F unit 226, and a media input/output unit 228. Note that the above components may be omitted as appropriate depending on the purpose for which the computer 200 is used.

The processor 212 is composed of one or more arithmetic processing devices (CPU, MPU, GPU, DSP, etc.) and operates as a control unit that controls the entire computer 200. The memory 214 stores various data and programs 230, and is composed of, for example, a volatile memory (DRAM, SRAM, etc.) that functions as a main memory, and a non-volatile memory (ROM, flash memory, etc.).

The input device 216 is, for example, a keyboard, a mouse, a numeric keypad, an electronic pen, etc. The display device 218 is, for example, a liquid crystal display, an organic EL display, electronic paper, a projector, etc. The input device 216 and the display device 218 may be integrally configured, such as a touch panel display. The storage device 220 is, for example, a HDD, an SSD, etc., and stores various data necessary for executing the operating system and the program 230.

The communication I/F unit 222 is connected to a network 240 such as the Internet or an intranet by wire or wirelessly, and transmits and receives data to and from other computers according to a predetermined communication standard. The external device I/F unit 224 is connected to an external device 250 such as a printer or scanner by wire or wirelessly, and transmits and receives data to and from the external device 250 according to a predetermined communication standard. The I/O device I/F unit 226 is connected to an I/O device 260 such as various sensors and actuators, and transmits and receives various signals and data, such as detection signals from sensors and control signals to actuators, between the I/O device 260. The media input/output unit 228 is composed of a drive device such as a DVD drive or CD drive, and reads and writes data from and to media 270 such as DVDs and CDs.

In the computer 200 having the above configuration, the processor 212 calls up the program 230 into the work memory area of the memory 214, executes it, and controls each part of the computer 200 via the bus 210. The program 230 may be stored in the storage device 220 instead of the memory 214. The program 230 may be recorded in a non-transient recording medium such as a CD or DVD in an installable file format or an executable file format, and provided to the computer 200 via the media input/output unit 228. The program 230 may be provided to the computer 200 by downloading it via the network 240 via the communication I/F unit 222. In addition, the computer 200 may realize various functions realized by the processor 212 executing the program 230, for example, with hardware such as an FPGA or ASIC.

The computer 200 is, for example, a stationary computer or a portable computer, and is an electronic device of any form. The computer 200 may be a client-type computer, a server-type computer, or a cloud-type computer. The computer 200 may be applied to devices other than the machine learning device 6 and the machining surface determination device 7.

(Machine Learning Device 6)
FIG. 3 is a block diagram showing an example of the machine learning device 6 according to the first embodiment.

The machine learning device 6 includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a trained model storage unit 63. The machine learning device 6 is configured, for example, by the computer 200 shown in FIG. 2. In this case, the learning data acquisition unit 60 is configured by the communication I/F unit 222 or the I/O device I/F unit 226 and the processor 212, the machine learning unit 62 is configured by the processor 212, and the learning data storage unit 61 and the trained model storage unit 63 are configured by the storage device 220.

The learning data acquisition unit 60 is an interface unit that is connected to various external devices via a communication network and acquires learning data in which input data and output data are associated. Examples of the external devices include the imaging unit 4, the machining surface determination device 7, and the worker terminal 8 used by the worker.

The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60. The learning data includes first classification learning data for generating the first classification learning model 2A, and judgment learning data for generating the judgment learning model 2. The specific configuration of the database that constitutes the learning data storage unit 61 may be designed as appropriate.

The machine learning unit 62 performs machine learning using the learning data stored in the learning data storage unit 61. That is, the machine learning unit 62 inputs multiple sets of first classification learning data to the first classification learning model 2A, and generates the first classification learning model 2A by having the first classification learning model 2A learn the correlation between the input data and the output data included in the first classification learning data by machine learning. The machine learning unit 62 also inputs multiple sets of judgment learning data to the judgment learning model 2, and has the judgment learning model 2 learn the correlation between the input data and the output data included in the judgment learning data by machine learning, thereby generating the judgment learning model 2.

The trained model storage unit 63 is a database that stores the first classification learning model 2A and the judgment learning model 2 generated by the machine learning unit 62. The first classification learning model 2A and the judgment learning model 2 stored in the trained model storage unit 63 are provided to an actual system (e.g., the machining surface judgment device 7) via any communication network, recording medium, etc. The first classification learning model 2A and the judgment learning model 2 may be provided to an external computer (e.g., a server-type computer or a cloud-type computer) and stored in the memory of the external computer. In addition, although the training data storage unit 61 and the trained model storage unit 63 are shown as separate storage units in FIG. 3, they may be configured as a single storage unit.

Figure 4 is a data structure diagram showing an example of the first classification learning data.

The first classification learning data has a learning image 41 as input data, and includes as output data a classification result in which the state of the machining surface 100 contained in the learning image 41 is classified into one of a number of machining states, and is configured by associating these input data and output data.

The learning images 41 as input data are each a plurality of images generated by dividing an image 40 having a predetermined captured image area 400 in which the processed surface 100 of the object to be determined 10 is captured by the imaging unit 4 into learning image areas 410.

The captured image area 400 of the captured image 40 is an area captured by the imaging unit 4, and is determined by the angle of view of the imaging unit 4. The captured image area 400 shown in FIG. 4 is set to include a portion of one blade of the impeller, which is the object to be determined 10. Note that the captured image 40 shown in FIG. 4 captures not only the processing surface 100, but also the background 110, but the captured image area 400 may be set so that the background 110 is not captured.

As shown in FIG. 4, the learning image areas 410 of the learning image 41 are obtained by dividing the captured image area 400 of the captured image 40 into a grid pattern so that each learning image area 410 is a square. The number of images, shape, size, and aspect ratio of the learning image areas 410 may be changed as appropriate, and may be, for example, rectangular or another shape. The division method for dividing the captured image area 400 into the learning image areas 410 may be changed as appropriate, and may be, for example, a staggered division or division according to other criteria.

In supervised learning, the classification results as output data are referred to as, for example, teacher data or correct answer labels. When, for example, two classes, "good" and "poor", are adopted as the multiple machining states, the classification results are expressed as either "good" or "poor". When, for example, three classes, "good", "fair", and "poor", are adopted as the multiple machining states, the classification results are expressed as either "good", "fair", or "poor". Note that the multiple machining states when classifying the state of the machining surface 100 are not limited to the above classes, and may be classified into, for example, four or more classes, or may be classified from other perspectives.

Furthermore, if the edge of the machining surface 100 or the background 110 other than the machining surface 100 is present in the learning image area 410, it is also possible to add a class called "not to be judged" and classify it. In the above two-class example, the classification result when classifying as "not to be judged" because the edge of the machining surface 100 or the background 110 other than the machining surface 100 is present in the learning image area 410 is expressed as either "good", "poor" or "not to be judged", and in the above three-class example, it is expressed as either "good", "fair", "poor" or "not to be judged" as shown in FIG. 4. Note that, if both the machining surface 100 and the background 110 are captured in the learning image 41, for example, if the ratio of the background 110 is higher than a predetermined ratio, it may be classified into the "not to be judged" class, or it may not always be classified into the "not to be judged" class.

Figure 5 is a data structure diagram showing an example of judgment learning data.

The judgment learning data includes, as input data, the classification results when the state of the machined surface 100 for each of the multiple learning image areas 410 is classified into one of multiple machining states, and, as output data, the judgment results when the state of the machined surface 100 in each of the multiple learning image areas 410 is judged based on the classification results, and is configured by associating these input data and output data.

The classification results for multiple learning image regions 410 as input data are expressed as integer values of "0", "1", "2" and "3" when the condition of the machined surface 100 is classified as, for example, "good", "fair", "poor" or "not applicable".

In supervised learning, the judgment results as output data are referred to as, for example, teacher data or correct answer labels. The judgment results are obtained by judging the overall state of the machining surface 100 for the multiple learning image regions 410, i.e., the captured image region 400 before being divided into the multiple learning image regions 410.

The judgment result is a judgment of at least one of the following conditions of the machined surface 100: whether re-machining is required, in which the same machining process as when the machined surface 100 was machined is performed again; whether separate machining is required, in which a machining process different from that when the machined surface 100 was machined is performed; whether finishing is required, in which the worker performs finishing on the machined surface 100; and the machining range of the machined surface 100 to be subjected to re-machining, separate machining, or finishing. Note that the judgment result may be, instead of or in addition to the above, a judgment of whether the entire machined surface 100 is in one of a number of machining conditions including at least "good" and "bad".

The learning data acquisition unit 60 can employ various methods for acquiring the first classification learning data and the judgment learning data. For example, the learning data acquisition unit 60 acquires an image 40 captured by the imaging unit 4 of the judgment target object 10 after the processing step is performed by the processing unit 3, and generates multiple learning images 41 by dividing the captured image 40. Next, the learning data acquisition unit 60 displays the multiple learning images 41 on the display screen of the worker terminal 8 in a distinguishable state, for example, by superimposing a frame line constituting each learning image area 410 on the captured image 40.

The worker visually checks each of the learning images 41 on the display screen, and inputs the results (classification results) of classifying the states of the machining surface 100 included in each of the multiple learning images 41 into multiple machining states (classes), and also inputs the results (determination results) of judging the state of the machining surface 100 included in the captured image 40 through the worker terminal 8. Then, the learning data acquisition unit 60
accepts the input operation of the worker and acquires a plurality of first classification learning data by associating the learning image 41 (input data) with the classification result (output data) inputted for the learning image 41. The learning data acquisition unit 60 acquires judgment learning data by associating the classification result (input data) for the plurality of learning image areas 410 of each of the learning images 41 with the judgment result (output data) inputted for the captured image 40.

Therefore, the learning data acquisition unit 60 can acquire a number of first classification learning data corresponding to the number of divisions when one captured image 40 is divided into multiple learning images 41, and can further acquire a desired number of first classification learning data by repeating the above operation. Furthermore, the learning data acquisition unit 60 can acquire judgment learning data in conjunction with acquiring the first classification learning data. Therefore, the first classification learning data and judgment learning data can be easily collected.

Figure 6 is a schematic diagram showing an example of an inference model 20A applied to the first classification learning model 2A.

The inference model 20A employs a convolutional neural network (CNN) as a specific method of machine learning. The inference model 20A includes an input layer 21, an intermediate layer 22, and an output layer 23.

The input layer 21 has neurons whose number corresponds to the number of pixels in the training image 41 as input data, and the pixel value of each pixel is input to each neuron.

The intermediate layer 22 is composed of a convolutional layer 22a, a pooling layer 22b, and a fully connected layer 22c. The convolutional layer 22a and the pooling layer 22b are provided, for example, in multiple alternating layers. The convolutional layer 22a and the pooling layer 22b extract features from the image input via the input layer 21. The fully connected layer 22c converts the features extracted from the image by the convolutional layer 22a and the pooling layer 22b, for example, by an activation function, and outputs them as a feature vector. Note that the fully connected layer 22c may be provided in multiple layers.

The output layer 23 outputs output data including the classification result based on the feature vector output from the fully connected layer 22c. In addition to the classification result, the output data may also include, for example, a score indicating the reliability of the classification result.

Between each layer of the inference model 20A, there are synapses that connect the neurons between the layers, and weights are assigned to each synapse in the convolutional layer 22a and the fully connected layer 22c of the intermediate layer 22.

The machine learning unit 62 inputs the first classification learning data into the inference model 20A, and causes the inference model 20A to machine-learn the correlation between the learning image 41 and the classification result. Specifically, the machine learning unit 62 inputs the learning image 41 constituting the first classification learning data as input data to the input layer 21 of the inference model 20A. Note that the machine learning unit 62 may apply a predetermined image adjustment (e.g., image format, image size, image filter, image mask, etc.) to the learning image 41 as pre-processing when inputting the learning image 41 to the input layer 21.

The machine learning unit 62 repeatedly adjusts the weights associated with each synapse (backpropagation) so as to reduce the evaluation value of the error function using an error function that compares the classification result (inference result) indicated by the output data output from the output layer 23 with the classification result (teacher data) constituting the first classification learning data. When the machine learning unit 62 determines that a predetermined learning termination condition, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than an allowable value, is satisfied, the machine learning unit 62 terminates the machine learning and stores the inference model 20A at that time (all weights associated with each synapse) in the learned model storage unit 63 as the first classification learning model 2A.

Figure 7 is a schematic diagram showing an example of an inference model 20 applied to the judgment learning model 2.

As a specific method of machine learning, inference model 20 employs a convolutional neural network, similar to inference model 20A shown in FIG. 6. The following describes inference model 20, focusing on the differences between inference model 20A shown in FIG. 6.

The input layer 21 has neurons whose number corresponds to the number of divisions when the captured image area 400 is divided into multiple learning image areas 410, and the classification result for each learning image area 410 (e.g., integer values of 0, 1, 2, and 3) is input to each neuron.

The output layer 23 outputs output data including a judgment result based on the feature vector output from the fully connected layer 22c. In addition to the judgment result, the output data may also include, for example, a score indicating the reliability of the judgment result.

The machine learning unit 62 inputs the judgment learning data to the inference model 20 and causes the inference model 20 to machine-learn the correlation between the classification results for the multiple learning image regions 410 and the judgment results. Specifically, the machine learning unit 62 inputs the classification results for the multiple learning image regions 410, which constitute the judgment learning data, as input data to the input layer 21 of the inference model 20.

The machine learning unit 62 iteratively adjusts (backpropagates) the weights associated with each synapse so as to reduce the evaluation value of the error function using an error function that compares the judgment result (inference result) indicated by the output data output from the output layer 23 with the judgment result (teacher data) constituting the judgment learning data. When the machine learning unit 62 determines that a predetermined learning termination condition is met, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than an allowable value, the machine learning is terminated and the inference model 20 at that time (all weights associated with each synapse) is stored in the trained model storage unit 63 as the judgment learning model 2.

(Machine surface determination device 7)
FIG. 8 is a block diagram showing an example of the machining surface determining device 7 according to the first embodiment.

The machining surface determination device 7 includes a classification result acquisition unit 70A, a judgment result inference unit 71, a learned model storage unit 72, and an output processing unit 73. The machining surface determination device 7 is configured, for example, by a computer 200 as shown in FIG. 2. In this case, the classification result acquisition unit 70A is configured by a communication I/F unit 222 or an I/O device I/F unit 226 and a processor 212, the judgment result inference unit 71 and the output processing unit 73 are configured by the processor 212, and the learned model storage unit 72 is configured by a storage device 220.

The classification result acquisition unit 70A performs a classification result acquisition process (see FIG. 9 described later) for each of the multiple small image areas 430 into which the judgment image area 420 of the judgment image 42 is divided, to acquire the classification result when the state of the machining surface 100 is classified into one of multiple machining states.

The classification result acquisition unit 70A specifically includes an image acquisition unit 700 connected to the imaging unit 4 and acquiring an image of the processed surface 100 of the object to be determined 10 captured by the imaging unit 4 as a judgment image 42 having a judgment image area 420, a small image generation unit 701 that generates a plurality of small images 43 from the judgment image 42 by dividing the judgment image area 420 into a plurality of small image areas 430, and a first classification result inference unit 702A that infers classification results for the plurality of small image areas 430 by inputting the plurality of small images 43 into a first classification learning model 2A on a small image area 430 basis.

The first classification result inference unit 702A records, for example, the positional relationship of each small image area 430 to the judgment image area 420 as additional information of the small image 43 so that the pre-division judgment image 42 can be reconstructed from the multiple small images 43.

The judgment result inference unit 71 performs a judgment result inference process (see FIG. 10 described later) to infer a judgment result for the judgment image area 420 by inputting the classification results for the multiple small image areas 430 acquired by the classification result acquisition unit 70A to the judgment learning model 2.

The judgment result inferred by the judgment result inference unit 71 is a judgment of at least one of the necessity of re-machining, the necessity of separate machining, the necessity of finish machining, and the machining range of the machining surface 100 to be subjected to re-machining, separate machining, or finish machining. Instead of or in addition to the above, the judgment result may be a judgment of whether the entire machining surface 100 is in one of a number of machining states including at least "good" and "bad."

In addition, part or all of the classification result acquisition unit 70A and the judgment result inference unit 71 may be substituted by a processor of an external computer (e.g., a server-type computer or a cloud-type computer), and part or all of the classification result acquisition process by the classification result acquisition unit 70A and the judgment result inference process by the judgment result inference unit 71 may be executed by the external computer.

The trained model storage unit 72 is a database that stores the trained first classification learning model 2A used in the inference process of the classification result acquisition unit 70A and the trained judgment learning model 2 used in the inference process of the judgment result inference unit 71. The number of the first classification learning model 2A and the judgment learning model 2 stored in the trained model storage unit 72 is not limited to one, and for example, multiple trained models with different conditions such as machine learning methods, processing steps by the processing unit 3, and judgment target object 10 may be stored and selectively used. The trained model storage unit 72 may be substituted by a storage unit of an external computer (for example, a server-type computer or a cloud-type computer), and in that case, the classification result acquisition unit 70A and the judgment result inference unit 71 may access the external computer to perform the above-mentioned classification result acquisition process and judgment result inference process.

The output processing unit 73 performs output processing for outputting the judgment result inferred by the judgment result inference unit 71. As a specific output means for outputting the judgment result, various means can be adopted. For example, the output processing unit 73 may transmit an operation command for reprocessing or another processing to the processing unit 3 via the control panel 50 according to the judgment result, notify the operator of the execution of the finishing processing by display or sound via the operation display panel 51 or the operator terminal 8, or store the result in the storage means of the control panel 50 as the operation history of the processing unit 3. Note that the output processing unit 73 may only output (transmit, notify, store) the judgment result by the judgment result inference unit 71, or may further output (transmit, notify, store) the classification result for the multiple small image areas 430 by the classification result acquisition unit 70A in addition to the judgment result by the judgment result inference unit 71.

Figure 9 is a functional diagram showing an example of the classification result acquisition process performed by the classification result acquisition unit 70A.

The image area for determination 420 of the image for determination 42 is an area captured by the imaging unit 4, and is determined by the angle of view of the imaging unit 4. The image area for determination 420 shown in Fig. 9 is set to include a part of one blade of the impeller, which is the object to be determined 10, similar to the captured image area 400 shown in Fig. 4. Note that the image area for determination 420 may be set at a position different from that of the captured image area 400, and the number of images, shape, size, and aspect ratio of the two may be different.

As shown in FIG. 9, the small image regions 430 of the small image 43 are obtained by dividing the judgment image region 420 of the judgment image 42 into a grid pattern so that each small image region 430 is a square. The small image regions 430 of the small image 43 correspond to the learning image region 410 of the learning image 41 when the first classification learning model 2A is generated by the machine learning device 6, and it is preferable that the number of images, shape, size, and aspect ratio of both images are the same or similar.

Therefore, if the number of images, shape, size, and aspect ratio of the small image area 430 correspond to the number of images, shape, size, and aspect ratio of the learning image area 410, the division method for dividing the judgment image area 420 into the small image areas 430 may be changed as appropriate, for example, it may be divided in a staggered pattern, or it may be divided according to other criteria. In this case, the division method for dividing the judgment image area 420 into the small image areas 430 may be the same as or different from the division method for dividing the captured image area 400 into the learning image areas 410.

Here, the first classification learning model 2A is a model obtained by machine learning the correlation between the learning image 41 having the learning image area 410 corresponding to the small image area 430 and the classification result in which the state of the machining surface 100 included in the learning image 41 is classified into one of a plurality of machining states, by the machine learning device 6. Therefore, the first classification result inference unit 702A functions as a classifier that classifies the state of the machining surface 100 in the small image area 430 into one of a plurality of machining states by inputting a plurality of small images 43 into the first classification learning model 2A in units of small image area 430. When two classes (good, bad) are adopted as the plurality of machining states, the classification result is expressed by two classes (good, bad), and when three classes (good, fair, bad) are adopted as the plurality of machining states, the classification result is expressed by three classes (good, fair, bad).

In addition, the first classification learning model 2A may be a model in which the learning image 41 in which at least one of the machining surface 100 or the background 110 other than the machining surface 100 is captured is machine-learned by the machine learning device 6 to classify the state of the machining surface 100 captured in the learning image 41 into one of a plurality of machining states, or the classification result when the state is classified as not to be determined due to the presence of the edge of the machining surface 100 or the background 110 other than the machining surface 100 in the small image area 430 of the learning image 41. In this case, the first classification result inference unit 702A of the classification result acquisition unit 70A inputs a plurality of small images 43 into the first classification learning model 2A in units of small image areas 430, and functions as a classifier that classifies the state of the machining surface 100 in the small image area 430 into one of a plurality of machining states, or classifies as not to be determined due to the presence of the edge of the machining surface 100 or the background 110 other than the machining surface 100 in the small image area 430. The classification results include not eligible for judgment in addition to the multiple processing states, so the classification results in the above example are expressed as three classes (good, poor, not eligible for judgment) or four classes (good, fair, poor, not eligible for judgment).

The classification result for the small image region 430 may include a score (confidence) for each class. In this case, if the classification result is expressed as four classes (good, fair, poor, not applicable), the scores for each class for a specific small image region 430 are output as, for example, "0.02", "0.10", "0.95", and "0.31". Any method may be adopted for using the scores. For example, the class with the highest score (in the above example, poor with a score of "0.95") may be used as the classification result, or the score of a predetermined class may exceed a predetermined score reference value (in the above example, the "poor" class has a score of "0.9"), and the score of the "poor" class may be output as the classification result.
5 exceeds the score reference value of 0.80), the class may be determined as the classification result.

In addition, it is preferable to store the classification results for the small image region 430 in the trained model storage unit 72 or another storage device (not shown), and past classification results can be used, for example, as first classification learning data to be used for online learning or re-learning in order to further improve the inference accuracy of the trained first classification learning model 2A.

Figure 10 is a functional explanatory diagram showing an example of the judgment result inference process by the judgment result inference unit 71. In the following, it is assumed that one judgment image 42 is divided into 60 small images 43 by dividing the judgment image area 420 into 60 small image areas 430, as shown in Figure 10.

The learning model for judgment 2 is a model that has been machine-learned to learn the correlation between the classification results for the multiple learning image areas 410 corresponding to the multiple small image areas 430 and the judgment results when the state of the machining surface 100 in the multiple learning image areas 410 is judged based on the classification results. Therefore, the judgment result inference unit 71 infers the state of the machining surface 100 in the multiple small image areas 430, i.e., the judgment result for the machining surface 100 in the judgment image area 420, by inputting the classification results for the multiple small image areas 430 acquired by the classification result acquisition unit 70A to the learning model for judgment 2.

When the state of the machining surface 100 is determined to be whether re-machining, additional machining, or finishing is required, the judgment result is expressed, for example, as a real value ranging from 0 to 1, with the closer to "0" the more "no" and the closer to 1 the more "required." Furthermore, when the machining range is used as the state of the machining surface 100, the classification result for the multiple small image areas 430 is determined to be, for example, the range that includes at least the small image areas 430 classified as "defective."

The judgment result inference unit 71 may perform a predetermined post-processing on the judgment result inferred by the judgment learning model 2 as described above. For example, as post-processing, the judgment result inference unit 71 may compare the judgment result values for whether reprocessing is necessary, the judgment result values for whether separate processing is necessary, and the judgment result values for whether finishing processing is necessary, and select the processing with the largest judgment result value among them, and use this as the final judgment result.

(Method of determining machined surface)
Fig. 11 is a flow chart showing an example of a machining surface determination method by the machining surface determination device 7 according to the first embodiment. The series of machining surface determination methods shown in Fig. 11 are repeatedly executed by the machining surface determination device 7 at a predetermined timing. The predetermined timing may be any timing, for example, after the machining process by the machining unit 3 is completed, during the machining process, or when a predetermined event occurs (when an operator operates, when an instruction is given from the production management system, etc.). In the following, a case will be described in which the machining surface determination method is executed on the determination object 10 machined by the machining process after the machining process by the machining unit 3 is completed.

First, in step S100, when the processing step by the processing unit 3 is completed, the processing surface 100 of the evaluation object 10 processed by the processing step is imaged by the imaging unit 4, and the captured image is sent to the processing surface evaluation device 7 via the control device 5, and the image acquisition unit 700 of the classification result acquisition unit 70A acquires the captured image as an evaluation image 42.

Next, in step S110, the small image generating unit 701 performs preprocessing on the determination image 42 by dividing the determination image area 420 of the determination image 42 into a plurality of small image areas 430, thereby generating a plurality of small images 43 from the determination image 42.

Next, in steps S120 to S128, the first classification result inference unit 702A performs loop processing by incrementing the variable i from "1" to "K" with K being the division number of the multiple small images 43 and assigning serial numbers (1≦n≦K) to each of the multiple small images 43.

Specifically, in step S120, the first classification result inference unit 702A initializes a variable i to "1". Next, in step S122, the first classification result inference unit 702A selects the i-th small image 43 and inputs it to the input layer 21 of the first classification learning model 2A, thereby inferring the classification result output from the output layer 23 of the first classification learning model 2A.

Next, in step S126, the variable i is incremented, and in step S128, it is determined whether the variable i exceeds the division number K. The first classification result inference unit 702A then repeats the above steps S122 and S126 until the variable i exceeds the division number K, thereby obtaining classification results for the multiple small image regions 430.

Next, in step S130, the judgment result inference unit 71 inputs the classification results for the multiple small image regions 430 to the input layer 21 of the judgment learning model 2, and infers the judgment result (e.g., whether reprocessing is necessary, whether additional processing is necessary, whether finishing processing is necessary, the processing range, etc.) output from the output layer 23 of the judgment learning model 2.

Next, in step S140, the output processing unit 73 outputs information corresponding to the judgment result inferred by the judgment result inference unit 71 to an output means (e.g., the control device 5, the worker terminal 8, etc.). Then, the series of steps in the machining surface judgment method shown in FIG. 11 is terminated. In the machining surface judgment method, step S100 corresponds to an image acquisition step, steps S100 to S128 correspond to a classification result acquisition step, step S130 corresponds to a judgment result inference step, and step S140 corresponds to an output processing step.

As described above, according to the machining surface judgment device 7 and machining surface judgment method of this embodiment, the classification result acquisition unit 70A divides the judgment image area 420 into small image areas 430 to generate a plurality of small images 43 from the judgment image 42, and inputs each of the plurality of small image areas 430 into the first classification learning model 2A, thereby inferring the classification result for the plurality of small image areas 430. Then, the judgment result inference unit 71 inputs the classification results for the plurality of small image areas 430 into the judgment learning model 2, thereby inferring the state of the machining surface 100 as the judgment result.

Therefore, the classification results by the first classification learning model 2A are inferred in units of small image region 430 by inputting each of the multiple small images 43 into which the judgment image 42 is divided. This makes it easier to collect the learning data required for machine learning and improves the accuracy of the first classification learning model 2A compared to inputting a single judgment image 42 to the first classification learning model 2A. Then, the classification results for the multiple small image regions 430 by the first classification learning model 2A are input to the judgment learning model 2, and the state of the machined surface 100 included in the judgment image 42 is determined. Therefore, the state of the machined surface 100 of the judgment target object 10 can be automatically determined.

Second Embodiment
In the processing system 1 according to the first embodiment, a case where the first classification learning model 2A and the judgment learning model 2 are adopted in the learning phase and inference phase of machine learning is described. In contrast, in the processing system 1 according to the second embodiment, a case where the second classification learning model 2B and the judgment learning model 2 are adopted is described. Note that the basic configuration and operation of the processing system 1 according to the second embodiment are the same as those of the first embodiment, and therefore, the following description will focus on the parts related to the second classification learning model 2B, which is a difference from the first embodiment.

(Machine Learning Device 6)
FIG. 12 is a block diagram showing an example of a machine learning device 6 according to the second embodiment.

The machine learning device 6, like the first embodiment, includes a learning data acquisition unit 60, a learning data storage unit 61, a machine learning unit 62, and a trained model storage unit 63.

The learning data acquisition unit 60 is an interface unit that is connected to various external devices via a communication network and acquires learning data. The learning data storage unit 61 is a database that stores multiple sets of learning data acquired by the learning data acquisition unit 60. The learning data includes second classification learning data for generating the second classification learning model 2B, and judgment learning data similar to that of the first embodiment.

The machine learning unit 62 generates the second classification learning model 2B by inputting multiple sets of second classification learning data to the second classification learning model 2B and having the second classification learning model 2B learn by machine learning the correlation between the input data and the output data contained in the second classification learning data. Also, the machine learning unit 62 generates the judgment learning model 2B using the judgment learning data, as in the first embodiment.

The trained model storage unit 63 is a database that stores the second classification learning model 2B and the judgment learning model 2 generated by the machine learning unit 62.

Figure 13 is a data structure diagram showing an example of second classification learning data.

The second classification learning data includes, as input data, pixel classification results for multiple learning pixel regions 411 obtained from the learning image 41, and, as output data, classification results that classify the state of the machining surface 100 included in the learning image 41 into one of multiple machining states, and is configured by associating these input data and output data.

The pixel classification results for the multiple learning pixel regions 411 as input data are obtained for each learning pixel region 411, which indicates the classification results for the learning pixel regions 411 based on the pixel values within the learning pixel regions 411, for the multiple learning pixel regions 411 that constitute the learning image 41.

The learning pixel area 411 is an area equivalent to one pixel, and the pixel value in the learning pixel area 411 is expressed, for example, by RGB values, grayscale values, brightness values, etc. When four classes, for example, "good", "passable", "poor" and "not to be judged", are adopted as the multiple processing states, the pixel classification result is, for example, a comparison of the pixel value in the learning pixel area 411 with three predetermined thresholds (third threshold < second threshold < first threshold), and when the pixel value is equal to or greater than the first threshold, a classification result of "good" (0) is assigned; when the pixel value is less than the first threshold and equal to or greater than the second threshold, a classification result of "passable" (1); when the pixel value is less than the second threshold and equal to or greater than the third threshold, a classification result of "poor" (2); and when the pixel value is less than the third threshold, a classification result of "not to be judged" (3).

As in the first embodiment, the classification results as output data are, for example, as shown in FIG. 13, “good”, “fair”, “not bad”, “not good ...
The product is indicated as either "poor" or "not applicable."

The learning data acquisition unit 60 can employ various methods for acquiring the second classification learning data and the judgment learning data. For example, similar to the first embodiment, the learning data acquisition unit 60 acquires an image 40 captured by the imaging unit 4 of the judgment target 10 after the processing step is performed by the processing unit 3, divides the image 40 to generate multiple learning images 41, and displays the multiple learning images 41 on the display screen of the worker terminal 8.

The worker visually checks each of the learning images 41 on the display screen, inputs the results (classification results) of classifying the state of the machining surface 100 contained in each of the multiple learning images 41 into multiple machining states (classes), and inputs the results (judgment results) of judging the state of the machining surface 100 contained in the captured image 40 through the worker terminal 8. The learning data acquisition unit 60 then accepts the input operation of the worker and acquires multiple second classification learning data by associating the pixel classification results (input data) for the multiple learning pixel areas 411 acquired from the learning image 41 with the classification results (output data) inputted for the learning image 41. The learning data acquisition unit 60 also acquires judgment learning data by associating the classification results (input data) for the multiple learning image areas 410 of each of the learning images 41 with the judgment results (output data) inputted for the captured image 40.

Therefore, the learning data acquisition unit 60 can acquire a number of second classification learning data corresponding to the number of divisions when one captured image 40 is divided into multiple learning images 41, and can further acquire a desired number of second classification learning data by repeating the above operation. Furthermore, the learning data acquisition unit 60 can acquire judgment learning data in conjunction with acquiring the second classification learning data. Therefore, the second classification learning data and judgment learning data can be easily collected.

Figure 14 is a schematic diagram showing an example of an inference model 20B applied to the second classification learning model 2B.

As a specific method of machine learning, inference model 20B employs a convolutional neural network, similar to inference model 20A shown in FIG. 6. The following describes inference model 20B, focusing on the differences from inference model 20A shown in FIG. 6.

The input layer 21 has neurons whose number corresponds to the number of pixels in the training image 41 as input data, and pixel classification results for multiple training pixel regions 411 are input to each neuron.

The output layer 23 outputs output data including the classification result based on the feature vector output from the fully connected layer 22c. In addition to the classification result, the output data may also include, for example, a score indicating the reliability of the classification result.

The machine learning unit 62 inputs the second classification learning data to the inference model 20B and causes the inference model 20B to perform machine learning of the pixel classification results for the multiple learning pixel regions 411 and the correlation between the classification results. Specifically, the machine learning unit 62 inputs the pixel classification results for the multiple learning pixel regions 411 that constitute the second classification learning data as input data to the input layer 21 of the inference model 20B.

The machine learning unit 62 repeatedly adjusts the weights associated with each synapse (backpropagation) so as to reduce the evaluation value of the error function using an error function that compares the classification result (inference result) indicated by the output data output from the output layer 23 with the classification result (teacher data) constituting the second classification learning data. When the machine learning unit 62 determines that a predetermined learning termination condition, such as the above series of processes being repeated a predetermined number of times or the evaluation value of the error function being smaller than a tolerable value, is satisfied, the machine learning is terminated, and the inference model 20B at that time (all weights associated with each synapse) is stored in the trained model storage unit 63 as the second classification learning model 2B.

(Machine surface determination device 7)
FIG. 15 is a block diagram showing an example of a machining surface determining device 7 according to the second embodiment.

The machining surface determination device 7, like the first embodiment, includes a classification result acquisition unit 70B, a determination result inference unit 71, a trained model storage unit 72, and an output processing unit 73.

The classification result acquisition unit 70B performs a classification result acquisition process (see FIG. 16 described later) for each of the multiple small image regions 430 into which the judgment image region 420 of the judgment image 42 is divided, to acquire the classification result when the state of the machining surface 100 is classified into one of multiple machining states.

The classification result acquisition unit 70B includes an image acquisition unit 700 and a small image generation unit 701 similar to those in the first embodiment, a pixel classification result acquisition unit 703 that acquires pixel classification results indicating classification results for a plurality of pixel regions constituting each of a plurality of small images 43 on a pixel region basis based on pixel values within the pixel region for each of the plurality of pixel regions constituting each of the plurality of small images 43, and a second classification result inference unit 702B that infers classification results for a plurality of small image regions 430 by inputting the pixel classification results for the plurality of pixel regions to a second classification learning model 2B on a small image region 430 basis.

The judgment result inference unit 71 performs judgment result inference processing to infer a judgment result for the judgment image area 420 by inputting the classification results for the multiple small image areas 430 acquired by the classification result acquisition unit 70B into the judgment learning model 2.

The trained model storage unit 72 is a database that stores the trained second classification learning model 2B used in the inference process of the classification result acquisition unit 70B and the trained judgment learning model 2 used in the inference process of the judgment result inference unit 71.

Figure 16 is a functional diagram showing an example of the classification result acquisition process performed by the classification result acquisition unit 70B.

The judgment image area 420 of the judgment image 42 is an area captured by the imaging unit 4, as in the first embodiment, and the small image area 430 of the small image 43 is obtained by dividing the judgment image area 420 of the judgment image 42 into a grid pattern. The small image area 430 of the small image 43 corresponds to the learning image area 410 of the learning image 41, and the multiple pixel areas 431 constituting the small image 43 correspond to the multiple learning pixel areas 411 constituting the learning image 41.

Here, the second classification learning model 2B is a model that has been machine-learned to learn the correlation between the pixel classification results for the multiple learning pixel regions 411 corresponding to the multiple pixel regions 431 and the classification results when the state of the machining surface 100 in the multiple learning pixel regions 411 is classified into one of the multiple machining states based on the pixel classification results. Therefore, the second classification result inference unit 702B functions as a classifier that classifies the state of the machining surface 100 in the small image region 430 into one of the multiple machining states by inputting the pixel classification results for the multiple pixel regions 431 that constitute each of the multiple small images 43 to the second classification learning model 2B in units of small image region 430.

(Method of determining machined surface)
FIG. 17 is a flowchart showing an example of a machining surface determination method by the machining surface determination device 7 according to the second embodiment.

First, in step S100, the image acquisition unit 700 of the classification result acquisition unit 70B acquires the judgment image 42.

Next, in step S110, the small image generating unit 701 performs preprocessing on the judgment image 42 by dividing the judgment image area 420 of the judgment image 42 into multiple small image areas 430 to generate multiple small images 43 from the judgment image 42.

Then, in step S112, the pixel classification result acquisition unit 703 acquires pixel classification results for each of the multiple pixel regions 431 constituting each of the multiple small images 43 based on the pixel values within the pixel regions 431 on a pixel region 431 basis.

Next, in steps S120 to S128, the second classification result inference unit 702B executes a loop process by incrementing the variable i from "1" to "K" with K being the number of divisions of the multiple small images 43 and assigning serial numbers (1≦n≦K) to each of the multiple small images 43.

Specifically, in step S120, the second classification result inference unit 702B initializes a variable i to "1". Next, in step S124, the second classification result inference unit 702B selects the i-th small image 43, and inputs the pixel classification results for the multiple pixel regions 431 that make up the small image 43 to the input layer 21 of the second classification learning model 2B, thereby inferring the classification result output from the output layer 23 of the second classification learning model 2B.

Next, in step S126, the variable i is incremented, and in step S128, it is determined whether the variable i exceeds the division number K. The second classification result inference unit 702B then repeats the above steps S124 and S126 until the variable i exceeds the division number K, thereby obtaining classification results for the multiple small image regions 430.

Next, in step S130, the judgment result inference unit 71 inputs the classification results for the multiple small image regions 430 to the input layer 21 of the judgment learning model 2, and infers the judgment result (e.g., whether reprocessing is necessary, whether additional processing is necessary, whether finishing processing is necessary, the processing range, etc.) output from the output layer 23 of the judgment learning model 2.

Next, in step S140, the output processing unit 73 outputs information corresponding to the judgment result inferred by the judgment result inference unit 71 to an output means (e.g., the control device 5, the worker terminal 8, etc.). Then, the series of steps in the machining surface judgment method shown in FIG. 17 is terminated. In the machining surface judgment method, step S100 corresponds to an image acquisition step, steps S100 to S128 correspond to a classification result acquisition step, step S130 corresponds to a judgment result inference step, and step S140 corresponds to an output processing step.

As described above, according to the machining surface judgment device 7 and machining surface judgment method of this embodiment, the classification result acquisition unit 70B generates multiple small images 43 from the judgment image 42 by dividing the judgment image area 420 into small image areas 430, and inputs the pixel classification results for the multiple pixel areas 431 that make up each of the multiple small images 43 to the second classification learning model 2B, thereby inferring the classification results for the multiple small image areas 430. Then, the judgment result inference unit 71 inputs the classification results for the multiple small image areas 430 to the judgment learning model 2, thereby inferring the state of the machining surface 100 as a judgment result.

Therefore, the classification results by the second classification learning model 2B are inferred in units of small image region 430 by inputting each of the multiple small images 43 into which the judgment image 42 is divided. This makes it easier to collect the learning data required for machine learning and improves the accuracy of the second classification learning model 2B compared to inputting a single judgment image 42 into the second classification learning model 2B. Then, the classification results for the multiple small image regions 430 by the second classification learning model 2B are input into the judgment learning model 2, and the state of the machined surface 100 contained in the judgment image 42 is determined. Therefore, the state of the machined surface 100 of the judgment target object 10 can be automatically determined.

Other Embodiments
The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and scope of the present invention. All such modifications are included in the technical concept of the present invention.

For example, in the above embodiment, the image area for judgment 420 is set to include a portion of one blade of the impeller of the object to be judged 10 as the processed surface 100 to be judged. In contrast, the image area for judgment 420 may be set to include multiple blades of the impeller as multiple processed surfaces 100 to be judged by expanding it to the entire impeller. In other words, when the object to be judged 10 has multiple processed surfaces 100 each processed through different processing steps by the processing unit 3, the image area for judgment 420 may be set to include multiple processed surfaces 100.

In this case, the classification result acquisition unit 70B acquires a judgment image 42 in which multiple processing surfaces 100 are captured, and sets a judgment image area 420 for each processing surface so that the judgment image 42 is separated at the boundaries of the multiple processing surfaces 100. The boundaries of the processing surfaces 100 may be set in advance, or may be set by image processing on the judgment image 42. Then, the classification result acquisition unit 70B acquires classification results for multiple small image areas 430 into which the judgment image area 420 for each processing surface is respectively divided. Next, the judgment result inference unit 71 infers a judgment result for the judgment image 42 for each processing surface by inputting the classification results for the multiple small image areas 430 to the judgment learning model 2 for each processing surface.

In the above embodiment, a case has been described in which CNN (see FIGS. 6 and 7) is adopted as a specific method of machine learning by the machine learning unit 62, but the machine learning unit 62 may adopt any other machine learning method. Examples of other machine learning methods include tree types such as decision trees and regression trees, ensemble learning such as bagging and boosting, neural network types (including deep learning) such as recurrent neural networks and convolutional neural networks, clustering types such as hierarchical clustering, non-hierarchical clustering, k-nearest neighbors, and k-means, multivariate analyses such as principal component analysis, factor analysis, and logistic regression, and support vector machines.

(machined surface judgment program)
The present invention can be provided in the form of a program (machining surface determination program) 230 that causes the computer 200 shown in Fig. 2 to function as each unit included in the machining surface determination device 7 according to the above embodiment. The present invention can also be provided in the form of a program (machining surface determination program) 230 that causes the computer 200 shown in Fig. 2 to execute each step included in the machining surface determination method according to the above embodiment.

(Inference device, inference method, and inference program)
The present invention can be provided not only in the form of the machining surface judgment device 7 (machining surface judgment method or machining surface judgment program) according to the above embodiment, but also in the form of an inference device (inference method or inference program) used to judge the state of the machining surface 100. In that case, the inference device (inference method or inference program) can include a memory and a processor, and the processor executes a series of processes. The series of processes includes a classification result acquisition process (classification result acquisition step) for acquiring a classification result when the state of the machining surface 100 is classified into one of a plurality of machining states for a plurality of small image regions 430 obtained by dividing the judgment image region 420 of the judgment image 42, for each small image region 430, and a judgment result inference process (judgment result inference step) for inferring the state of the machining surface 100 included in the judgment image 42 as a judgment result for the judgment image 42 when the classification results for the plurality of small image regions 430 are acquired in the classification result acquisition process.

By providing it in the form of an inference device (inference method or inference program), it can be easily applied to various devices compared to the case where the machining surface judgment device 7 is implemented. It can be naturally understood by those skilled in the art that when the inference device (inference method or inference program) infers the state of the machining surface 100, the inference method implemented by the judgment result inference unit 71 of the machining surface judgment device 7 may be applied using the trained judgment learning model 2 generated by the machine learning device 6 according to the above embodiment.

1... Processing system, 2... Judgment learning model,
2A: First classification learning model, 2B: Second classification learning model, 3: Processing unit, 4: Imaging unit, 5: Control device, 6: Machine learning device,
7...machined surface judgment device, 8...operator terminal, 10...object to be judged,
20, 20A, 20B... inference model, 21... input layer, 22... intermediate layer,
22a...convolutional layer, 22b...pooling layer, 22c...fully connected layer, 23...output layer,
40: captured image, 41: learning image, 42: judgment image, 43: small image,
50: control panel, 51: operation display panel, 60: learning data acquisition unit, 61: learning data storage unit, 62: machine learning unit, 63: model storage unit, 70A, 70B: classification result acquisition unit, 71: judgment result inference unit, 72: model storage unit, 73: output processing unit, 100: machining surface, 110: background, 200: computer,
400: captured image area, 410: learning image area, 411: learning pixel area, 420: judgment image area, 430: small image area, 431: pixel area, 700: image acquisition unit, 701: small image generation unit,
702A: first classification result inference unit, 702B: second classification result inference unit, 703: pixel classification result acquisition unit

Claims (12)

A machined surface judgment device that judges a state of a machined surface of an object to be judged based on an image for judgment obtained by capturing an image of the machined surface, comprising:
a classification result acquisition unit that acquires, for each of a plurality of small image regions obtained by dividing a judgment image region of the judgment image, a classification result when classifying the state of the processed surface into one of a plurality of processing states for each of the small image regions;
a judgment result inference unit that infers the judgment result for the judgment image by inputting the classification results for the multiple small image regions acquired by the classification result acquisition unit into a judgment learning model that has been machine-learned to learn a correlation between the classification results for multiple learning image regions corresponding to the multiple small image regions and a judgment result when the state of the processing surface in the multiple learning image regions is judged based on the classification results,
Machined surface judgment device. The classification result acquisition unit
an image acquisition unit that acquires the determination image having the determination image area;
a small image generating unit that generates a plurality of the small images from the determination image by dividing the determination image region into a plurality of the small image regions;
a first classification result inference unit that infers the classification result for the plurality of small image regions by inputting the plurality of small images to a first classification learning model that has been machine-learned to learn a correlation between a learning image having the learning image region and the classification result when the state of the processing surface included in the learning image is classified into one of the plurality of processing states, in units of the small image region;
The machined surface determination device according to claim 1 . The classification result acquisition unit
an image acquisition unit that acquires the determination image having the determination image area;
a small image generating unit that generates a plurality of the small images from the determination image by dividing the determination image region into a plurality of the small image regions;
a pixel classification result acquisition unit that acquires, for each of a plurality of pixel regions constituting each of the plurality of small images, a pixel classification result indicating the classification result for the pixel region based on pixel values within the pixel region, for the pixel region;
a second classification result inference unit that infers the classification results for the plurality of small image regions by inputting the pixel classification results for the plurality of pixel regions into a second classification learning model that has been machine-learned to determine a correlation between the pixel classification results for a plurality of learning pixel regions corresponding to the plurality of pixel regions and the classification results when the state of the processed surface in the plurality of learning pixel regions is classified into one of the plurality of processing states based on the pixel classification results, on a small image region basis;
The machined surface determination device according to claim 1 . The classification result acquisition unit
The classification result is obtained for each of a plurality of small image areas obtained by dividing the determination image area for each of the processing surfaces included in the determination image in which the plurality of processing surfaces are captured, for each of the small image areas;
The judgment result inference unit,
the classification results for the plurality of small image regions are input to the judgment learning model for each of the processing surfaces, thereby inferring the judgment result for the judgment image for each of the processing surfaces;
The machined surface determining device according to any one of claims 1 to 3. The classification result acquisition unit
For each of the plurality of small image regions, the state of the processed surface in the small image region is classified into one of the plurality of processed states including at least good and bad, or the classification result is acquired on a small image region basis when the state of the processed surface in the small image region is classified as not to be determined due to the presence of an edge of the processed surface or a background other than the processed surface in the small image region.
The machined surface determining device according to any one of claims 1 to 4. The judgment result inference unit,
As a result of the determination,
Whether or not reprocessing is necessary by repeating the same processing step as that used to process the processed surface;
Whether or not a separate processing step different from that used to process the processed surface is required;
Whether or not a finishing process is required for the machined surface to be finished by an operator; and
A processing range of the processing surface to be subjected to the re-processing, the separate processing, or the finish processing;
Infer at least one of
The machined surface determining device according to any one of claims 1 to 5. The processed surface is a surface of the object to be determined when the object to be determined is processed by a polishing process, a grinding process, a cutting process, or a casting process.
The machined surface determining device according to any one of claims 1 to 6. The object to be determined is a fluid machine or a fluid part constituting the fluid machine.
The machined surface determining device according to any one of claims 1 to 7. A method for making a computer function as the machined surface judgment device according to any one of claims 1 to 8,
Machining surface judgment program. A method for judging a state of a processed surface of an object to be judged based on an image for judgment obtained by capturing an image of the processed surface, comprising the steps of:
a classification result acquisition step of acquiring a classification result for each of a plurality of small image regions obtained by dividing a judgment image region of the judgment image, the classification result being obtained when the state of the processed surface is classified into one of a plurality of processing states, for each of the small image regions;
a judgment result inference step of inferring the judgment result for the judgment image by inputting the classification results for the multiple small image regions acquired by the classification result acquisition step into a judgment learning model that has been machine-learned to learn a correlation between the classification results for multiple learning image regions corresponding to the multiple small image regions and a judgment result when the state of the processing surface in the multiple learning image regions is judged based on the classification results.
Method for determining machined surface. The machined surface judgment device according to any one of claims 1 to 8,
A processing unit that processes the object to be determined;
An imaging unit that images a processed surface of the object to be determined;
A control unit that controls the processing surface determination device, the processing unit, and the imaging unit,
Processing system. A machine learning device that generates a judgment learning model used in a processed surface judgment device that judges a state of a processed surface of an object to be judged based on a judgment image obtained by capturing the processed surface, the machine learning device comprising:
a learning data storage unit that stores multiple sets of learning data each including, as input data, a classification result when the state of the processing surface is classified into one of multiple processing states for each of multiple learning image areas corresponding to multiple small image areas obtained by dividing the judgment image area of the judgment image, and a judgment result when the state of the processing surface in each of the multiple learning image areas is judged based on the classification result, as output data;
a machine learning unit that learns the judgment learning model by inputting a plurality of sets of the learning data, the judgment learning model inferring a correlation between the input data and the output data;
A trained model storage unit that stores the judgment learning model trained by the machine learning unit.
Machine learning device.

Images