JP7604444B2 - PROGRAM, INFORMATION PROCESSING METHOD AND INFORMATION PROCESSING APPARATUS - Google Patents

Description

The present invention relates to a program, an information processing method, and an information processing device.

In recent years, there has been an increasing need to match and recognize matching objects, such as products or parts (components) assembled from products, for quality control and distribution management. Patent Document 1 discloses a matching system that matches a matching object with a registered object based on the statistical values of the luminance values of pixels in a registered image that correspond to pixel positions estimated to cause specular reflection in a matching image in a specified matching area of the matching object.

International Publication No. 2016/136900

However, the invention of Patent Document 1 may not be able to automatically extract the matching area (area image) of the matching object.

In one aspect, the objective is to provide a program etc. capable of extracting appropriate matching regions from a matching object.

In one aspect, the program causes a computer to execute a process of acquiring a part image of a part including multiple feature objects, recognizing each object in the acquired part image, extracting multiple area images of a predetermined size at a predetermined offset position from each recognized object, acquiring part-specific information corresponding to each extracted area image, determining the number of matches in the part-specific information corresponding to each area image in each of the area images, and authenticating the part based on the number of matches in the part-specific information and a predetermined threshold value.

In one aspect, it is possible to extract appropriate matching regions for matching objects.

FIG. 1 is an explanatory diagram showing an overview of a traceability system. FIG. 2 is a block diagram showing an example of the configuration of an image processing server. FIG. 2 is a block diagram showing a configuration example of a parts server; FIG. 2 is an explanatory diagram illustrating an example of a record layout of a parts specific information DB. FIG. 2 is a block diagram showing an example of the configuration of a DB server. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a product management DB. FIG. 13 is an explanatory diagram illustrating an example of a record layout of a parts detail DB. FIG. 2 is a block diagram showing an example of the configuration of a factory server. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a product DB. FIG. 11 is an explanatory diagram for explaining an operation of outputting part specific information. FIG. 2 is an explanatory diagram for explaining an object recognition model. 13 is a flowchart showing a processing procedure for outputting part specific information. 13 is a flowchart showing a processing procedure for outputting part specific information. FIG. 11 is an explanatory diagram for explaining an operation of outputting a product ID in association with a part ID. 10 is a flowchart showing a processing procedure for outputting a product ID in association with a part ID. 10 is a flowchart showing a processing procedure for outputting a product ID in association with a part ID. FIG. 11 is an explanatory diagram showing an overview of a traceability system according to a third embodiment. 11 is a block diagram showing a configuration example of a deletion server. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a parts management DB. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a production plan DB. 13 is a flowchart showing a processing procedure for deleting a registered part. FIG. 1 is an explanatory diagram for improving object recognition. FIG. 13 is an explanatory diagram showing how part images are captured by two photographing devices. 10 is a flowchart showing a processing procedure for acquiring a part image. FIG. 11 is an explanatory diagram for explaining correction of an offset position. 10 is a flowchart showing a processing procedure for correcting a part image in which area distortion has occurred. FIG. 23 is a block diagram showing an example of the configuration of an image processing server according to a seventh embodiment. FIG. 23 is a block diagram showing an example of the configuration of a DB server according to a seventh embodiment. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a parts damage information DB. 10 is a flowchart showing a processing procedure for outputting both part-specific information and information regarding damage to a part. FIG. 23 is an explanatory diagram showing an example of a record layout of a parts specific information DB according to the eighth embodiment. FIG. 23 is an explanatory diagram showing an example of a record layout of a part detail DB according to the eighth embodiment. 10 is a flowchart showing a processing procedure for outputting part storage information in association with a part ID. 13 is a flowchart showing a processing procedure for detecting a retained part. FIG. 13 is a block diagram showing an example of the configuration of an image processing server according to a ninth embodiment. FIG. 13 is a block diagram showing an example of the configuration of a DB server according to a ninth embodiment. FIG. 13 is an explanatory diagram illustrating an example of a record layout of a matching result DB. FIG. 11 is an explanatory diagram for explaining a wear identification model. 11 is a flowchart showing a processing procedure when checking parts. 11 is a flowchart showing a processing procedure when performing a part registration process. 10 is a flowchart showing a processing procedure when performing a part matching process. 1 is an explanatory diagram showing an example of a cylindrical object having a plurality of objects distributed on an end surface of the cylindrical object;

The present invention will be described in detail below with reference to drawings showing an embodiment of the invention.

(Embodiment 1)
The first embodiment relates to a form in which an object in a part image is recognized, an area image is extracted at a predetermined offset position from the recognized object, and part-specific information corresponding to the extracted area image is output. The part is a part (part or component) for constituting the entire product such as an automobile, a personal computer, a home appliance, a smartphone, or a machine tool. For example, if the product is an automobile, the part is a door panel, a mirror, a tire, or the like. The object is a feature point on the part, such as a press punched shape, a drill hole, an edge shape, a paint, a seal, a forced dent hole, a laser marking, a three-dimensional character, or the like.

The part-specific information includes the part's Fingerprint of Things, manufacturing information, material information, and attachment information. The Fingerprint of Things is a fine pattern (e.g., bumps or patterns) on the surface of a part (object). The Fingerprint of Things occurs naturally during the manufacturing stage, and the Fingerprint of Things itself does not change even if the part is later scratched, etc., so each part can be distinguished using Fingerprint of Things recognition technology (image recognition technology).

Figure 1 is an explanatory diagram showing an overview of a traceability system. The system of this embodiment includes an information processing device 1, a photographing device 2, an information processing device 3, a database device 4, and an information processing device 5, and each device transmits and receives information via a network N such as the Internet.

The information processing device 1 is an information processing device that processes, stores, and transmits/receives various types of information. The information processing device 1 is, for example, a server device, an edge computer, a personal computer, or a general-purpose tablet PC (personal computer). In this embodiment, the information processing device 1 is a server device, and for the sake of simplicity, will be referred to as an image processing server 1 below.

The photographing device 2 photographs parts including an object to generate an image. The photographing device 2 of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits and receives photographed images to and from the image processing server 1, etc., via the network N. Note that instead of the photographing device 2, an information processing terminal capable of photographing may also be used. The information processing terminal is, for example, an information processing device such as a smartphone, a mobile phone, a tablet, or a personal computer terminal.

The information processing device 3 is an information processing device that performs registration, authentication, etc. of parts. The information processing device 3 is, for example, a server device, a personal computer, or a general-purpose tablet PC. In this embodiment, the information processing device 3 is a server device, and for the sake of brevity, will be referred to as a parts server 3 below.

The database (DB) device 4 is a storage device that stores part-specific information and information about products that use the parts. In the following description of the present embodiment, for the sake of simplicity, the database device 4 will be referred to as the DB server 4.

The information processing device 5 is an information processing device that manages manufacturing information and material information of each part for each product in a factory or the like. The information processing device 5 is, for example, a server device, a personal computer, or a general-purpose tablet PC. In this embodiment, the information processing device 5 is a server device, and for simplicity, will be referred to as the factory server 5 below.

In the present embodiment, an example including one information processing device 1 and one photographing device 2 has been described, but the present invention is not limited to this, and may include multiple information processing devices 1 and multiple photographing devices 2. For example, when using a combination of one information processing device 1 (for registration) and one photographing device 2 (for registration), and a combination of one information processing device 1 (for authentication) and one photographing device 2 (for authentication), the information processing device 1 (for registration) performs a process of registering the part image photographed by the photographing device 2 (for registration) in the parts server 3. The information processing device 1 (for authentication) performs a process of authenticating the part image photographed by the photographing device 2 (for authentication) by comparing it with the part images already registered in the parts server 3.

The image processing server 1 according to this embodiment acquires a part image including an object that is a feature point from the photographing device 2, and recognizes the object in the acquired part image. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object, and acquires part-specific information corresponding to the extracted area image. The image processing server 1 assigns a part ID to the acquired part-specific information, and outputs the assigned part ID and part-specific information to the part server 3 and DB server 4.

Figure 2 is a block diagram showing an example of the configuration of the image processing server 1. The image processing server 1 includes a control unit 11, a storage unit 12, a communication unit 13, an input unit 14, a display unit 15, a reading unit 16, and a large-capacity storage unit 17. Each component is connected by a bus B.

The control unit 11 includes an arithmetic processing device such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), and a GPU (Graphics Processing Unit), and performs various information processing, control processing, and the like related to the image processing server 1 by reading and executing a control program 1P stored in the storage unit 12. Note that while the control unit 11 is described in FIG. 2 as being a single processor, it may also be a multi-processor.

The storage unit 12 includes memory elements such as RAM (Random Access Memory) and ROM (Read Only Memory), and stores the control program 1P or data required for the control unit 11 to execute processing. The storage unit 12 also temporarily stores data required for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information between the imaging device 2, parts server 3, DB server 4, and factory server 5 via the network N.

The input unit 14 is an input device such as a mouse, keyboard, touch panel, or button, and outputs received operation information to the control unit 11. The display unit 15 is a liquid crystal display or an organic EL (electroluminescence) display, etc., and displays various information according to instructions from the control unit 11.

The reading unit 16 reads the portable storage medium 1a, which may be a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control unit 11 may read the control program 1P from the portable storage medium 1a via the reading unit 16 and store it in the mass storage unit 17. The control unit 11 may also download the control program 1P from another computer via a network N or the like and store it in the mass storage unit 17. The control unit 11 may also read the control program 1P from the semiconductor memory 1b.

The mass storage unit 17 includes a recording medium such as a hard disk drive (HDD) or a solid state drive (SSD). The mass storage unit 17 includes an object recognition model 171. The object recognition model 171 is a recognizer that recognizes an object in a part image based on the part image, and is a trained model generated by machine learning.

In this embodiment, the memory unit 12 and the large-capacity memory unit 17 may be configured as an integrated memory device. The large-capacity memory unit 17 may also be configured from multiple memory devices. Furthermore, the large-capacity memory unit 17 may be an external memory device connected to the image processing server 1.

In this embodiment, the image processing server 1 is described as a single information processing device, but the processing may be distributed across multiple devices, or the image processing server 1 may be configured as a virtual machine.

Figure 3 is a block diagram showing an example of the configuration of the parts server 3. The parts server 3 includes a control unit 31, a storage unit 32, a communication unit 33, an input unit 34, a display unit 35, a reading unit 36, and a large-capacity storage unit 37. Each component is connected by a bus B.

The control unit 31 includes a processor such as a CPU, MPU, or GPU, and performs various information processing, control processing, and the like related to the parts server 3 by reading and executing the control program 3P stored in the storage unit 32. Note that while the control unit 31 is described in FIG. 3 as being a single processor, it may be a multiprocessor.

The storage unit 32 includes memory elements such as RAM and ROM, and stores the control program 3P or data required for the control unit 31 to execute processing. The storage unit 32 also temporarily stores data required for the control unit 31 to execute arithmetic processing. The communication unit 33 is a communication module for performing communication-related processing, and transmits and receives information to and from the image processing server 1 via the network N.

The input unit 34 is an input device such as a mouse, keyboard, touch panel, or button, and outputs received operation information to the control unit 31. The display unit 35 is a liquid crystal display or an organic EL display, etc., and displays various information according to instructions from the control unit 31.

The reading unit 36 reads the portable storage medium 3a, which may include a CD-ROM or DVD-ROM. The control unit 31 may read the control program 3P from the portable storage medium 3a via the reading unit 36 and store it in the mass storage unit 37. The control unit 31 may also download the control program 3P from another computer via the network N or the like and store it in the mass storage unit 37. The control unit 31 may also read the control program 3P from the semiconductor memory 3b.

The large-capacity storage unit 37 includes a recording medium such as a HDD or SSD. The large-capacity storage unit 37 includes a part-specific information DB 371. The part-specific information DB 371 stores part-specific information in association with a part ID.

In this embodiment, the memory unit 32 and the large-capacity memory unit 37 may be configured as an integrated memory device. The large-capacity memory unit 37 may also be configured with multiple memory devices. Furthermore, the large-capacity memory unit 37 may be an external memory device connected to the parts server 3.

In this embodiment, the parts server 3 is described as a single information processing device, but the processing may be distributed across multiple devices, or the parts server 3 may be configured as a virtual machine.

Figure 4 is an explanatory diagram showing an example of a record layout of parts-specific information DB371. Parts-specific information DB371 includes a part ID column, a part image column, and a registration date and time column. The part ID column stores a uniquely specified part ID to identify each part. The part image column stores an area image of the whole or a part. The registration date and time column stores date and time information when the part was registered. Note that manufacturing information, material information, installation information, etc. of a part belonging to the parts-specific information may also be stored in parts-specific information DB371.

Figure 5 is a block diagram showing an example of the configuration of the DB server 4. The DB server 4 includes a control unit 41, a storage unit 42, a communication unit 43, a reading unit 44, and a large-capacity storage unit 45. Each component is connected by a bus B.

The control unit 41 includes a processor such as a CPU, MPU, or GPU, and performs data reading, data storage, and the like by reading and executing a control program 4P stored in the storage unit 42. Note that while the control unit 41 is described in FIG. 5 as being a single processor, it may be a multiprocessor.

The storage unit 42 includes memory elements such as RAM and ROM, and stores the control program 4P or data required for the control unit 41 to execute processing. The storage unit 42 also temporarily stores data required for the control unit 41 to execute arithmetic processing. The communication unit 43 is a communication module for performing communication-related processing, and transmits and receives information to and from the image processing server 1 via the network N.

The reading unit 44 reads the portable storage medium 4a, which may include a CD-ROM or DVD-ROM. The control unit 41 may read the control program 4P from the portable storage medium 4a via the reading unit 44 and store it in the mass storage unit 45. The control unit 41 may also download the control program 4P from another computer via the network N or the like and store it in the mass storage unit 45. The control unit 41 may also read the control program 4P from the semiconductor memory 4b.

The mass storage unit 45 includes a recording medium such as an HDD or SSD. The mass storage unit 45 includes a product management DB 451 and a parts details DB 452. The product management DB 451 stores information such as the correspondence between a product ID and the part IDs that assemble the product. The parts details DB 452 stores detailed information about parts in association with the part IDs.

In this embodiment, the memory unit 42 and the large-capacity memory unit 45 may be configured as an integrated memory device. The large-capacity memory unit 45 may also be configured from multiple memory devices. Furthermore, the large-capacity memory unit 45 may be an external memory device connected to the DB server 4.

In this embodiment, the DB server 4 is described as being a single database device, but processing may be distributed across multiple devices, or the DB server 4 may be configured as a virtual machine.

Figure 6 is an explanatory diagram showing an example of a record layout of product management DB 451. Product management DB 451 includes a product ID column, a type column, a product name column, and a part ID column. The product ID column stores a unique product ID to identify each product. The type column stores product type information. The product name column stores the name of the product. The part ID column stores part IDs that identify the parts that are assembled into the product.

Figure 7 is an explanatory diagram showing an example of a record layout of part details DB 452. Part details DB 452 includes a part ID column, a manufacturing information column, a material information column, and an attachment information column. The part ID column stores a unique part ID to identify each part.

The manufacturing information column includes a processing machine ID column, a processing machine manufacturer name column, a location address column, and a processing date and time column. The processing machine ID column stores a uniquely specified ID of the processing machine to identify each processing machine. The processing machine manufacturer name column stores the name of the processing machine manufacturer. The location address column stores the address or name of the production line where the part was processed. The processing date and time column stores information on the date and time when the part was processed.

The material information column includes a material ID column, a material manufacturer ID column, a lot number column, and a delivery date and time column. The material ID column stores a unique material ID to identify each material. The material manufacturer ID column stores a unique material manufacturer ID to identify each material manufacturer. The lot number column stores the material lot number assigned to each production unit when the material is manufactured. The delivery date and time column stores information on the date and time when the material was delivered.

The installation information column includes an installation robot ID column, an installation manufacturer name column, and a location address column. The installation robot ID column stores a unique ID of the installation robot to identify each installation robot. The installation manufacturer name column stores the name of the manufacturer of the installation robot used when installing the part on the product. The location address column stores the address or name of the production line where the part was installed.

Figure 8 is a block diagram showing an example of the configuration of the factory server 5. The factory server 5 includes a control unit 51, a storage unit 52, a communication unit 53, an input unit 54, a display unit 55, a reading unit 56, and a large-capacity storage unit 57. Each component is connected by a bus B.

The control unit 51 includes a processor such as a CPU, MPU, or GPU, and performs various information processing, control processing, and the like related to the factory server 5 by reading and executing the control program 5P stored in the memory unit 52. Note that while the control unit 51 is described in FIG. 8 as being a single processor, it may be a multiprocessor.

The storage unit 52 includes memory elements such as RAM and ROM, and stores the control program 5P or data required for the control unit 51 to execute processing. The storage unit 52 also temporarily stores data required for the control unit 51 to execute arithmetic processing. The communication unit 53 is a communication module for performing communication-related processing, and transmits and receives information to and from the image processing server 1 via the network N.

The input unit 54 is an input device such as a mouse, keyboard, touch panel, or button, and outputs received operation information to the control unit 51. The display unit 55 is a liquid crystal display or an organic EL display, and displays various information according to instructions from the control unit 51.

The reading unit 56 reads the portable storage medium 5a, which includes a CD-ROM or DVD-ROM. The control unit 51 may read the control program 5P from the portable storage medium 5a via the reading unit 56 and store it in the mass storage unit 57. The control unit 51 may also download the control program 5P from another computer via the network N or the like and store it in the mass storage unit 57. The control unit 51 may also read the control program 5P from the semiconductor memory 5b.

The mass storage unit 57 includes a recording medium such as an HDD or SSD. The mass storage unit 57 includes a product DB 571. The product DB 571 stores information about the product and information about the parts that assemble the product.

In this embodiment, the memory unit 52 and the large-capacity memory unit 57 may be configured as an integrated memory device. The large-capacity memory unit 57 may also be configured from multiple memory devices. Furthermore, the large-capacity memory unit 57 may be an external memory device connected to the factory server 5.

In this embodiment, the factory server 5 is described as a single information processing device, but processing may be distributed across multiple devices, or the factory server 5 may be configured as a virtual machine.

Figure 9 is an explanatory diagram showing an example of a record layout of product DB 571. Product DB 571 includes a product ID column, a product type column, a part type column, a part feature column, a manufacturing information column, a material information column, and an attachment information column.

The product ID column stores a unique product ID to identify each product. The product type column includes an ID column and a name column. The ID column stores the ID of the product type. The name column stores the name of the product type. The part type column includes an ID2 column and a name2 column. The ID2 column stores the ID of the part type. The name2 column stores the name of the part type. Note that the layout of the product type column or part type column is not limited to the above. The product type column or part type column may store other information in addition to the ID and name.

The part feature column includes an offset position column, an area image size column, and an area image column. The offset position column stores distance information from the object. The area image size column stores the size of the area image for obtaining part-specific information. The area image column stores an image (a part of the part image) that shows the fingerprint (feature) of the part.

The manufacturing information column, material information column, and installation information column are similar to those in part details DB452, so explanations will be omitted.

Note that the storage format of each DB described above is just an example, and other storage formats are also possible as long as the relationships between data are maintained.

Figure 10 is an explanatory diagram explaining the operation of outputting part-specific information. The control unit 11 of the image processing server 1 acquires a part image including an object that is a feature point from the photographing device 2 via the communication unit 13. Note that, in this embodiment, an example in which the part image is transmitted from the photographing device 2 has been described, but this is not limited to this. For example, the part image photographed by the photographing device 2 may be transferred to the image processing server 1 via a memory card.

The control unit 11 inputs the acquired part image to a trained object recognition model 171 that outputs a recognition result of recognizing an object in the part image when the acquired part image is input, and outputs a recognition result of recognizing an object in the part image. The object recognition process will be described later.

When the control unit 11 recognizes an object based on the recognition result, it extracts an area image of a predetermined size at a predetermined offset position from the recognized object. The offset position is the distance away from the object, for example, 20 pixels to the right of the object. The predetermined size is the size of the area image extracted from the part image in pixel (picture element) units, for example, 120 x 120 pixels. The predetermined offset position and predetermined size are stored in advance in the memory unit 12 or the large-capacity memory unit 17 for each type of object.

The control unit 11 acquires part-specific information corresponding to the extracted area image from the factory server 5. Specifically, the control unit 11 transmits the part type ID and the offset position by the object together with the extracted area image to the factory server 5 via the communication unit 13. The control unit 51 of the factory server 5 receives the part type ID, offset position, and area image transmitted from the image processing server 1 via the communication unit 53. The control unit 51 acquires the corresponding part-specific information from the product DB 571 of the mass storage unit 57 based on the received part type ID, offset position, and area image. The part-specific information includes manufacturing information, material information, and attachment information for the part to the product.

Specifically, the control unit 51 acquires the size of the area image from the received area image. The control unit 51 extracts an area image stored in the area image sequence of the product DB 571 based on the acquired size of the area image and the received part type ID and offset position. The control unit 51 performs a matching process between the received area image and the area image extracted from the product DB 571. If the control unit 51 determines that the degree of match is equal to or greater than a predetermined threshold, it acquires the corresponding part-specific information from the product DB 571. Note that the control unit 51 may compare the area image captured directly with the area image stored in the product DB 571.

The control unit 51 transmits the acquired part-specific information, including manufacturing information, material information, and product attachment information, to the image processing server 1 via the communication unit 53. The control unit 11 of the image processing server 1 receives the part-specific information transmitted from the factory server 5 via the communication unit 13, and assigns a part ID to the received part-specific information. Note that, in this embodiment, an example has been described in which the control unit 11 assigns a part ID, but this is not limiting. For example, the control unit 11 may obtain and use a part ID for management purposes at the factory server 5.

The control unit 11 registers the part corresponding to the assigned part ID in the parts server 3. Specifically, the control unit 11 transmits (outputs) the extracted area image to the parts server 3 via the communication unit 13, in association with the assigned part ID. The control unit 31 of the parts server 3 receives the part ID and area image transmitted from the image processing server 1 via the communication unit 33. The control unit 31 stores the area image in the parts unique information DB 371 of the mass storage unit 37, in association with the received part ID. Specifically, the control unit 31 stores the part ID, area image, and registration date and time in the parts unique information DB 371 as one record.

The control unit 11 outputs the part-specific information (part detail information) corresponding to the assigned part ID to the DB server 4. Specifically, the control unit 11 transmits (outputs) the received part-specific information to the DB server 4 via the communication unit 13 in association with the assigned part ID. The control unit 41 of the DB server 4 receives the part ID and part-specific information transmitted from the image processing server 1 via the communication unit 43. The control unit 41 stores the part-specific information in the part detail DB 452 of the mass storage unit 45 in association with the received part ID. Specifically, the control unit 41 stores the part ID, manufacturing information (machine ID, machine manufacturer name, location address, processing date and time), material information (material ID, material manufacturer ID, lot number, delivery date and time), and installation information (installation robot ID, installation manufacturer name, location address) as one record in the part detail DB 452.

Next, the object recognition process using the object recognition model 171 will be described. FIG. 11 is an explanatory diagram of the object recognition model 171. The object recognition model 171 is used as a program module that is part of artificial intelligence software. The object recognition model 171 is a recognizer that has constructed a neural network that receives a part image as input and outputs a recognition result including the position and type of the object in the part image.

The object recognition model 171 of this embodiment performs estimation using an RCNN (Regions with Convolutional Neural Network) as an example. The object recognition model 171 includes a region candidate extraction unit 71a and a classification unit 71b. The classification unit 71b includes a neural network (not shown). The neural network includes a convolution layer, a pooling layer, and a fully connected layer.

The control unit 11 of the image processing server 1 performs learning using a combination of part images and training data in which the positions (coordinate ranges of object areas) and types (e.g., triangular paint marks, star-shaped stickers, etc.) of objects in the part images are associated with each other. The training data is data in which the objects in the part images are labeled.

A part image is input to the trained object recognition model 171. The area candidate extraction unit 71a extracts area candidates of various sizes from the part image. The classification unit 71b calculates the features of the extracted area candidates. Based on the calculated features, the classification unit 71b classifies whether the subject shown in the area candidate is an object or not, and if the subject is an object, the type of object. The object recognition model 171 repeats the extraction and classification of area candidates.

The object recognition model 171 outputs the probability that an object is included, the object's position (coordinate range of the area), and the object's type for area candidates that are classified as having an object included with a probability higher than a predetermined threshold (e.g., 85%). In the example shown in FIG. 11, an area that has an object included with a probability of 90% (object position) and an object type are detected.

Note that while FIG. 11 illustrates an example in which a single object in a parts image is recognized, this is not limiting. If there are multiple objects in a parts image, the control unit 11 can recognize the multiple objects using the object recognition model 171.

In this case, the classification unit 71b calculates the feature amounts of the area candidates extracted by the area candidate extraction unit 71a, and classifies whether each subject shown in the area candidate is an object or not based on the calculated feature amounts, and if each subject is an object, the type of the object. The object recognition model 171 repeats the extraction and classification of area candidates, and outputs the recognition result including the position and type of each object.

In place of RCNN, any object detection algorithm may be used, such as CNN (Convolution Neural Network), Fast RCNN, Faster RCNN, Mask RCNN, SSD (Single Shot Multibook Detector), YOLO (You Only Look Once), SVM (Support Vector Machine), Bayesian network, or regression tree.

In this embodiment, the object recognition model 171 may be constructed by the image processing server 1, or may be constructed by an external information processing device. If the object recognition model 171 is constructed by an external information processing device, the image processing server 1 may download and install the trained object recognition model 171 from the external information processing device.

Note that the object recognition process is not limited to the machine learning process described above. For example, objects in part images may be recognized using local feature extraction methods such as A-KAZE (Accelerated KAZE), SIFT (Scale Invariant Feature Transform), SURF (Speed-Up Robust Features), ORB (Oriented FAST and Rotated BRIEF), and HOG (Histograms of Oriented Gradients).

Figures 12 and 13 are flowcharts showing the processing procedure when outputting part-specific information. The control unit 11 of the image processing server 1 acquires part images from the photographing device 2 via the communication unit 13 (step S101). The control unit 11 inputs the acquired part images to a trained object recognition model 171 that outputs a recognition result that recognizes an object in the part image when the acquired part image is input, and outputs the recognition result that recognizes the object in the part image (step S102).

Based on the recognition result, the control unit 11 determines whether or not an object has been recognized (step S103). If the control unit 11 determines that an object has not been recognized (NO in step S103), the process returns to step S101. If the control unit 11 determines that an object has been recognized (YES in step S103), the control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S104).

Specifically, the control unit 11 identifies the type of object (e.g., a star sticker, a round forced dent hole, etc.) based on the recognition result. The control unit 11 reads out the offset position and the size of the area image corresponding to the identified type of object from the storage unit 12 or the mass storage unit 17. For example, the offset position corresponding to the recognized object is 20 pixels to the right from the center of the bounding box of the object in the horizontal direction. The control unit 11 extracts an area image of the size of the read area image (e.g., 120 x 120 pixels) at the offset position read out from the center of the bounding box of the recognized object. Note that the area image extraction process is not limited to the center of the bounding box of the object described above. For example, the control unit 11 may extract an area image at an offset position starting from the coordinates of a specific corner of the object.

The control unit 11 transmits the part type ID, the offset position by the object, and the extracted area image to the factory server 5 via the communication unit 13 (step S105). The control unit 51 of the factory server 5 receives the part type ID, the offset position, and the area image transmitted from the image processing server 1 via the communication unit 53 (step S501).

The control unit 51 acquires part-specific information including manufacturing information, material information, and product attachment information of the part from the product DB 571 of the mass storage unit 57 based on the received part type ID, offset position, and area image (step S502). The control unit 51 transmits the acquired part-specific information to the image processing server 1 via the communication unit 53 (step S503). The control unit 11 of the image processing server 1 receives the part-specific information transmitted from the factory server 5 via the communication unit 13 (step S106) and assigns a part ID to the received part-specific information (step S107).

The control unit 11 transmits the extracted area image to the parts server 3 via the communication unit 13 in association with the assigned part ID (step S108). The control unit 31 of the parts server 3 receives the part ID and area image transmitted from the image processing server 1 via the communication unit 33 (step S301). The control unit 31 stores the area image in the part specific information DB 371 of the mass storage unit 37 in association with the received part ID (step S302).

The control unit 11 transmits the received part-specific information to the DB server 4 via the communication unit 13 in association with the assigned part ID (step S109). The control unit 41 of the DB server 4 receives the part ID and part-specific information transmitted from the image processing server 1 via the communication unit 43 (step S401). The control unit 41 stores the part-specific information in the part details DB 452 of the mass storage unit 45 in association with the received part ID (step S402).

According to this embodiment, it is possible to obtain part-specific information corresponding to an area image extracted at a specified offset position from an object in a part image.

According to this embodiment, it is possible to recognize objects in a part image using an object recognition model 171 based on the part image.

According to this embodiment, by aligning with the position of the object, it is possible to achieve highly accurate alignment without using a position correction system that uses a laser or the like.

(Embodiment 2)
The second embodiment relates to a form in which the product ID of a product using a part is output to the DB server 4 in association with the part ID of the part. Note that a description of the contents that overlap with the first embodiment will be omitted.

Figure 14 is an explanatory diagram explaining the operation of outputting a product ID in association with a part ID. The control unit 11 of the image processing server 1 acquires a part image from the photographing device 2 via the communication unit 13. The control unit 11 inputs the acquired part image to the object recognition model 171, and outputs the recognition result of recognizing an object in the part image. When the control unit 11 recognizes an object based on the recognition result, it extracts an area image of a predetermined size at a predetermined offset position from the recognized object.

Next, the process of acquiring a part ID from the parts server 3 will be described. The control unit 11 transmits the extracted area image to the parts server 3 via the communication unit 13. The control unit 31 of the parts server 3 receives the area image transmitted from the image processing server 1 via the communication unit 33, and extracts the features of the received area image. The control unit 31 compares the features of the extracted image area with the features of the part image stored in the part-specific information DB 371 of the mass storage unit 37.

When the control unit 31 determines that the features of both images match, it acquires the corresponding part ID from the part-specific information DB 371. Note that the feature comparison process is not limited to a perfect match, and may output a result indicating that the features of both images match if, for example, the degree of match is equal to or greater than a predetermined threshold. The feature comparison process may also be performed using a trained image recognition model generated by machine learning. The control unit 31 transmits the acquired part ID to the image processing server 1 via the communication unit 33. The control unit 11 of the image processing server 1 receives the part ID transmitted from the part server 3 via the communication unit 13.

Next, the process of acquiring the product ID from the factory server 5 will be described. The control unit 11 transmits the part type ID and the offset position by the object together with the extracted area image to the factory server 5 via the communication unit 13. The control unit 51 of the factory server 5 receives the part type ID, offset position, and area image transmitted from the image processing server 1 via the communication unit 53.

The control unit 51 acquires the corresponding product ID from the product DB 571 in the mass storage unit 57 based on the received part type ID, offset position, and area image. The control unit 51 transmits the acquired product ID to the image processing server 1 via the communication unit 53. The control unit 11 of the image processing server 1 receives the product ID transmitted from the factory server 5 via the communication unit 13, and transmits (outputs) the received product ID to the DB server 4 in association with the part ID.

The control unit 41 of the DB server 4 receives the product ID and part ID sent from the image processing server 1 via the communication unit 43. The control unit 41 associates the received product ID with the part ID and stores it in the product management DB 451 of the mass storage unit 45. Specifically, the control unit 41 stores the product ID, product type, product name, and part ID as one record in the product management DB 451.

Figures 15 and 16 are flowcharts showing the processing steps when outputting a product ID in association with a part ID. Note that the same reference numerals are used for the same contents as in Figures 12 and 13, and the description will be omitted. The control unit 11 of the image processing server 1 executes steps S101 to S104. The control unit 11 transmits the extracted area image to the parts server 3 via the communication unit 13 (step S121).

The control unit 31 of the parts server 3 receives the area image transmitted from the image processing server 1 via the communication unit 33 (step S321). Based on the received area image, the control unit 31 acquires the corresponding part ID from the part specific information DB 371 of the mass storage unit 37 (step S322). The control unit 31 transmits the acquired part ID to the image processing server 1 via the communication unit 33 (step S323). The control unit 11 of the image processing server 1 receives the part ID transmitted from the parts server 3 via the communication unit 13 (step S122).

The control unit 11 executes step S105. The control unit 51 of the factory server 5 executes step S501. The control unit 51 acquires the corresponding product ID from the product DB 571 of the mass storage unit 57 based on the received part type ID, offset position by the object, and area image (step S521). The control unit 51 transmits the acquired product ID to the image processing server 1 via the communication unit 53 (step S522).

The control unit 11 of the image processing server 1 receives the product ID sent from the factory server 5 via the communication unit 13 (step S123), associates the received product ID with the part ID, and sends it to the DB server 4 (step S124). The control unit 41 of the DB server 4 receives the product ID and part ID sent from the image processing server 1 via the communication unit 43 (step S421). The control unit 41 associates the received product ID with the part ID, and stores it in the product management DB 451 of the mass storage unit 45 (step S422).

According to this embodiment, by storing and managing information that associates parts with the products that use those parts, it is possible to ensure traceability in the process after shipment.

(Embodiment 3)
The third embodiment relates to a form in which a registered part is deleted. Note that a description of the contents that overlap with the first and second embodiments will be omitted. When the registration period of a second part associated with a registered first part based on a production plan for a product has passed a predetermined period, an alert or command to delete the registered first part can be output.

Figure 17 is an explanatory diagram showing an overview of a traceability system of embodiment 3. Note that the same reference numerals are used for the same contents as in Figure 1, and the description will be omitted. The system of this embodiment includes an information processing device 6.

The information processing device 6 is an information processing device that detects expired parts and transmits instructions to delete registered parts associated with the expired parts. The information processing device 6 is, for example, a server device, a personal computer, or a general-purpose tablet PC. In this embodiment, the information processing device 6 is a server device, and for simplicity, will be referred to as a deletion server 6 below.

Figure 18 is a block diagram showing an example of the configuration of the deletion server 6. The deletion server 6 includes a control unit 61, a storage unit 62, a communication unit 63, an input unit 64, a display unit 65, a reading unit 66, a large-capacity storage unit 67, and a clock unit 68. Each component is connected by a bus B.

The control unit 61 includes a processor such as a CPU, MPU, or GPU, and performs various information processing, control processing, and the like related to the deletion server 6 by reading and executing a control program 6P stored in the storage unit 62. Note that while the control unit 61 is described in FIG. 18 as being a single processor, it may be a multi-processor.

The storage unit 62 includes memory elements such as RAM and ROM, and stores the control program 6P or data necessary for the control unit 61 to execute processing. The storage unit 62 also temporarily stores data necessary for the control unit 61 to execute arithmetic processing. The communication unit 63 is a communication module for performing communication-related processing, and transmits and receives information between the parts server 3 and the DB server 4, etc., via the network N.

The input unit 64 is an input device such as a mouse, keyboard, touch panel, or button, and outputs received operation information to the control unit 61. The display unit 65 is a liquid crystal display or an organic EL display, and displays various information according to instructions from the control unit 61.

The reading unit 66 reads the portable storage medium 6a, which includes a CD-ROM or DVD-ROM. The control unit 61 may read the control program 6P from the portable storage medium 6a via the reading unit 66 and store it in the mass storage unit 67. The control unit 61 may also download the control program 6P from another computer via the network N or the like and store it in the mass storage unit 67. The control unit 61 may also read the control program 6P from the semiconductor memory 6b.

The mass storage unit 67 includes a recording medium such as an HDD or SSD. The mass storage unit 67 includes a parts management DB 671 and a production plan DB 672. The parts management DB 671 stores management information for managing the registration deadlines of parts based on the production plan of the product. The production plan DB 672 stores the production plan of the product. The production plan is information about the production plan, such as information about the product, the parts to assemble the product, the quantity of the parts, and the registration (delivery) date and time.

In this embodiment, the memory unit 62 and the large-capacity memory unit 67 may be configured as an integrated memory device. The large-capacity memory unit 67 may also be configured from multiple memory devices. Furthermore, the large-capacity memory unit 67 may be an external memory device connected to the deletion server 6.

In this embodiment, the deletion server 6 is described as being a single information processing device, but the processing may be distributed across multiple devices, or the deletion server 6 may be configured as a virtual machine. Furthermore, the deletion server 6 may be integrated with the factory server 5 or another server.

Figure 19 is an explanatory diagram showing an example of a record layout of the parts management DB 671. The parts management DB 671 includes a management ID column, a production plan ID column, a parts ID column, a registration status column, and a registration date and time column. The management ID column stores a uniquely specified ID of the management data to identify each management data. The production plan ID column stores a production plan ID that identifies the production plan. The parts ID column stores a part ID that identifies the part. The registration status column stores the registration status of the part on the parts server 3. The registration status column may store, for example, "registered" or "not registered". The registration date and time column stores date and time information when the part was registered on the parts server 3.

Figure 20 is an explanatory diagram showing an example of a record layout of production plan DB 672. Production plan DB 672 includes a production plan ID column, a product ID column, a parts number column, a parts ID column, and a registration deadline column. The production plan ID column stores a unique production plan ID to identify each production plan. The product ID column stores a product ID that identifies the product. The parts number column stores the quantity of parts to assemble the product. The parts ID column stores part IDs that identify the parts. The registration deadline column stores deadline information for registering parts in parts server 3.

The control unit 61 of the deletion server 6 extracts the part ID (multiple parts are possible) whose registration status is "unregistered" and the production plan ID corresponding to the part ID from the part management DB 671 of the mass storage unit 67. Based on the extracted part ID and production plan ID, the control unit 61 obtains the registration deadline of the part corresponding to the part ID from the production plan DB 672 of the mass storage unit 67.

The control unit 61 obtains the current date and time from the clock unit 68, and compares the obtained current date and time with the registration expiration date of the part. If the control unit 61 determines that the registration expiration date of the part has passed, it associates the part ID with the part and transmits (outputs) an alert to delete the part to the parts server 3 or DB server 4. The alert may be, for example, a message that includes a message that the registration expiration date of the part has passed. The control unit 61 may also create a report for the expired part and output the created report.

Furthermore, the control unit 61 can transmit (output) a command to delete the part to the parts server 3 or DB server 4 in association with the expired part ID. When the parts server 3 or DB server 4 receives a command to delete an expired part from the deletion server 6, it deletes the corresponding registered part that has been stored.

Figure 21 is a flowchart showing the processing procedure when deleting a registered part. The control unit 61 of the deletion server 6 extracts part IDs whose registration status is "unregistered" and the production plan IDs corresponding to the part IDs from the parts management DB 671 of the mass storage unit 67 (step S631). The extracted part ID may be a single part ID or multiple part IDs. An example of multiple part IDs is described below.

The control unit 61 acquires the registration deadline corresponding to each part ID from the production plan DB 672 of the mass storage unit 67 based on each extracted part ID and the production plan ID corresponding to the part ID (step S632). The control unit 61 acquires the current date and time using the clock unit 68, compares the acquired current date and time with the registration deadline of the part, and extracts expired part IDs (step S633).

The control unit 61 generates a report of the expired parts based on the extracted part ID (step S634). The report includes, for example, the part ID and the registration expiration date of the part. The control unit 61 transmits the generated report and a deletion instruction to delete the expired parts to the parts server 3 and the DB server 4 via the communication unit 63 (step S635). An alert including a notice to delete the expired parts may also be transmitted.

The control unit 31 of the parts server 3 receives the report and the deletion instruction sent from the deletion server 6 via the communication unit 33 (step S331). The control unit 31 reads the part ID from the received report and deletes the corresponding part from the part-specific information DB 371 of the mass storage unit 37 (step S332).

The control unit 41 of the DB server 4 receives the report and the deletion instruction sent from the deletion server 6 via the communication unit 43 (step S431). The control unit 41 reads the part ID from the received report and deletes the corresponding part from the part details DB 452 of the mass storage unit 45 (step S432).

Note that the process is not limited to the above. For example, the deletion server 6 may send only the generated report to the parts server 3 or DB server 4. In this case, the parts server 3 or DB server 4 receives and displays the report sent from the deletion server 6. After the person in charge checks the contents of the report, they may input a deletion instruction for the parts to be deleted to the parts server 3 or DB server 4.

According to this embodiment, if an error occurs when registering (authenticating) a part, an alert or deletion instruction is output to delete a registered part associated with that part, which can be useful for process and quality improvement.

(Embodiment 4)
The fourth embodiment relates to an embodiment for improving the recognition of objects. Note that a description of the contents that overlap with the first to third embodiments will be omitted.

To improve recognition of objects in the part image, a virtual camera lens center line or a lighting device can be used. The virtual camera lens center line will be described later. When a lighting device is used, the photographing device 2 photographs the part image after the lighting that illuminates the object is turned on. The control unit 11 of the image processing server 1 acquires the photographed part image from the photographing device 2. Thereafter, similar to the processing in embodiment 1, the control unit 11 performs processing to recognize objects in the part image, extraction processing of area images, and acquisition processing of part-specific information corresponding to the area images.

The lighting device is a device equipped with a light source that emits illumination light used to illuminate an object, and has optical functions such as light distribution control from the light source and light color adjustment. The light source device may be an LED (light emitting diode) or a projector. Other light source devices, such as an electrodeless light source, may also be used. In this embodiment, the lighting device is separate from the imaging device 2, but this is not limited to this. For example, the lighting device may be an integrated type built into the imaging device 2.

Figure 22 is an explanatory diagram regarding improved object recognition. Figure 22A is an image showing a part image without illumination. 10a is an object that is a feature point. 10b is an area image extracted from the part image at a specified offset position from the object and of a specified size.

Figure 22B is an image showing a part image captured using the virtual camera lens center line. 10c is a virtual camera lens center line that is an assumed camera lens center line. The virtual camera lens center line is an assumed straight line that passes through the center of the camera lens and is horizontal to the lens surface. For example, the virtual camera lens center line may be set by projection from a projector. If the shutter timing for object recognition is near the center of the camera (virtual camera lens center line), the camera takes multiple images in this vicinity. The image processing server 1 determines the image quality conditions, such as the focus condition, selects an image that is in good condition for Fingerprint of Object authentication, and performs Fingerprint of Object registration or authentication.

Figure 22C is an image showing a part image captured using striped illumination. 10d is an area image of a part image illuminated using striped illumination. A striped pattern is a design made up of multiple parallel or intersecting lines using two or more different colors or shades of the same color. By using a lighting device to project a striped pattern onto the surface of a part including an object and capturing the image, the object becomes easier to recognize due to the change in brightness that occurs on the part surface. Note that the illumination is not limited to striped illumination, and color LED illumination (red, yellow, green, or blue, etc.) may also be used.

Figure 22D is an image showing a part image captured using pinpoint lighting. 10e is an area image of a part image illuminated using pinpoint lighting. Pinpoint lighting is a type of lighting that focuses light on one point. By illuminating only the object with pinpoint lighting, the object becomes easier to recognize.

In the above process, an example of recognizing an object using various types of lighting has been described, but the present invention is not limited to this. It can also be applied to the Fingerprint of Things of a part surface (area image). For example, a striped pattern is projected onto the surface of a part with fine irregularities such as a highly glossy surface such as metal, and an image is captured. Features are extracted from the brightness changes that occur on the part surface, and the Fingerprint of Things of the part surface is recognized based on the extracted features.

According to this embodiment, it is possible to capture part images using a virtual camera lens center line, stripe pattern, or pinpoint lighting.

In this embodiment, by using a part image captured using a virtual camera lens center line, stripe pattern, or pinpoint lighting, it becomes easier to recognize objects in the part image.

(Embodiment 5)
The fifth embodiment relates to a form in which part images are acquired by two photographing devices. Note that a description of the contents that overlap with the first to fourth embodiments will be omitted. The system of the present embodiment includes a photographing device 2a and a photographing device 2b instead of the photographing device 2.

The photographing device 2a photographs the entire part. For example, if the photographing target area is the front door of a vehicle, the photographing device 2a photographs the front door to generate an image. For example, a wide-angle lens with a deep depth of field may be disposed in the photographing device 2a.

Photographing device 2b is a photographing device with a higher magnification ratio than photographing device 2a, and photographs a magnified portion of the part image photographed by photographing device 2a. Photographing device 2b may be equipped with a telephoto lens for taking a close-up image of a subject. Photographing device 2b is installed in a position where it can photograph an enlarged image of a part including an object when photographing device 2a recognizes the target part image.

The photographing device 2a or the photographing device 2b of this embodiment includes a wireless communication unit. The wireless communication unit is a wireless communication module for performing processing related to communication, and transmits and receives photographed images to and from the image processing server 1, etc., via the network N. Note that instead of the photographing device 2a or the photographing device 2b, an information processing terminal capable of photographing (e.g., a smartphone, a mobile phone, a tablet, a personal computer terminal, etc.) may be used.

Figure 23 is an explanatory diagram of capturing part images using two photographing devices. 11a is an object that is a feature point. 11b is an area image extracted with a specified size at a specified offset position from the object. The control unit 11 of the image processing server 1 acquires, via the communication unit 13, the first photographed image of the part being transported that was captured by the photographing device 2a.

The control unit 11 extracts the feature amounts of the acquired first captured image, and determines whether or not a part image is present in the first captured image based on the extracted feature amounts. For example, if the captured range is the front door, the control unit 11 may make a determination by comparing the feature amounts of the acquired first captured image with the feature amounts of the front door stored in advance. The feature amount extraction process may extract the feature amounts using a local feature amount extraction method such as A-KAZE or SIFT. Note that the part image determination process may be performed automatically using a model constructed by machine learning.

When the control unit 11 determines that a part image is present in the first captured image, it acquires the second captured image (an enlarged image of the part including the object) captured by the image capture device 2b via the communication unit 13. The control unit 11 inputs the acquired second captured image to the object recognition model 171 and outputs the recognition result of recognizing the object in the second captured image. Note that the object recognition process is the same as in embodiment 1, and therefore a description thereof will be omitted.

When the control unit 11 determines that no part image is present in the first captured image, it transmits an adjustment instruction for the capture angle, etc. to the capture device 2a. The capture device 2a adjusts the capture control parameters in response to the adjustment instruction transmitted from the image processing server 1.

Figure 24 is a flowchart showing the processing procedure when acquiring a part image. Note that the description will be given assuming that the parts in the part image have been registered in advance in the parts server 3. The photographing device 2a captures a first photographed image of the part being transported, and transmits the captured first photographed image to the image processing server 1 (step S241). The control unit 11 of the image processing server 1 acquires the first photographed image captured by the photographing device 2a via the communication unit 13 (step S641). The control unit 11 extracts features of the acquired first photographed image (step S642). The control unit 11 determines whether or not a part image is present in the first photographed image based on the extracted features (step S643).

When the control unit 11 determines that a part image is present in the first captured image (YES in S643), it transmits a capture instruction for the second captured image to the photographing device 2b via the communication unit 13 (step S644). The photographing device 2b captures the second captured image in response to the capture instruction for the second captured image transmitted from the image processing server 1 (step S245). The photographing device 2b transmits the captured second captured image to the image processing server 1 (step S246). The control unit 11 of the image processing server 1 receives the second captured image transmitted from the photographing device 2b via the communication unit 13 (step S645). The control unit 11 inputs the received second captured image to the object recognition model 171 and outputs a recognition result for recognizing the object in the second captured image (step S646).

When the control unit 11 determines that no part image exists in the first captured image (NO in step S643), it transmits an adjustment instruction for the capture angle, etc. to the capture device 2a via the communication unit 13 (step S647). The capture device 2a receives the adjustment instruction transmitted from the image processing server 1 (step S242). The capture device 2a adjusts the capture control parameters (e.g., angle, etc.) in accordance with the received adjustment instruction (step S243). The capture device 2a recaptures the first captured image (step S244) and returns to step S241. Note that if the capture device 2a is not adjusted, after the processing of NO in step S643, it returns to step S641 and waits until the next part is transported.

Since the photographing device 2a photographs the entire part, in addition to the object recognition processing described above, damage to the part such as scratches, dirt, cracks, wrinkles, deformations, etc. can be detected from the part image. The image processing server 1 may input the first photographed image of the part photographed by the photographing device 2a to a trained damage recognition model generated by machine learning, for example, and output a recognition result that recognizes damage to the part. The image processing server 1 may also extract features of the first photographed image using a local feature extraction method such as A-KAZE, and detect damage to the part based on the extracted features. The process of detecting damage to the part will be described later in embodiment 7.

According to this embodiment, by using two image capture devices to capture images of parts, it is possible to improve the recognition rate of objects.

(Embodiment 6)
The sixth embodiment relates to a form in which, when a distortion or tilt is detected in a part image, the offset position or the part image is corrected in accordance with the distortion or tilt. Note that a description of the contents that overlap with the first to fifth embodiments will be omitted.

If distortion or tilt is detected in the captured part image, the offset position or part image must be corrected. Figure 25 is an explanatory diagram explaining the correction of the offset position. Figure 25A is an explanatory diagram explaining a part image without distortion or tilt. 12a is an object in the part image. 12b is an area image extracted with a specified size at a specified offset position from the object.

Figure 25B is an explanatory diagram illustrating a part image in which left-right distortion (deformation) has occurred. This is an example of left-right distortion in which the vertical width of an object in a part image is equal to the reference (normal) vertical width, but the horizontal width is shorter than the reference horizontal width. In this case, if an area image is extracted at a specified offset position from the object, a positional shift will occur with respect to the area image to be extracted, and the offset position must be corrected.

Specifically, the control unit 11 of the image processing server 1 calculates the aspect ratio after transformation. The control unit 11 corrects the offset position based on the calculated aspect ratio and the reference aspect ratio. For example, the offset position is moved 20 pixels further to the right from the original offset position. In addition to directly correcting the offset position, the offset position may also be corrected by correcting the entire part image. Specifically, the control unit 11 performs aspect ratio correction processing based on the reference aspect ratio to bring the image closer to the reference part image, and extracts a region image at a specified offset position from the corrected part image.

Figure 25C is an explanatory diagram illustrating a part image with vertical distortion. In this example, the width of an object in a part image is equal to the reference width, but vertical distortion occurs such that the height is shorter than the reference height. The correction process for the offset position is the same as the correction process in Figure 25B, so a description of it will be omitted.

Figure 25D is an explanatory diagram explaining a part image in which area reduction distortion has occurred. Differences arise in the area of each area image depending on the perspective of the area image. If the photographing device 2 is installed at a great distance from the part to be photographed, the photographed part image will appear small. In this case, the offset position can be corrected by performing area (size) correction on the part image. Specifically, the control unit 11 performs enlargement processing on the area image by increasing the number of pixels that make up the area image of the part image based on the area of the reference part image.

Figure 25E is an explanatory diagram explaining a part image in which area enlargement distortion has occurred. When the photographing device 2 is installed at a close distance from the part to be photographed, the photographed part image appears large. In this case, the offset position can be corrected by performing area correction on the part image. Specifically, the control unit 11 performs a reduction process on the area image by reducing the number of pixels that make up the area image of the part image based on the area of the reference part image.

For example, nearest neighbor interpolation may be used for image scaling. Nearest neighbor interpolation is a linear interpolation method in which the pixel after scaling is divided by the scaling ratio, and the resulting value is rounded off to obtain the pixel at the coordinates that is used as is. In addition, other image scaling techniques such as bilinear interpolation, bicubic interpolation, and Lanczos interpolation may also be used.

Figure 25F is an explanatory diagram explaining a tilted part image. If there is a deviation in the shooting angle, it is necessary to perform tilt correction on the captured part image. Specifically, the control unit 11 calculates the tilt angle of the tilted part image based on the reference part image. The control unit 11 performs a rotation process according to the calculated tilt angle to correct the tilted part image.

Regarding the image processing described above, the entire part image may be corrected, or an area image of the object and offset position may be extracted from the part image, and the extracted area image may be corrected.

Figure 26 is a flowchart showing the processing steps when correcting a part image in which area distortion has occurred. The following shows an example in which area reduction distortion or area enlargement distortion has occurred. The control unit 11 of the image processing server 1 acquires the part image transmitted from the photographing device 2 via the communication unit 13 (step S151). The control unit 11 determines whether or not area distortion has occurred in the acquired part image (step S152). If the control unit 11 determines that area distortion has not occurred (NO in step S152), it ends the processing. If the control unit 11 determines that area distortion has occurred (YES in step S152), it determines whether or not area reduction has occurred (step S153).

When the control unit 11 determines that the area is being reduced (YES in step S153), it performs an enlargement process on the area image by increasing the number of pixels that make up the area image of the part image based on the area of the reference part image (step S154). The reference part image is stored in advance in the storage unit 12 or the mass storage unit 17. When the control unit 11 determines that the area is not being reduced (NO in step S153), it performs a reduction process on the area image by decreasing the number of pixels that make up the area image of the part image based on the area of the reference part image (step S155).

According to this embodiment, by performing offset position correction processing on distorted or tilted part images, it is possible to extract area images with high accuracy.

(Embodiment 7)
The seventh embodiment relates to a form in which both part-specific information (such as a fingerprint of an object) and information on damage to a part are output. Note that a description of the contents that overlap with the first to sixth embodiments will be omitted. Damage to a part includes scratches, stains, cracks, wrinkles, deformations, etc. of the part.

Figure 27 is a block diagram showing an example of the configuration of the image processing server 1 of embodiment 7. Note that the same reference numerals are used for the contents that overlap with those in Figure 2, and the description will be omitted. The large-capacity storage unit 17 stores a damage recognition model 172. The damage recognition model 172 is a recognizer that recognizes damage to parts in a part image based on the part image, and is a trained model generated by machine learning.

Figure 28 is a block diagram showing an example of the configuration of the DB server 4 in embodiment 7. Note that the same reference numerals are used for the contents that overlap with those in Figure 5, and the description will be omitted. The large-capacity storage unit 45 stores a part damage information DB 453. The part damage information DB 453 stores information about damage to parts detected based on part images. The information about damage to parts includes the location and type of damage, etc.

Figure 29 is an explanatory diagram showing an example of a record layout of part damage information DB453. Part damage information DB453 includes a part ID column, a damage type column, a position column, and a detection date and time column. The part ID column stores a part ID that identifies a part. The damage type column stores the type of damage to the part. The position column stores the position of the damage (e.g., coordinates, etc.). The detection date and time column stores date and time information when damage to the part was detected.

The image processing server 1 can acquire and output both part-specific information and information relating to damage to a part from the same part image captured by the photographing device 2. Note that the process of acquiring part-specific information is similar to the process of acquiring part-specific information in embodiment 1, and therefore a description thereof will be omitted. The process of acquiring information relating to damage to a part will be described in detail below.

The control unit 11 of the image processing server 1 acquires part images from the photographing device 2 via the communication unit 13. The control unit 11 inputs the acquired part images to a trained damage recognition model 172 that outputs a recognition result that recognizes damage to a part in the part image when the acquired part image is input, and outputs a recognition result that recognizes damage in the part image.

The damage recognition model 172 is used as a program module that is part of artificial intelligence software. The damage recognition model 172 is a recognizer that has constructed a neural network that takes a part image as input and outputs a recognition result including the location and type of damage to the part in the part image.

In this embodiment, the damage recognition model 172 performs estimation using an RCNN as an example. The damage recognition model 172 includes a region candidate extraction unit and a classification unit. The classification unit includes a neural network.

The control unit 11 of the image processing server 1 performs learning using a combination of part images and training data in which the part images are associated with the types of damage (e.g., scratches, stains, cracks, wrinkles, deformations, etc.) in the part images. The training data is data in which the damage in the part images is labeled for each part image.

When a part image is input to the trained damage recognition model 172, the area candidate extraction unit extracts area candidates of various sizes from the part image. The classification unit calculates the feature values of the extracted area candidates. Based on the calculated feature values, the classification unit classifies whether the subject shown in the area candidate is damage to a part, and if the subject is damage, the type of damage. The damage recognition model 172 repeats the extraction and classification of area candidates.

For candidate areas classified as having damage with a probability higher than a predetermined threshold (e.g., 85%), the damage recognition model 172 outputs the probability that damage to the part is present, the location of the damage (coordinate range of the area), and the type of damage.

In addition to the above-mentioned processing, it is also possible to use a dedicated recognition model that has been trained for each type of damage to a part. For example, the control unit 11 performs learning using a combination of training data in which part images and scratches in the part images are associated with each other, and constructs a scratch recognition model. The control unit 11 uses the constructed scratch recognition model to input the part image and output the recognition result of recognizing scratches in the part image.

In place of RCNN, any object detection algorithm may be used, such as CNN, Fast RCNN, Faster RCNN, Mask RCNN, SSD, YOLO, SVM, Bayesian network, or regression tree.

Note that the process of recognizing damage to parts is not limited to the machine learning process described above. For example, damage in part images may be recognized using local feature extraction methods such as A-KAZE and SIFT.

The control unit 11 assigns a part ID to the acquired part unique information and part damage information. The control unit 11 transmits the part unique information and part damage information (recognition result) to the DB server 4 via the communication unit 13 in association with the assigned part ID. The control unit 41 of the DB server 4 receives the part ID, part unique information, and part damage information transmitted from the image processing server 1 via the communication unit 43. The control unit 41 stores the received part ID, part unique information, and part damage information in the part details DB 452 and part damage information DB 453 of the mass storage unit 45.

Specifically, the control unit 41 associates the damage type, location, and detection date and time of the part with the part ID and stores them as one record in the part damage information DB 453. Note that the part-specific information storage process is the same as the part-specific information storage process in embodiment 1, and therefore a description thereof will be omitted.

Figure 30 is a flowchart showing the processing steps when outputting both part-specific information and information related to part damage. The control unit 11 of the image processing server 1 acquires part images from the photographing device 2 via the communication unit 13 (step S161). The control unit 11 acquires part-specific information based on the acquired part images (step S162). The control unit 11 inputs the acquired part images into the trained damage recognition model 172, and acquires information related to damage to the parts in the part images (damage recognition results) (step S163).

The control unit 11 assigns a part ID to the acquired part unique information and part damage information (step S164). The control unit 11 transmits the part unique information and part damage information to the DB server 4 via the communication unit 13 in association with the assigned part ID (step S165). The control unit 41 of the DB server 4 receives the part ID, part unique information, and part damage information transmitted from the image processing server 1 via the communication unit 43 (step S461). The control unit 41 stores the received part ID, part unique information, and part damage information in the part details DB 452 and part damage information DB 453 of the mass storage unit 45 (step S462).

According to this embodiment, by outputting both part-specific information and information regarding damage to the part, it is possible to simultaneously realize a traceability function using part-specific information (such as a fingerprint of an object) and an appearance inspection function.

(Embodiment 8)
The eighth embodiment relates to a form in which registered parts that have been stored in a storage location for a certain period of time or more are detected. Note that a description of the contents that overlap with the first to seventh embodiments will be omitted.

Figure 31 is an explanatory diagram showing an example of a record layout of the parts unique information DB 371 of the eighth embodiment. The parts unique information DB 371 is stored in the mass storage unit 37 of the parts server 3. Note that the description of the contents that overlap with FIG. 4 will be omitted. The parts unique information DB 371 includes a storage information column. The storage information column includes a storage location column, a location details column, a storage start date and time column, and a storage period column. The storage location column stores the storage location until the part is attached to the product. The location details column stores location information such as the latitude, longitude, or shelf number of the storage location. The storage start date and time column stores date and time information when storage of the part started. The storage period column stores the storage period until the part is attached to the product (for example, six months, ten months, or one year).

Figure 32 is an explanatory diagram showing an example of a record layout of the part details DB 452 of embodiment 8. The part details DB 452 is stored in the mass storage unit 45 of the DB server 4. Note that explanations of the contents that overlap with Figure 7 will be omitted. The part details DB 452 includes a storage information column and an installation information column. Note that the storage information column is similar to the storage information column of the part specific information DB 371, so explanations will be omitted. The installation information column includes an installation robot ID column, an installation manufacturer name column, a location address column, and an installation date and time column. The installation robot ID column, the installation manufacturer name column, and the location address column are similar to Figure 7, so explanations will be omitted. The installation date and time column stores information on the date and time when the part was installed on the product.

First, the parts storage information is registered in advance in the parts server 3. The storage information includes the storage location (e.g., location name, longitude, latitude, or shelf information, etc.), storage start date and time, and storage period until the part is attached to the product. Specifically, the parts server 3 acquires the parts storage information. For example, the parts server 3 may accept the parts storage information input by a person in charge at the parts production factory through the input unit 34. The parts server 3 stores the part storage location, detailed location information (longitude, latitude, or shelf number, etc.), storage start date and time, and storage period included in the acquired parts storage information in the parts specific information DB 371 in association with the part ID.

Next, the process of outputting part storage information to the DB server 4 in association with the part ID will be described. The image processing server 1 acquires a part image including an object that is a feature point from the photographing device 2. As in the first embodiment, the image processing server 1 recognizes an object in the acquired part image using the trained object recognition model 171. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object. The image processing server 1 transmits the extracted area image to the part server 3.

The parts server 3 receives the area image sent from the image processing server 1 and extracts the features of the received area image. The parts server 3 compares the features of the extracted area image with the features of the part image stored in the parts specific information DB 371. Note that the feature comparison process is the same as in the first embodiment, so a description thereof will be omitted. If the parts server 3 determines that the features of both match, it obtains the corresponding part ID and the storage information of the part corresponding to the part ID from the parts specific information DB 371.

The parts server 3 transmits the acquired part ID and part storage information to the image processing server 1. The image processing server 1 receives the part ID and part storage information transmitted from the parts server 3, and transmits the received part ID and part storage information to the DB server 4. The DB server 4 receives the part ID and part storage information transmitted from the image processing server 1. The DB server 4 stores the part storage location, detailed location information (longitude, latitude, shelf number, etc.), storage start date and time, and storage period in the parts details DB 452 in association with the received part ID.

Next, a process for detecting registered parts that have been retained in a storage location for a certain period of time or more will be described. The image processing server 1 determines whether a registered part has been retained in a storage location for a certain period of time or more, for example, at predetermined intervals by batch processing. Specifically, the image processing server 1 acquires a registered part ID and storage information of the part corresponding to the part ID from the parts server 3. The image processing server 1 determines whether the part has been retained for a certain period of time or more (for example, six months) based on the storage start date and time and storage period contained in the acquired storage information of the part. If the image processing server 1 determines that the part has been retained for a certain period of time or more, it acquires the part ID of the part.

The image processing server 1 determines whether the part has been attached to the product based on the acquired part ID. Specifically, the image processing server 1 determines whether the part attachment information exists in the part details DB 452 of the DB server 4 based on the part ID. The part attachment information includes the ID of the attachment robot, the manufacturer name of the attachment robot used when attaching the part to the product, the address or name of the production line where the part was attached, or the date and time when the part was attached to the product.

If the image processing server 1 determines that the installation information for the part exists in the part details DB 452, it determines that the part is not a retained part because it has already been installed in the product. If the image processing server 1 determines that the installation information for the part does not exist in the part details DB 452, it identifies the part as a retained part.

Figure 33 is a flowchart showing the processing steps when outputting part storage information in association with a part ID. The control unit 11 of the image processing server 1 acquires a part image including an object from the photographing device 2 via the communication unit 13 (step S171). The control unit 11 recognizes the object in the acquired part image using the learned object recognition model 171 (step S172). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S173). The control unit 11 transmits the extracted area image to the part server 3 via the communication unit 13 (step S174).

The control unit 31 of the parts server 3 receives the area image sent from the image processing server 1 via the communication unit 33 (step S371) and extracts features of the received area image (step S372). Based on the features of the extracted area image, the control unit 31 acquires the relevant part ID and storage information of the part corresponding to the part ID from the part specific information DB 371 (step S373). The control unit 31 transmits the acquired part ID and storage information of the part to the image processing server 1 via the communication unit 33 (step S374).

The control unit 11 of the image processing server 1 receives the part ID and part storage information transmitted from the parts server 3 via the communication unit 13 (step S175), and transmits the received part ID and part storage information to the DB server 4 (step S176). The control unit 41 of the DB server 4 receives the part ID and part storage information transmitted from the image processing server 1 via the communication unit 43 (step S471). The control unit 41 stores the part storage location, detailed location information (longitude, latitude, shelf number, etc.), storage start date and time, and storage period in the part details DB 452 of the mass storage unit 45 in association with the received part ID (step S472), and ends the process.

Figure 34 is a flowchart showing the processing steps for detecting retained parts. The control unit 11 of the image processing server 1 acquires multiple registered part IDs and storage information for parts corresponding to each part ID from the part specific information DB 371 of the parts server 3 via the communication unit 13 (step S181). The control unit 11 acquires one part ID from the multiple acquired part IDs (step S182). The control unit 11 determines whether the storage period (e.g., six months) has passed based on the storage start date and time included in the storage information for the part corresponding to the acquired part ID (step S183).

When the control unit 11 determines that the storage period of the part has not passed (NO in step S183), it returns to the process of step S182 and acquires the next part ID. When the control unit 11 determines that the storage period of the part has passed (YES in step S183), it determines whether the part has been attached to the product (step S184). Specifically, the control unit 11 transmits the part ID to the DB server 4 via the communication unit 13 and inquires whether the installation information of the part already exists in the part details DB 452. The installation information includes the ID of the installation robot, the manufacturer name of the installation robot, the location address, or the installation date and time. When the installation information of the part exists in the part details DB 452, the control unit 11 determines that the part has already been attached to the product. When the installation information of the part does not exist in the part details DB 452, the control unit 11 determines that the part has not been attached to the product.

If the control unit 11 determines that the part has already been attached to the product (YES in step S184), the control unit 11 returns to the process of step S182 and acquires the next part ID. If the control unit 11 determines that the part has not been attached to the product (NO in step S184), the control unit 11 identifies the part as a retention target (step S185). The control unit 11 determines whether the part ID is the last part ID among the multiple part IDs (step S186).

If the control unit 11 determines that the part ID is not the last one (NO in step S186), the process returns to step S182 and acquires the next part ID. If the control unit 11 determines that the part ID is the last one (YES in step S186), the control unit 11 generates a report for the part to be retained (step S187) and terminates the process. The report may be, for example, a PDF (Portable Document Format) file that describes the part ID to be retained.

Note that the process is not limited to the above. For example, the parts server 3 extracts parts that have been left in a storage location for a certain period of time from the parts specific information DB 371 at predetermined intervals and transmits them to the image processing server 1. If the image processing server 1 determines that a part transmitted from the parts server 3 is not attached to a product, it may identify the part as a retained part.

According to this embodiment, the storage location and storage period until the part is attached to the product can be output to DB server 4 in association with the part ID.

According to this embodiment, it is possible to strengthen parts management by detecting registered parts that have been stored in a storage location for a certain period of time or more.

(Embodiment 9)
The ninth embodiment relates to a form in which a part to be compared is compared with a registered part to determine whether a part has been replaced or the part state. Note that a description of the contents that overlap with the first to eighth embodiments will be omitted.

The part state refers to the degree of wear (deterioration) of the part as it is used. Note that in this embodiment, an example is described in which parts are classified into three wear levels according to their degree of wear, but other classification methods are similarly applicable. For example, wear level 1 is a normal state in which the part is not worn out. Wear level 2 is a level in which the part is not worn out very much and the part can continue to be used. Wear level 3 is a level in which the part is worn out and maintenance or replacement is recommended.

Figure 35 is a block diagram showing an example of the configuration of the image processing server 1 of embodiment 9. Note that the same reference numerals are used for the contents that overlap with those in Figure 2, and the description thereof will be omitted. A wear identification model 173 is stored in the large-capacity storage unit 17. The wear identification model 173 is a classifier that identifies the wear level of a part based on a part image, and is a trained model generated by machine learning.

Figure 36 is a block diagram showing an example of the configuration of the DB server 4 of the ninth embodiment. Note that the same reference numerals are used for the same contents as in Figure 5, and the description will be omitted. The large-capacity storage unit 45 stores a matching result DB 454. The matching result DB 454 stores the matching results obtained by matching the part to be matched with the registered parts.

Figure 37 is an explanatory diagram showing an example of a record layout of the matching result DB 454. The matching result DB 454 includes a part ID column, a part image column, a replaced/not replaced column, a wear level column, and a matching date/time column. The part ID column stores a part ID for identifying a part. The part image column stores an area image of the whole or a part of the part image. The replaced/not replaced column stores information on whether the part has been replaced. The wear level column stores the wear level of the part. The matching date/time column stores date/time information on when the part to be matched was matched with a registered part.

First, the image processing server 1 compares the part to be compared with registered parts. Specifically, the image processing server 1 acquires a part image of the part to be compared from the photographing device 2. As in the first embodiment, the image processing server 1 recognizes an object in the acquired part image using the trained object recognition model 171. The image processing server 1 extracts an area image of a predetermined size at a predetermined offset position from the recognized object.

The image processing server 1 transmits the extracted area image to the factory server 5. The factory server 5 receives the area image transmitted from the image processing server 1. The factory server 5 performs a matching process between the received area image and the area image stored in the product DB 571. The factory server 5 transmits the matching result including the degree of match to the image processing server 1. The image processing server 1 receives the matching result transmitted from the factory server 5.

Based on the received matching result, the image processing server 1 determines whether or not the part-specific information of the part exists in the product DB 571. For example, if the degree of match is equal to or greater than a predetermined threshold (e.g., 95%), the image processing server 1 determines that the part-specific information exists in the product DB 571. If the degree of match is less than the predetermined threshold, the image processing server 1 determines that the part-specific information does not exist in the product DB 571.

If the image processing server 1 determines that the part-specific information does not exist in the product DB 571, it determines that the part has been replaced with a part from another company or a part with a different model number. If the image processing server 1 determines that the part-specific information exists in the product DB 571, it determines the wear level of the part. Specifically, the image processing server 1 inputs a part image into a trained wear identification model 173 that outputs an identification result that identifies the wear level corresponding to the part image when a part image is input, and outputs an identification result that identifies the wear level corresponding to the part image. The wear level identification process will be described later.

The image processing server 1 transmits the result of matching between the part to be matched and the registered parts to the DB server 4. The result of matching includes the part ID of the part to be matched, the part image, whether or not it has been replaced, the wear level of the part, or the matching date and time. The DB server 4 stores the result of matching transmitted from the image processing server 1 in the matching result DB 454. Specifically, the DB server 4 associates the part ID with the part image, whether or not it has been replaced, the wear level, and the matching date and time, and stores them as one record in the matching result DB 454. Note that if the wear level has not been determined, a blank or "-" may be stored in the wear level column.

Next, the process of identifying the wear level of a part using the wear identification model 173 will be described. FIG. 38 is an explanatory diagram of the wear identification model 173. The wear identification model 173 is used as a program module that is part of artificial intelligence software. The wear identification model 173 is a classifier in which a neural network has been constructed that takes as input an image of the entire or partial area of a part image and outputs the results of identifying the wear level corresponding to the input image.

The image processing server 1 constructs (generates) the wear identification model 173 by performing deep learning to learn the features of wear areas within an entire or partial area image of a part image. For example, the wear identification model 173 is a CNN, and has an input layer that accepts input of a part image, an output layer that outputs the results of identifying the wear level corresponding to the part image, and an intermediate layer that has been trained by backpropagation.

The input layer has multiple neurons that accept input of pixel values of each pixel included in the part image, and passes the input pixel values to the intermediate layer. The intermediate layer has multiple neurons that extract features of the part image, and passes the extracted image features to the output layer. The intermediate layer is configured with an alternating convolution layer that convolves the pixel values of each pixel input from the input layer and a pooling layer that maps the pixel values convolved in the convolution layer, compressing the pixel information of the part image while ultimately extracting the image features. The intermediate layer then uses a fully connected layer whose parameters have been learned by backpropagation to identify (predict) the wear level corresponding to the part image. The identification result is output to the output layer, which has multiple neurons.

The image processing server 1 acquires multiple combinations of training data in which part images are associated with wear levels corresponding to the part images. The training data is data in which part images are labeled with wear levels corresponding to the part images. The image processing server 1 performs learning using the acquired training data.

The image processing server 1 inputs part images, which are training data, into the input layer, and after arithmetic processing in the intermediate layer, obtains a recognition result from the output layer that identifies the wear level corresponding to the part image. As shown in the figure, the recognition result output from the output layer is a continuous probability value (e.g., a value ranging from "0" to "1") indicating the wear level.

The image processing server 1 compares the classification results output from the output layer with the information labeled for the part images in the training data, i.e., the correct answer value, and optimizes the parameters used in the calculation process in the intermediate layer so that the output value from the output layer approaches the correct answer value. The parameters in question are, for example, the weights (coupling coefficients) between neurons and the coefficients of the activation functions used in each neuron. There are no particular limitations on the method of optimizing the parameters, but for example, the image processing server 1 optimizes various parameters using the backpropagation method.

The image processing server 1 performs the above processing on each part image included in the training data to generate a wear identification model 173. As a result, for example, the image processing server 1 can use the training data to learn the wear identification model 173, thereby constructing a model that can identify the wear level corresponding to the part image.

When the image processing server 1 acquires a part image, it inputs the acquired part image to the wear identification model 173. The image processing server 1 performs a calculation process to extract features of the part image in the middle layer of the wear identification model 173. The image processing server 1 inputs the extracted features to the output layer of the wear identification model 173, and acquires as an output an identification result that identifies the wear level corresponding to the part image.

As shown in the figure, a classification result is output in which the probability values of "wear level 1 (no wear)", "wear level 2", and "wear level 3" for the part image are "0.10", "0.84", and "0.06", respectively. A predetermined threshold value may also be used to output the classification result. For example, if the image processing server 1 determines that the probability value (0.84) of "wear level 2" is equal to or greater than a predetermined threshold value (e.g., 0.80), it outputs a classification result in which the wear level corresponding to the part image is "wear level 2". Note that, without using the above-mentioned threshold value, the wear level corresponding to the highest probability value from the probability values of each wear level identified by the wear identification model 173 may be output as the classification result.

The wear level identification process is not limited to the machine learning method described above. For example, the wear level may be identified using local feature extraction methods such as A-KAZE, SIFT, SURF, ORB, and HOG.

Figure 39 is a flowchart showing the processing steps when matching parts. The control unit 11 of the image processing server 1 acquires a part image of the part to be matched from the photographing device 2 via the communication unit 13 (step S191). The control unit 11 recognizes an object in the acquired part image using the trained object recognition model 171 (step S192). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from the recognized object (step S193).

The control unit 11 transmits the extracted area image to the factory server 5 via the communication unit 13 (step S194). The control unit 51 of the factory server 5 receives the area image transmitted from the image processing server 1 via the communication unit 53 (step S591). The control unit 51 performs a matching process between the received area image and the area image stored in the product DB 571 of the mass storage unit 57 (step S592). The control unit 51 transmits the matching result including the degree of match to the image processing server 1 via the communication unit 53 (step S593).

The control unit 11 of the image processing server 1 receives the matching result sent from the factory server 5 via the communication unit 13 (step S195). Based on the received matching result, the control unit 11 determines whether or not part-specific information of the part exists in the product DB 571 (step S196). For example, if the degree of match included in the matching result is equal to or greater than a predetermined threshold (e.g., 95%), the control unit 11 determines that part-specific information exists in the product DB 571.

When the control unit 11 determines that the part-specific information does not exist in the product DB 571 (NO in step S196), the control unit 11 transitions to the processing of step S198 described below. When the control unit 11 determines that the part-specific information exists in the product DB 571 (YES in step S196), the control unit 11 determines the wear level of the part using the trained wear identification model 173 (step S197). Specifically, the control unit 11 inputs a part image to the trained wear identification model 173, which outputs an identification result that identifies the wear level corresponding to the part image when a part image is input, and outputs an identification result that identifies the wear level corresponding to the part image.

The control unit 11 transmits the matching result, including the part ID, part image, replacement/non-replacement information, wear level, or matching date and time, to the DB server 4 via the communication unit 13 (step S198). The control unit 41 of the DB server 4 receives the matching result transmitted from the image processing server 1 via the communication unit 43 (step S491). The control unit 41 stores the matching result in the matching result DB 454 in association with the part ID (step S492), and ends the process.

According to this embodiment, by comparing the part to be compared with registered parts, it is possible to determine whether a part has been replaced or the part's condition.

According to this embodiment, by determining the part condition, it is possible to use the information as predictive data for the timing of part replacement or maintenance, etc.

(Embodiment 10)
The tenth embodiment relates to a form in which a registration or matching process is performed for a part in which a plurality of objects are distributed. Note that a description of the contents that overlap with the first to ninth embodiments will be omitted.

First, the process of registering a part in which multiple objects are distributed will be described. Figure 40 is a flowchart showing the processing steps for registering a part. The part in Figure 40 has multiple objects distributed.

The control unit 11 of the image processing server 1 acquires a part image including at least two or more objects from the photographing device 2 via the communication unit 13 (step S01). The control unit 11 recognizes each object in the acquired part image using the trained object recognition model 171 (step S02). Note that the object recognition process is the same as in embodiment 1, and therefore a description thereof will be omitted.

The control unit 11 extracts area images of a predetermined size at a predetermined offset position from each recognized object (step S03). The control unit 11 acquires part-specific information corresponding to any of the extracted multiple area images from the product DB 571 of the factory server 5 via the communication unit 13 (step S04). The control unit 11 assigns a part ID to the acquired part-specific information (step S05). The control unit 11 transmits the assigned part ID and multiple area images to the parts server 3 via the communication unit 13 (step S06). The control unit 11 transmits the assigned part ID and part-specific information to the DB server 4 via the communication unit 13 (step S07).

The control unit 31 of the parts server 3 receives the part ID and multiple area images transmitted from the image processing server 1 through the communication unit 33 (step S08). The control unit 31 stores the multiple area images in the part specific information DB 371 of the mass storage unit 37 in association with the received part ID (step S09), and ends the process. The control unit 41 of the DB server 4 receives the part ID and part specific information transmitted from the image processing server 1 through the communication unit 43 (step S10). The control unit 41 stores the part specific information in the part detail DB 452 of the mass storage unit 45 in association with the received part ID (step S11), and ends the process.

Next, we will explain the matching process for parts in which multiple objects are distributed. Figure 41 is a flowchart showing the processing steps for performing the matching process for parts. In Figure 41, multiple objects are distributed on the part.

The control unit 11 of the image processing server 1 acquires a part image including one or more objects from the photographing device 2 via the communication unit 13 (step S15). The control unit 11 recognizes each object in the acquired part image using the trained object recognition model 171 (step S16). The control unit 11 extracts an area image of a predetermined size at a predetermined offset position from each recognized object (step S17).

The control unit 11 determines whether or not part-specific information can be acquired from the product DB 571 of the factory server 5 based on the extracted multiple area images via the communication unit 13 (step S18). Specifically, if the control unit 11 has acquired part-specific information corresponding to any of the multiple area images, it determines that the part-specific information can be acquired. If the control unit 11 determines that part-specific information corresponding to all area images cannot be acquired, it determines that the part-specific information cannot be acquired.

If the control unit 11 determines that it cannot obtain the part-specific information (NO in step S18), it determines that authentication of the part has failed (step S22) and ends the process. If the control unit 11 determines that it is possible to obtain the part-specific information (YES in step S18), it performs a process of verifying the part-specific information (step S19).

Specifically, the control unit 11 compares the part-specific information corresponding to each area image in each area image. For example, when the control unit 11 extracts area image A, area image B, and area image C from the part image, it acquires the part-specific information corresponding to the acquired area image A, area image B, and area image C from the product DB 571. The control unit 11 compares the part-specific information corresponding to area image A with the part-specific information corresponding to area image B. The control unit 11 compares the part-specific information corresponding to area image A with the part-specific information corresponding to area image C. The control unit 11 compares the part-specific information corresponding to area image B with the part-specific information corresponding to area image C.

The control unit 11 determines whether the number of matches is equal to or greater than a predetermined threshold (e.g., 2) (step S20). If the control unit 11 determines that the number of matches is equal to or greater than the predetermined threshold (YES in step S20), it determines that authentication of the part has been successful (step S21) and ends the process. If the control unit 11 determines that the number of matches is less than the predetermined threshold (NO in step S20), it transitions to the process of step S22.

This embodiment makes it possible to perform registration or matching processes for parts in which multiple objects are distributed.

According to this embodiment, by registering multiple area images, it is possible to improve the accuracy of part authentication.

<Modification 1>
The registration or matching process for a cylindrical part will now be described. Fig. 42 is an explanatory diagram showing an example of a cylindrical part having a plurality of objects distributed on its end surface.

Figure 42A is an explanatory diagram showing an example of a captured image of a cylinder end surface for registration of a cylindrical object. As shown in the figure, three objects including 13a, 13b, and 13c are arranged on the cylinder end surface at 120 degree intervals. The image processing server 1 acquires a captured image of the cylinder end surface including objects 13a, 13b, and 13c from the photographing device 2. The image processing server 1 recognizes objects 13a, 13b, and 13c in the acquired captured image of the cylinder end surface.

The image processing server 1 extracts area images of a predetermined size at a predetermined offset position from each object, similar to the process described above. The image processing server 1 acquires part-specific information corresponding to any one of the extracted area images from the product DB 571 of the factory server 5. The image processing server 1 assigns a part ID to the acquired part-specific information. The image processing server 1 registers the part in the parts server 3 and DB server 4 in association with the assigned part ID. Note that the registration process is similar to the process described above, so a description thereof will be omitted.

Figure 42B is an explanatory diagram showing an example of a captured image of a tube end face for matching of a cylindrical object. Depending on the shooting position or the shape of the part, it may not be possible to capture the same location each time. For example, in the case of a cylindrical object, if the cylindrical object rotates, it may not be possible to identify the position and it may not be possible to capture the same location each time. Furthermore, when the entire tube end face of a cylindrical object is photographed, the area of the effective edge becomes smaller. Furthermore, when a large portion of the edge is photographed, there will be an area that is partially hidden.

As shown in the figure, when a large image of the edge of the cylinder end face is captured, the captured image of the cylinder end face includes the 13a and 13b objects, but does not include the 13c object. The image processing server 1 acquires the captured image of the cylinder end face including the 13a and 13b objects from the imaging device 2. The image processing server 1 recognizes the 13a and 13b objects in the acquired captured image of the cylinder end face. As in the process described above, the image processing server 1 determines whether or not part-specific information corresponding to the area image extracted based on the recognized 13a and 13b objects can be acquired from the product DB 571 of the factory server 5.

When the image processing server 1 determines that it cannot acquire both the part-specific information (first part-specific information) corresponding to the area image extracted based on the 13a object and the part-specific information (second part-specific information) corresponding to the area image extracted based on the 13b object, it determines that authentication of the part has failed because the part does not exist in the product DB 571. Furthermore, when the image processing server 1 acquires only either the first part-specific information or the second part-specific information, it determines that authentication of the part has failed.

When the image processing server 1 acquires both the first part-specific information and the second part-specific information, it compares the first part-specific information with the second part-specific information. When the image processing server 1 determines that the first part-specific information and the second part-specific information match, it determines that authentication of the part has been successful.

With this modified example, it is possible to improve authentication accuracy by matching based on multiple objects for cylindrical objects, for which it is difficult to photograph the same location each time.

(Embodiment 11)
The eleventh embodiment relates to a form in which a part is photographed based on the parameters of the photographing conditions when the reference image was photographed. Note that a description of the contents that overlap with the first to tenth embodiments will be omitted.

The reference image including the object that is to be a feature point is captured by, for example, the camera 2. Note that the camera that captures the reference image is not limited to the camera 2, and may be, for example, an external camera. A predetermined tolerance range set according to the parameters of the shooting conditions under which the reference image was captured is stored in advance in the memory unit 12 of the image processing server 1. The parameters of the shooting conditions include a similarity score, a frequency component distribution score, a variability score, a luminance score, or a hue score. The tolerance range is the range within which the value of each parameter can be adjusted.

The similarity score is a numerical value that indicates the degree of similarity between the reference image and the image to be photographed. The higher the similarity score, the higher the similarity between the reference image and the image to be photographed. The frequency component distribution score is a numerical value that indicates the frequency components obtained by transforming the image data, for example, by a fast Fourier transform (FFT). The frequency component distribution includes the distribution (proportion) of low-frequency components, mid-frequency components, and high-frequency components. The variability score is a standard deviation that indicates the magnitude of variability. The focus of the image can be confirmed by using the frequency component distribution score and the variability score. The luminance score is a numerical value that indicates the brightness of each color component R (red), G (green), and B (blue), and includes the MAX luminance, MIN luminance, or average luminance score. The hue score is a numerical value that indicates the color tone.

The image processing server 1 acquires the predetermined allowable range of the parameters from the memory unit 12. When the photographing device 2 photographs a part to be registered or matched, it transmits the part image being photographed to the image processing server 1. The image processing server 1 receives the part image transmitted from the photographing device 2 and determines whether the parameters of the received part image are within the predetermined allowable range. If the image processing server 1 determines that the parameters are within the predetermined allowable range, it transmits an instruction to photograph the part to the photographing device 2.

The photographing device 2 receives the photographing instruction sent from the image processing server 1, and photographs the part in accordance with the received photographing instruction. The photographing device 2 transmits the photographed part image to the image processing server 1. Thereafter, the image processing server 1 performs part registration or matching processing using the part image transmitted from the photographing device 2, as in the first to tenth embodiments.

In the present embodiment, an example has been described in which the process of determining whether the parameters of the part image are within the predetermined allowable range is performed on the image processing server 1 side, but this is not limiting and the process may be performed on the photographing device 2 side. Specifically, the photographing device 2 acquires the predetermined allowable range set according to the parameters of the photographing conditions from the storage unit or the image processing server 1. When the photographing device 2 determines that the parameters of the part image being photographed are within the predetermined allowable range, it photographs the part. The photographing device 2 transmits the photographed part image to the image processing server 1.

According to this embodiment, it is possible to obtain an optimal part image by photographing the part based on the parameters of the shooting conditions when the reference image was captured.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the claims.

1. Information processing device (image processing server)
REFERENCE SIGNS LIST 11 Control unit 12 Storage unit 13 Communication unit 14 Input unit 15 Display unit 16 Reading unit 17 Large-capacity storage unit 171 Object recognition model 172 Damage recognition model 173 Wear identification model 71a Area candidate extraction unit 71b Classification unit 1a Portable storage medium 1b Semiconductor memory 1P Control program 2 Photographing device 2a Photographing device 2b Photographing device 3 Information processing device (parts server)
31 Control unit 32 Storage unit 33 Communication unit 34 Input unit 35 Display unit 36 Reading unit 37 Large-capacity storage unit 371 Part-specific information DB
3a Portable storage medium 3b Semiconductor memory 3P Control program 4 Database device (DB server)
41 Control unit 42 Storage unit 43 Communication unit 44 Reading unit 45 Large-capacity storage unit 451 Product management DB
452 Parts Details DB
453 Parts Damage Information DB
454 Matching result DB
4a Portable storage medium 4b Semiconductor memory 4P Control program 5 Information processing device (factory server)
51 Control unit 52 Storage unit 53 Communication unit 54 Input unit 55 Display unit 56 Reading unit 57 Large-capacity storage unit 571 Product DB
5a Portable storage medium 5b Semiconductor memory 5P Control program 6 Information processing device (deletion server)
61 Control unit 62 Storage unit 63 Communication unit 64 Input unit 65 Display unit 66 Reading unit 67 Large-capacity storage unit 671 Parts management DB
672 Production planning DB
68 Clock unit 6a Portable storage medium 6b Semiconductor memory 6P Control program

Claims (17)

Acquire a part image of a part including a plurality of feature point objects;
Recognizing each object in the acquired part image;
Extracting a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects;
Acquire part-specific information corresponding to each of the extracted area images;
determining a number of matches of part specific information corresponding to each of the area images;
A program for causing a computer to execute a process of authenticating the part based on the number of matches of the part unique information and a predetermined threshold value. inputting the acquired part image into an image recognition model that has been trained to identify the position and type of an object in the part image when the part image is input, and outputting the position and type of the object in the part image;
Refer to a storage unit in which offset positions are stored for each type of object, read out the offset position corresponding to the output type,
The program according to claim 1 , further comprising: extracting an area image corresponding to the offset position read from the output position of the object. Obtaining a part ID corresponding to the part unique information;
The program according to claim 1 or 2, which causes the program to execute a process of outputting the acquired part ID and the part unique information. Allocating a part ID to the part unique information;
The program according to claim 1 or 2, which causes the program to execute a process of outputting the assigned part ID and the part unique information. Acquire a part ID corresponding to part unique information of the region image;
Acquire a product ID of a product using a part corresponding to the acquired part ID;
5. The program according to claim 1, further comprising: outputting the acquired product ID to a database device in association with the part ID. The program according to claim 1 , further comprising: acquiring a part image of the part photographed after a light for illuminating the part is turned on. The program according to claim 1 , further comprising: when a distortion or tilt is detected in the part image, correcting the offset position or the part image in accordance with the distortion or tilt. Acquire a first photographed image photographed by a first photographing device;
determining whether the part image is present in the acquired first captured image;
When it is determined that the part image exists, a second photographing device having a higher magnification than the first photographing device is used to obtain a second photographing image that is a part of the part image;
The program according to claim 2 , further comprising: inputting the acquired second captured image into the image recognition model. 9. The program according to claim 1, further comprising: a process for outputting, to a database device, a storage location and a storage period until the part is attached to a product, in association with a part ID of the part. 10. The program according to claim 1, further comprising: determining whether or not a part to be verified has been replaced or a part state by verifying the part to be verified with registered parts. The parts are distributed with multiple objects,
acquiring a part image including at least two objects among the plurality of objects;
Recognizing each object in the acquired part image;
Extracting an area image of a predetermined size at a predetermined offset position from each recognized object;
Acquire part-specific information corresponding to any one of the extracted multiple area images;
Allocating a part ID to the acquired part unique information;
The program according to claim 1 , further comprising: outputting the assigned part ID and the part specific information. the part is a cylindrical object, and a plurality of objects are distributed on an end surface of the part;
The program according to claim 1 , further comprising: acquiring a photographed image of the end face of the cylinder including one or more objects. Extracting a plurality of area images of a predetermined size at a predetermined offset position from an object in the photographed image of the tube end surface;
The program according to claim 12 , further comprising: executing a process of authenticating the part based on part specific information corresponding to each of the extracted area images. A predetermined allowable range is obtained according to parameters of the photographing conditions when the reference image including the object as the feature point was photographed;
The program according to claim 1 , further comprising: outputting an instruction to photograph the part when it is determined that a parameter of a part image to be photographed including the object falls within the predetermined allowable range. Acquire a part image of a part including a plurality of feature point objects;
Recognizing each object in the acquired part image;
Extracting a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects;
Acquire part-specific information corresponding to each of the extracted area images;
determining a number of matches of part specific information corresponding to each of the area images;
The information processing method causes a computer to execute a process of authenticating the part based on the number of matches of the part unique information and a predetermined threshold value. a first acquisition unit that acquires a part image of a part including objects that are feature points;
a recognition unit that recognizes each object in the acquired part image;
an extracting unit that extracts a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects;
a second acquisition unit that acquires part specific information corresponding to each of the extracted area images;
a determination unit for determining a number of matches of part unique information corresponding to each of the area images;
and an authentication unit that authenticates the part based on the number of matches of the part unique information and a predetermined threshold value. a first acquisition unit that acquires a part image of a part including objects that are feature points;
a recognition unit that recognizes each object in the acquired part image;
an extracting unit that extracts a plurality of area images of a predetermined size at a predetermined offset position from each of the recognized objects;
a second acquisition unit that acquires part specific information corresponding to each of the extracted area images;
a determination unit for determining a number of matches of part unique information corresponding to each of the area images;
an authentication unit that authenticates the part based on the number of matches of the part unique information and a predetermined threshold value;
an output unit that outputs the acquired part specific information;
The recognition unit recognizes damage to a part in the part image acquired by the first acquisition unit,
The information processing device according to claim 1, wherein the output unit outputs information regarding damage to a part recognized by the recognition unit.

Images