JP7764537B2 - Computer-implemented method and tool for optical quality control of intermediate or final products of a production facility and production facility control device - Google Patents

Description

The present invention relates to a computer-implemented method for optical quality control of intermediate or final products of a production facility according to the generic concept of claim 1, a computer-implemented tool for optical quality control of intermediate or final products of a production facility according to the generic concept of claim 6, and a production facility control device according to the generic concept of claim 11.

No matter in what field or domain the production process of a production facility is carried out with the corresponding production facility, it is never without defects and often needs to be ensured by optical quality control. This applies in particular to automated production processes, where optical quality control involves acquiring an image of the product or part of the product and calculating, based on this image, whether all characteristics meet the requirements. Furthermore, setting up quality control is associated with engineering costs that must be adjusted to new products at minimal cost, which is particularly disadvantageous in autonomous, automated production facilities in the industrial sector.

A typical example of such a production process is the assembly of products in the industrial domain using robotic or automation systems, where versatile mobile machines are used to perform processing, service and/or manufacturing tasks for the assembly, and after a joining process, for example, it is necessary to check whether the parts to be installed are correctly positioned and fully engaged.

For this purpose, it is known to provide tools to the commissioning engineers of production facilities, in particular robotic systems, to define quality rules as simply as possible. For example, certain optical features in optically acquired images, such as circles, lines, patterns, etc., must be in a specific position or have specific dimensions. The commissioning engineer then checks these rules using several example images and adjusts the parameters until the desired reliability is achieved.

The problem underlying the present invention is to identify a computer-implemented method and tool, as well as a production facility control device, for optical quality control of intermediate or final products in a production facility, which reduces the cost of establishing quality control in the face of constantly changing conditions and relationships in the production facility, so that quality control is performed by adaptively adjusting the use of the production facility accordingly.

This problem is solved by the features recited in the characterizing part of claim 1, based on the computer-implemented method defined in the generic concept of claim 1.

The present problem is also solved by the features described in the characterizing part of claim 6, based on the computer-implemented tool defined in the generic concept of claim 6.

Furthermore, this problem is solved by the features described in the characterizing part of claim 11, based on the production equipment control device defined in the generic concept of claim 11.

The idea underlying the inventions described in independent claims 1, 6 and 11 is to use product images of a production facility acquired by an image acquisition device for given internal and external parameters for optical quality control of intermediate or final products of the production facility, and to use digital twin data of a digital twin of the production facility (PA), the digital twin being synchronized with the production facility during runtime of the production facility; (i) rendering a synthetic simulation image based on the digital twin data, the rendered synthetic simulation image being based on the same internal and external parameters as when the product image was acquired; (ii) transferring the product image from a real domain to an artificial domain using trained domain adaptation, whereby a synthetic product image is generated from the product image based on the domain transfer parameters acquired by training; (iii) comparing the synthetic product image with the synthetic simulation image using a comparison operator; and (iv) outputting a comparison result for qualitatively evaluating the product.

Compared to traditional rule-based approaches, the present invention pursues a fundamentally different approach that offers better scaling potential.

Here, one essential aspect of the present invention is to use domain transformation from the real image domain to the artificial image domain for quality control, thereby achieving easier image comparison to detect image differences.

In one advantageous development of the invention as defined in claims 2 and 7, domain adaptation (DA) is implemented as a "machine learning" model according to the principles of "Generative Adversarial Networks (GAN)," and for data generation, two competing artificial neural networks, called a generative network and a discriminative network, are used, the generative network generating artificial data, which the discriminative network checks against real data, e.g., obtained from images, and the two networks are logically and mathematically linked to each other in such a way that the artificial data generated by the generative network looks increasingly realistic until, ultimately, the discriminative network can no longer distinguish real data from real data.

Also advantageously, according to claims 3 and 8, the trained domain adaptation with domain transfer parameters is carried out by the following steps "S1" and "S2":
Step "S1": generating a dataset based on a number "n" of image pairs consisting of acquired product images and associated synthetic simulation images, each for uniformly given internal and external parameters;
Step "S2": Based on the generated dataset, a two-stage training is carried out, which includes training the transfer of product image-related data to simulation image-related data using a learning method, for example, a "generative adversarial network (GAN)."

The present invention also advantageously comprises:
According to claims 4 and 9, the comparison operator is configured such that the comparison is performed pixel by pixel,
According to claims 5 and 10, the production facility is characterized in that it is a robotic or automation system with universally usable mobile machines for carrying out processing, service and/or manufacturing tasks.

The following describes the fundamental scenario underlying this invention.

The starting point for the scenario described here is the prerequisite that the details of the digital twin of the production facility are available at runtime. This digital twin:
- The geometry and/or texture of the objects, robots and production equipment components used - The runtime, i.e., the expected positions and poses of the objects, robots and production equipment components at a given point in time.

This prerequisite is generally met for autonomously operating production equipment or systems, but can also be met for or retrofitted to conventional production equipment or systems.

According to this or based on this, the inventive approach involves taking a realistic image of the part or product to be checked of the production equipment, e.g., an intermediate or final product, and transferring this image to a simulated image space using domain adaptation. The transferred image is then compared with a synthetically rendered image, generated using a digital twin. Now, a "synthetic-looking" image available in the same image space can be checked for differences using a comparison operator. Here, the output of the comparison result can be, for example, a quality result of "OK" or not "OK."
In that case, the quality results can be automatically taken into account in further processes, for example, production defective products (e.g. intermediate products or final products) can be automatically sorted out or support can be requested from the user of the production facility.

The approach of the present invention has two main steps:
1. Training for domain adaptation: In this step, we create the basis for transferring images from a real domain to an artificial domain. This is a part of domain adaptation. One possible exemplary approach for training for domain adaptation is the above-mentioned "generative adversarial network (GAN)" technique.

Training can be characterized as follows:
a. In a first step, for optical acquisition, a data set is acquired consisting of a real image and its associated synthetic image, each of the same perspective and the same parameters (internal and external) of the image acquisition device, e.g., camera.

- In this case, data acquisition and training can be carried out at the early stage of commissioning the production equipment and only need to be carried out once for the production equipment according to the variety of products (intermediate or final products) produced. In other words, new training is not required when products are changed.

- In this case, data acquisition is carried out jointly during the testing or commissioning phase. In this case, where quality control is required, realistic images are acquired and saved, as are camera parameters (internal and external parameters of the image acquisition device). Furthermore, as part of data acquisition for training, the digital twin is used to render related synthetic images based on the same perspective and with the same parameters, which are saved in the dataset.

b. Finally, a training step is performed in which the transfer of real data to synthetic data is learned. For this purpose, for example, the aforementioned "generative adversarial network (GAN)" technique can be used.

2. Production equipment operation:
a) Position the image capture device in a suitable position so that the product or part, intermediate product or final product is visible. The suitable position can be calculated by heuristics at run-time of the production facility. This applies especially to autonomous production facilities.
b) Obtain a product image.
c. Transforming realistic product images into the image domain of synthetic (simulated) images using domain adaptation, e.g., Generative Adversarial Networks (GAN) techniques.
d. Rendering a composite image captured with identical parameters (internal and external parameters of the image capture device) based on the same position in the digital twin.
e. Compare the images using a comparison operator, where a neural network may be used in one advantageous embodiment of the present invention, although traditional pixel-based approaches are also suitable.

Further advantages of the present invention will become apparent from the following description of an example embodiment of the invention based on one drawing.

FIG. 1 shows a production facility PA that uses optical quality control of intermediate or final products of the production facility, which can be used in various technical fields (technical domains) for product production using appropriate methods and techniques, for example in factories, production lines, large scale equipment, etc. FIG. 2 shows a production facility PA using optical quality control of intermediate or final products of the production facility, which can be used in various technical fields (technical domains) for product production using appropriate methods and techniques, for example in factories, production lines, large scale equipment, etc.

In a first variant of the invention for the optical quality control of intermediate or final products according to option "A", the production facility PA includes a production facility control device PAS, a database DB, an image acquisition device BEE, which is configured, for example, in the form and shape of a camera, and an output device AEH. The database DB and the image acquisition device BEE are connected to the production facility control device PAS for accessing the production facility control device PAS, while the output device AEH, like the database DB and the image acquisition device BEE, is either connected to the production facility control device PAS for accessing the production facility control device PAS in the first embodiment according to option "I", or is included in the production facility control device PAS in the second embodiment according to option "II".

As an alternative to the first variant of the invention according to option "A", in a second variant of the invention for optical quality control of intermediate or final products according to option "B", the production equipment PA, the production equipment control device PAS, the database DB and the image acquisition device BEE are not "integrated" in the production equipment PA, but rather they all function as separate devices, and the production equipment control device PAS is connected for access to the production equipment PA, the database DB and the image acquisition device BEE, while the output device AEH is either included in the production equipment PA for access by the production equipment control device PAS in the first embodiment according to option "I", or is included in the production equipment control device PAS in the second embodiment according to option "II".

Other variations between these two "extreme" variations of the invention are also possible (not explicitly shown), in which either only the database DB, the image acquisition device BEE or the production equipment control device PAS are included in the production equipment PA, or two of the above devices are each included in the production equipment PA.

In the two illustrated variants according to options "A" and "B", the production equipment control device PAS is preferably a control device STE for a robotic or automation system having universally usable mobile machines for performing processing, service and/or manufacturing tasks.

Regarding the variant illustrated by option "B", the production equipment control device PAS can alternatively be a general personal computer or control device.

The following explanations regarding the description of the figures apply to both the illustrated variants of the invention for optical quality control of intermediate or final products according to options "A" and "B".

To this end,
The image acquisition device BEE acquires both product images PB of the production equipment PA, which may relate to an intermediate product or a final product, for given internal and external parameters of the image acquisition device BEE, and product images PB1, ..., PBn, for uniformly given internal and external parameters of the image acquisition device BEE, to generate a dataset based on a plurality of "n" image pairs.
- The database DB stores the digital twin data DZD of the digital twin DZ of the production equipment PA, and when the production equipment PA is executed, the digital twin DZ is synchronized with the production equipment PA.
A computer-implemented tool CIW is used, which is preferably a computer program product CPP configured as an app and which can be loaded into the production facility control device PAS for the optical quality control of intermediate or final products of the production facility PA.

The computer-implemented tool CIW comprises a non-volatile readable memory SP in which processor-readable control program instructions of a program module PGM for optical quality control are stored, and a processor PZ connected to the memory SP, which executes the control program instructions of the program module PGM for optical quality control of intermediate or final products of the production facility PA.
Includes.

For this purpose, the computer implemented tool CIW:
- product images PB of the production equipment PA acquired by the image acquisition device BEE for given internal and external parameters, and product images PB1, ..., PBn acquired by the image acquisition device BEE for given internal and external parameters to generate a data set based on a number "n" of image pairs;
- Digital twin data DZD of the digital twin DZ of the production equipment PA stored in the database DB,
The digital twin DZ is synchronized with the production equipment PA when the production equipment PA is executed.

These data are requested by the processor PZ through access as input data when loading the computer-implemented tool CIW into the production equipment control device PAS, and are subsequently either collected or provided.

The program module PGM of the computer-implemented tool CIW and the processor PZ of the computer-implemented tool C, which executes the control program instructions of the program module PGM for optical quality control,
Rendering a synthetic simulation image SB _syn based on the digital twin data DZD, wherein the rendered synthetic simulation image SB _syn is based on the same internal and external parameters as those used to acquire the product image;
Transferring product images PB from a real domain to an artificial domain using the trained domain adaptation algorithm,
Domain adaptation is preferably implemented, for example, according to the principles of "generative adversarial networks (GAN)" as a "machine learning" model, in which two competing artificial neural networks, called a generative network and a discriminative network, are used for data generation, the generative network generating artificial data, which the discriminative network checks against real data, e.g., obtained from images, and the two networks are logically and mathematically linked to each other in such a way that the artificial data generated by the generative network looks more and more realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the real data;
The domain adaptation DA has domain transfer parameters DTP obtained by training trn, and the trained domain adaptation DA having the domain transfer parameters DTP preferably performs the following steps "S1" and "S2", namely:
"S1": generating a dataset based on a number "n" of image pairs consisting of acquired product images PB1, ..., PBn and their associated synthetic simulation images SB1 _syn , ..., SBn _syn for uniformly given intrinsic and extrinsic parameters;
"S2": A step is carried out in a two-stage training trn, which comprises training the transfer of product image-related data to simulation image-related data by a learning method, for example, a "generative adversarial network (GAN)", based on the generated dataset;
Trained using
After training, the trained domain adaptation algorithm generates a synthetic product image _PB from the product image PB according to the domain transfer parameter DTP, thereby generating an image pair in the artificial image space consisting of the synthetic simulation image _SB and the synthetic product image _PB for comparison purposes;
Steps and
- comparing vgl the composite product image PB _syn with the composite simulation image SB _syn using a comparison operator VO, which is for example preferably configured such that the comparison is performed pixel by pixel;
- outputting a comparison result VGE that qualitatively evaluates the product, said comparison result VGE being preferably output by an output device AEH of the production facility PA or of a production facility control device PAS of said production facility PA;
is configured to run.

Claims (12)

1. A computer-implemented method for optical quality control of intermediate or final products of a production facility, comprising:
Product images (PB) of a production facility (PA) are used, acquired by an image acquisition device (BEE) for given internal and external parameters,
Digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is used, and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running.
1. A computer-implemented method comprising:
a) rendering (rdn) a synthetic simulation image (SB _syn ) based on said digital twin data (DZD), said rendered synthetic simulation image (SB _syn ) being based on said internal and external parameters identical to those used when a product image was acquired;
b) transferring (trf) said product image (PB) from a real domain to an artificial domain using a trained (trn) domain adaptation (DA), said domain adaptation (DA) having domain transfer parameters (DTP) obtained by said training (trn), said domain adaptation (DA) generating a synthetic product image (PB _syn ) from said product image (PB) according to said domain transfer parameters (DTP), thereby generating an image pair in the artificial image space consisting of said synthetic simulation image (SB _syn ) and said synthetic product image (PB _syn ) for comparison purposes;
c) comparing (vgl) said synthetic product image (PB _syn ) with said synthetic simulation image (SB _syn ) using a comparison operator (VO);
d) outputting (asg) a comparison result (VGE) for qualitatively evaluating the product, in particular by an output device (AEH) of said production facility (PA) or by a production facility control device (PAS) of said production facility (PA);
1. A computer-implemented method comprising: The domain adaptation (DA) is implemented as a "machine learning" model according to the principles of "generative adversarial networks (GAN)." For data generation, two competing artificial neural networks, called a generative network and a discriminative network, are used. The generative network generates artificial data, which the discriminative network checks based on real data, e.g., obtained from images.
the two networks are logically and mathematically linked together such that the artificial data generated by the generative network appears increasingly realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the genuine data.
10. The computer-implemented method of claim 1. The trained (trn) domain adaptation (DA) with the domain transfer parameters (DTP) is
"S1": generating a dataset based on a number "n" of image pairs consisting of acquired product images (PB1, ..., PBn) and their associated synthetic simulation images (SB1 _syn , ..., SBn _syn ) for uniformly given intrinsic and extrinsic parameters;
"S2": Training the transfer of product image-related data to simulation image-related data by a learning method, such as a "generative adversarial network (GAN)", based on the generated dataset;
2. The computer-implemented method of claim 1, wherein the method is performed in a two-stage training (trn) comprising: The computer-implemented method of claim 1, wherein the comparison operator (VO) is configured such that the comparison is performed pixel by pixel. The computer-implemented method of claim 1, wherein the production equipment (PA) is a robotic or automation system having universally usable mobile machines for performing processing, service, and/or manufacturing tasks. A computer-implemented tool (CIW) for the optical quality control of intermediate or final products of a production facility, in particular a computer program product (CPP) configured as an app, which for said quality control comprises:
Product images (PB) of a production facility (PA) are used, acquired by an image acquisition device (BEE) for given internal and external parameters,
Digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is used, and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running.
In computer implemented tools (CIW),
a non-volatile readable memory (SP) in which processor-readable control program instructions of a program module (PGM) for optical quality control are stored; and a processor (PZ) connected to said memory (SP) for executing said control program instructions of said program module (PGM) for optical quality control of said intermediate or final products of a production facility;
When loaded by a computer system including
a) a synthetic simulation image (SB _syn ) based on the digital twin data (DZD) is rendered (rdn), and the rendered synthetic simulation image (SB _syn ) is based on the same internal and external parameters as when the product image was acquired;
b) the product image (PB) is transferred (trf) from a real domain to an artificial domain using a trained (trn) domain adaptation (DA), the domain adaptation (DA) having domain transfer parameters (DTP) obtained by the training (trn), and the domain adaptation (DA) generates a synthetic product image (PB _syn ) from the product image (PB) according to the domain transfer parameters (DTP), thereby generating an image pair consisting of the synthetic simulation image (SB _syn ) and the synthetic product image (PB _syn ) in the artificial image space for comparison purposes;
c) the composite product image (PB _syn ) is compared (vgl) with the composite simulation image (SB _syn ) using a comparison operator (VO);
d) the comparison result (VGE) for qualitatively evaluating the product is output (asg) in particular by an output device (AEH) of the production facility (PA) or a production facility control device (PAS) of the production facility (PA),
expressing a computer system in which said program module (PGM) is configured and in which said processor (PZ) is configured to execute said control program instructions of said program module (PGM) for optical quality control,
A computer implemented tool (CIW) comprising: The processor (PZ) and the program module (PGM) for optical quality control are configured such that the domain adaptation is implemented according to the principles of "generative adversarial networks (GAN)" as a "machine learning" model, and two competing artificial neural networks, called a generative network and a discriminative network, are used for data generation, the generative network generating artificial data and the discriminative network checking the artificial data based on real data, e.g., obtained from images,
the two networks are logically and mathematically linked together such that the artificial data generated by the generative network appears increasingly realistic until, eventually, the discriminative network can no longer distinguish the realistic data from the genuine data.
A computer implemented tool (CIW) according to claim 6, characterized in that it The processor (PZ) and the program module (PGM) for optical quality control train (trn) the domain adaptation (DA) with the domain transfer parameter (DTP) by the following steps "S1" and "S2":
Step "S1": generating a dataset based on a number "n" of image pairs consisting of acquired product images (PB1, ..., PBn) and their associated synthetic simulation images (SB1 _syn , ..., SBn _syn ) for uniformly given intrinsic and extrinsic parameters;
Step "S2": training the transfer of product image-related data to simulation image-related data by a learning method, for example, a "generative adversarial network (GAN)", based on the generated dataset;
7. A computer implemented tool (CIW) according to claim 6, characterized in that it is adapted to be performed in a two-stage training (trn) comprising: The computer-implemented tool (CIW) of claim 6, wherein the processor (PZ), the program module (PGM), and the comparison operator (VO) for optical quality control are configured so that the comparison is performed pixel by pixel. The computer-implemented tool (CIW) of claim 6, characterized in that the production equipment (PA) is a robotic or automation system having universally usable mobile machines for performing processing tasks, service tasks and/or manufacturing tasks. A production installation control system (PAS) for optical quality control of intermediate or final products of a production installation (PA), comprising:
an image acquisition device (BEE) for acquiring product images (PB) of said production equipment (PA) for given internal and external parameters is either a component of said production equipment (PA) and thus connected to said production equipment control device (PAS) or is assigned to said production equipment (PA) and thus connected to said production equipment control device (PAS);
A database (DB) storing digital twin data (DZD) of the digital twin (DZ) of the production equipment (PA) is assigned to the production equipment (PA) and is thus connected to the production equipment control device (PAS), and the digital twin (DZ) is synchronized with the production equipment (PA) when the production equipment (PA) is running;
In the production equipment control system (PAS),
7. A production installation control system (PAS) characterized by a computer implemented tool (CIW) according to claim 6, which is loadable into said production installation control system (PAS) for carrying out the method according to claim 1. The production equipment control device (PAS) according to claim 11, characterized in that it is a control device (STE) for a robotic system or automation system having universally usable mobile machines for performing processing tasks, service tasks and/or manufacturing tasks.