JP7225068B2 - Defect inspection device for printed matter, defect inspection method and program, and printing system - Google Patents

Description

The present invention relates to a defect inspection apparatus for printed matter, a defect inspection method and program, and a printing system, and more particularly to a technique for acquiring defect information from an image using a deep learning model.

2. Description of the Related Art A defect inspection apparatus is widely used for inspecting defects of an object by comparing image data obtained by imaging the object using a scanner and reference data serving as a reference. As an application of the defect inspection method based on this technique, there is known a print defect inspection apparatus for inspecting print defects such as streaks and ink defects on printed materials printed by a printing apparatus.

Japanese Unexamined Patent Application Publication No. 2002-100002 describes a technique for inspecting for streaks occurring on a printed matter from an image obtained by imaging the printed matter during printing using an imaging device. In the technique described in Patent Document 1, a captured image is divided into a plurality of local areas in a first direction, which is the line head relative scanning direction, and signal processing is performed for each local area.

JP 2017-181094 A

In the conventional defect inspection of printed matter, a method of acquiring in advance reference data under the same printing conditions and comparing the reference data with captured image data during printing is widely used. However, there is a problem that such defect inspection of printed matter cannot be applied to printing such as variable printing in which patterns change each time, or to the first printed matter after changing printing conditions.

In addition, when comparing imaging data under the same printing conditions, this defect inspection of printed matter can be compared because the basic data characteristics are the same. It is difficult to make a comparison when imaging data with different printing conditions such as profiles or print data based on a print source image is used as the reference data because the data characteristics are different.

If the printing conditions are different, for example, if the paper is different, the luminance value of the entire data will be different. As an example, when the reference data is obtained using glossy paper, and the same pattern is printed again on high-quality paper for inspection, there is a large difference between the acquired reference data for glossy paper and the imaging data for high-quality paper. If the comparison is made as it is, it is necessary to lower the inspection performance. In addition, in order to carry out the inspection without lowering the inspection performance, it is necessary to reacquire the reference data using high-quality paper again.

When print data based on a print source image is used as reference data, a 4-channel color space of CMYK (cyan, magenta, yellow, and black) and a 3-channel color space of RGB (red, green, and blue) are used. Differences such as different data formats occur between the two. Therefore, preprocessing is required to match the characteristics. Regarding the difference between CMYK and RGB, the conventional technique uses a method of matching color characteristics by creating a table for matching color profiles between print data and imaging data. Profiles differ according to different printing conditions, and there are a wide variety of conditions such as differences in printing devices and paper, so creating tables corresponding to each of them is time-consuming and complicated. .

For such problems, it is conceivable to use deep learning, which has been widely used in recent years, for automating the extraction of parameters and feature quantities. Deep learning is one of the machine learning techniques, and is known to have much higher performance than conventional machine learning techniques in the field of image recognition.

In the technique described in Patent Document 1, in order to detect streaks extending in the first direction, an image is divided while allowing overlap in the first direction. The technique described in Patent Literature 1 is based on the premise that the same processing is performed on each pixel of the entire inspection image, and therefore division processing in the second direction different from the first direction is unnecessary.

On the other hand, when a deep learning model is used for inspection processing, learning is generally performed with images that are smaller than the actual images. Therefore, during the inspection process, it is necessary to divide the image data into sizes to be input to the model, including the second direction. Here, the reliability of the inspection performance improves as the overlap amount at the time of division increases, but the number of processing times increases proportionally, so there is a trade-off relationship with the processing time. Therefore, there is a problem that it is necessary to divide the image data by an appropriate amount of overlap.

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances. The purpose is to provide a system.

In order to achieve the above object, one aspect of the apparatus for inspecting defects in printed matter includes an imaging data acquisition unit that acquires imaging data based on an imaged image of the printed matter, and a print based on the original image of the printed matter. A print data acquisition unit that acquires data, an inspection processing unit that uses at least one deep learning model, receives imaging data and print data as input, and outputs defect information on printed matter, and a convolution layer of the deep learning model. a preprocessing unit that performs division processing for dividing imaging data and print data by overlapping the number of pixels according to the filter size, the pooling size of the pooling layer, the number of convolution layers and the number of pooling layers, and an inspection processing unit. is a printed matter defect inspection apparatus that inputs divided imaging data and print data.

According to this aspect, the imaging data and the print data are divided by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of convolutional layers and pooling layers of the deep learning model. Therefore, it is possible to divide image data and print data by an appropriate amount of overlap when inspecting printed matter using deep learning.

The number of pixels is n, where n is the total number of convolution layers and pooling layers, and Sk is the filter size of the k-th convolution layer after performing j pooling with a pooling size of Sj.

It is preferable that it is above. Thereby, the imaging data and the print data can be divided by an appropriate overlap amount.

Note that the k-th convolutional layer is the k-th convolutional layer when the convolutional layers and the pooling layers are the 1st to n-th convolutional layers in the order of processing, and 1≦k≦n. Let Sj be the pooling size of a pooling layer that performs a pooling process of size Sj×Sj, and let Si be the pooling size of the i-th pooling layer out of j times. Furthermore, let Sk be the filter size of a convolution layer that performs convolution processing of size Sk×Sk.

Also, "floor" is the floor function. A floor function is a function that outputs the largest integer among integers smaller than an input value.

The number of pixels when the imaging data is divided is n, which is the total number of convolution layers and pooling layers related to the imaging data, and the filter size of the k-th convolution layer after j times of pooling with a pooling size of Sj. Calculate as Sk. Also, the number of pixels when the print data is divided is n, which is the total number of convolution layers and pooling layers related to the print data, and the filter of the k-th convolution layer after j times of pooling with a pooling size of Sj. Calculate the size as Sk.

Neighboring defect information group consisting of N adjacent defect information, which is a natural number of 2 or more, existing spatially adjacent to the defect information of interest or existing in the printing direction of the printed matter with respect to the defect information group output by the inspection processing unit. It is preferable to provide a statistical processing unit that determines a defect by statistically processing the . Thereby, a defect can be judged appropriately.

Note that the defect information of interest and neighboring defect information spatially adjacent to each other are the defect information output from the image data having the defect information of interest and the image data spatially adjacent to the image data having the defect information of interest among the divided image data. may contain. Accordingly, when the same defect exists across a plurality of divided adjacent image data, the defect can be determined appropriately.

Further, the defect information of interest and the neighboring defect information existing in the printing direction of the printed matter may include the defect information output from the imaging data having the defect information of interest among the divided imaging data and the imaging data existing in the printing direction. good. The printing direction is the direction of relative movement between the substrate and the ink applying section when the printed material is printed by the printing apparatus. The ink applicator is, for example, a printing element that applies ink to a substrate.

Depending on the printing method of the printing device, areas that are not spatially adjacent may be printed by the same printing element. For example, in the case of a single-pass printing device, since the same printing element is used to print the area along the transport direction (corresponding to the printing direction) of the substrate, the defect information of interest and the defect information existing in the transport direction are different. It may be a defect printed with the same printing element. Therefore, by statistically processing the defect information of interest and a group of adjacent defect information existing in the printing direction of the printed matter, it is possible to detect defects caused by the same printing element in a plurality of divided image data existing in the printing direction. However, the defect can be properly determined.

The defect information preferably includes the presence or absence of defects. As a result, the printed matter can be properly inspected.

It is preferable that the statistical processing unit includes statistical processing for determining that a defect exists when the number of pieces of defect information indicating that there is a defect in the adjacent defect information group is equal to or greater than M, which is a natural number equal to or less than N. Thereby, a defect can be judged appropriately.

Preferably M and N are equal. Thereby, a defect can be judged appropriately.

At least one of M and N is preferably a value according to the type of base material of the printed matter. Thereby, a defect can be judged appropriately.

At least one of M and N is preferably a value corresponding to the type of net of the printed matter. Thereby, a defect can be judged appropriately.

It is preferable that at least one of M and N has a different value between a printing element in which an abnormality has occurred and a printing element in which an abnormality has not occurred in the printing apparatus that printed the printed matter. Thereby, a defect can be judged appropriately.

At least one of M and N may differ in value between a portion compensated for by the compensation function for compensating for an abnormality in the printing element of the printing device that printed the printed matter by correcting the print data and a portion not compensated for. preferable. Thereby, a defect can be judged appropriately.

The defect information preferably includes the perceived strength value of the defect. As a result, the printed matter can be properly inspected.

It is preferable that the statistical processing section include statistical processing for judging whether or not the defect is a defect based on the average value of the recognition intensity values of the adjacent defect information group. Thereby, a defect can be judged appropriately.

The defect information preferably includes the location of the defect. As a result, the printed matter can be properly inspected.

Preferably, the preprocessing unit performs processing for reducing the data size in at least one direction on the imaging data and the print data. As a result, the load on the inspection processing unit can be reduced.

It is preferable that the preprocessing section reduces the data size at least with respect to the conveying direction of the base material at the time of printing. As a result, the data size can be appropriately reduced.

The deep learning model preferably receives imaging data and print data, and outputs at least one of the presence/absence of a defect, the position of a defect, and the recognition strength value of a defect. As a result, the printed matter can be properly inspected.

The deep learning model includes a first network including a first input layer with imaging data as an input, a first intermediate layer with the first input layer as an input, and a second network with print data as an input. a second network including an input layer of and a second intermediate layer whose input is the second input layer; and a first output layer whose output is defect information, wherein the first output layer Preferably, the first network and the second network are combined at least once before. As a result, it is possible to appropriately output the defect information of the printed matter.

The preprocessing unit divides the imaging data by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of convolutional layers and pooling layers of the first intermediate layer, and divides the captured data into the second intermediate layer. It is preferable to divide the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of convolution layers and pooling layers. As a result, the imaging data and the print data can be divided by overlapping by an appropriate number of pixels.

In order to achieve the above object, one aspect of a printing system includes: a printing device that prints a printed matter with a printing element based on print data; an imaging device that captures the printed matter and obtains a captured image; and a defect inspection device for printed matter.

According to this aspect, the imaging data and the print data are divided by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of convolutional layers and pooling layers of the deep learning model. Therefore, it is possible to divide image data and print data by an appropriate amount of overlap when inspecting printed matter using deep learning.

In order to achieve the above object, one aspect of the apparatus for inspecting defects in printed matter includes an imaging data acquisition step of acquiring imaging data based on an imaged image of the printed matter, and printing based on the original image of the printed matter. A print data acquisition step for acquiring data, an inspection processing step for using at least one deep learning model, imaging data and print data as input, and outputting defect information for printed matter, and a convolution layer of the deep learning model. a pre-processing step of performing a division process of dividing the imaging data and the print data by overlapping the number of pixels corresponding to the filter size, the pooling size of the pooling layer, the number of convolution layers and the number of pooling layers; is a print defect inspection method in which divided imaging data and print data are input.

According to this aspect, the imaging data and the print data are divided by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of convolutional layers and pooling layers of the deep learning model. Therefore, it is possible to divide image data and print data by an appropriate amount of overlap when inspecting printed matter using deep learning.

A program for causing a computer to execute the defect inspection method for printed matter is also included in this aspect. A non-temporary recording medium storing this program is also included in this aspect.

According to the present invention, it is possible to divide the inspection image by an appropriate amount of overlap and inspect the printed matter using deep learning.

FIG. 1 is a block diagram showing the configuration of a defect inspection apparatus. FIG. 2 is a functional block diagram showing main functions of the deep learning model. FIG. 3 is a flow chart showing processing of a defect inspection method for printed matter by the defect inspection apparatus. FIG. 4 is a diagram for explaining an area where data obtained by convolution processing is unreliable. FIG. 5 is a diagram for explaining an area where data obtained by pooling processing is unreliable. FIG. 6 is a diagram for explaining the division processing of imaging data. FIG. 7 is a diagram for explaining the division processing of imaging data. FIG. 8 is a diagram showing another aspect of coupling between the first network and the second network of the deep learning model. FIG. 9 is a diagram showing another aspect of coupling between the first network and the second network of the deep learning model. FIG. 10 is a diagram showing another aspect of the connection between the first network and the second network of the deep learning model. FIG. 11 is an overall configuration diagram of an inkjet printer to which the defect inspection device is applied. FIG. 12 is a block diagram showing the internal configuration of the inkjet printer.

Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

<Defect inspection device>
A defect inspection apparatus 10 according to the present embodiment is an apparatus that outputs defect information of a printed matter to be inspected. FIG. 1 is a block diagram showing the configuration of the defect inspection apparatus 10. As shown in FIG. The defect inspection apparatus 10 includes an imaging data acquisition section 12 , a print data acquisition section 14 , a preprocessing section 16 , an inspection processing section 18 , a statistical processing section 20 and an output section 22 .

The imaged data acquisition unit 12 acquires imaged data based on an imaged image of a printed matter printed by a printing device (not shown). The imaging data is, for example, bitmap data of a captured image. The print data acquisition unit 14 acquires print data based on the print source image of the printed matter. The print data is, for example, RIP (Raster Image Processor) data of a print source image.

The preprocessing unit 16 preprocesses the input image data. The preprocessing includes processing for reducing the data size in at least one direction for the imaging data and print data.

Also, the pre-processing includes division processing for dividing imaging data and print data. The preprocessing unit 16 includes an imaging data division processing unit 24 and a print data division processing unit 26 . The imaging data division processing unit 24 performs division processing for dividing the imaging data. The print data division processing unit 26 performs division processing for dividing the print data.

The inspection processing unit 18 receives imaging data and print data, and performs inspection processing for outputting defect information of printed matter. Here, the inspection processing unit 18 receives as input areas corresponding to the imaging data divided by the imaging data division processing unit 24 and the print data divided by the print data division processing unit 26, and divides the printed matter. output the defect information for each area.

The inspection processing unit 18 outputs defect information of the printed matter. The inspection processing unit 18 includes a deep learning model 28 . The deep learning model 28 is a trained model that inputs imaging data and print data and outputs defect information of printed matter. The deep learning model 28 has a multiple layer structure and holds multiple weight parameters. The deep learning model 28 can change from an unlearned model to a learned model by updating the weight parameter from the initial value to the optimum value. The defect information includes at least one of the presence/absence of a defect, the position of the defect, and the recognition strength value of the defect.

FIG. 2 is a functional block diagram showing major functions of the deep learning model 28. As shown in FIG. Deep learning model 28 comprises a first network 30 , a second network 32 and a first output layer 34 .

First network 30 includes a first input layer 36 and a first hidden layer 38 . The first input layer 36 receives imaging data. The first intermediate layer 38 receives the first input layer 36 as input. The first input layer 36, the first intermediate layer 38, and the first output layer 34 each have a structure in which a plurality of nodes are connected by edges.

The first intermediate layer 38 is a layer for extracting features from an image (captured data in this case) input from the first input layer 36 . The first intermediate layer 38 includes multiple sets of convolutional layers 40 and pooling layers 42 . In the example shown in FIG. 2, first intermediate layer 38 includes convolutional layer 40A, pooling layer 42A, convolutional layer 40B, pooling layer 42B, .

The convolution layer 40 performs a convolution operation using a filter of a predetermined filter size on nearby nodes in the previous layer to obtain a feature map. The pooling layer 42 reduces the feature map output from the convolution layer 40 by a predetermined pooling size to obtain a new feature map. The convolution layer 40 plays a role of feature extraction such as edge extraction from an image, and the pooling layer 42 plays a role of providing robustness so that the extracted features are not affected by translation or the like. The first intermediate layer 38 is not limited to the case where the convolutional layer 40 and the pooling layer 42 are set as one set, and the convolutional layer 40 may be continuous, and the normalization layer may also be included. good too. The first intermediate layer 38 may have a fully connected layer at the end that connects all of the nodes of the immediately preceding layer.

The second network 32 includes a second input layer 46 with print data as input and a second intermediate layer 48 with second input layer 46 as input. The second input layer 46, the second intermediate layer 48, and the first output layer 34 each have a structure in which a plurality of nodes are connected by edges.

A second intermediate layer 48 is a layer that extracts features from the print data input from the second input layer 46 . The second intermediate layer 48 includes a convolutional layer 50A, a pooling layer 52A, a convolutional layer 50B, a pooling layer 52B, . Convolutional layer 50 and pooling layer 52 are similar in configuration, operation, and function to convolutional layer 40 and pooling layer 42 . The second intermediate layer 48 is not limited to the case where the convolutional layer 50 and the pooling layer 52 are set as one set, and the convolutional layer 50 may be continuous, or a normalization layer may be included. . A second intermediate layer 48 may have a fully connected layer at the end that connects all of the nodes of the immediately preceding layer.

First network 30 and second network 32 are combined at least once prior to first output layer 34 .

The first output layer 34 is a layer that outputs recognition results including defect information of printed matter based on the features extracted by the first intermediate layer 38 and the second intermediate layer 48 .

For example, when the deep learning model 28 classifies the presence or absence of defects in printed matter, the defect information is output as two scores corresponding to "defect present" and "defect free". The sum of the two scores is 100%.

Returning to the description of FIG. 1 , the imaging data division processing unit 24 of the preprocessing unit 16 uses the filter size of the convolution layer 40 of the first network 30 of the deep learning model 28, the pooling size of the pooling layer 42, the convolution layer 40 and the pooling The imaging data is divided by overlapping by the number of pixels corresponding to the number of layers 42 . In addition, the print data division processing unit 26 divides the number of pixels according to the filter size of the convolution layer 50, the pooling size of the pooling layer 52, and the numbers of the convolution layers 50 and pooling layers 52 of the second network 32 of the deep learning model 28. Divide the print data by overlapping only. Details of the division processing will be described later.

The statistical processing unit 20 statistically processes the defect information for each divided area output from the inspection processing unit 18 to determine defects in the printed matter. Here, the statistical processing unit 20 statistically processes a neighboring defect information group consisting of N neighboring defect information, which is a natural number of 2 or more. For example, the statistical processing unit 20 determines that a defect exists when the number of pieces of defect information indicating that there is a defect in the adjacent defect information group is equal to or greater than M, which is a natural number equal to or less than N.

The statistical processing unit 20 statistically processes the target defect information and neighboring defect information spatially adjacent to the target defect information. The statistical processing unit 20 may statistically process the target defect information and neighboring defect information existing in the printing direction of the printed matter. The printing direction is the direction of relative movement between the substrate and the ink applying section when the printed material is printed by the printing device. The ink applying section is, for example, a printing element.

The output unit 22 outputs quality of printed matter based on the statistical processing result of the statistical processing unit 20 .

<Defect inspection method>
FIG. 3 is a flow chart showing the processing of the defect inspection method for printed matter by the defect inspection apparatus 10 . Here, an example will be described in which an imaging device is used to inspect for streak-like defects that occur in printed matter printed by a so-called single-pass printing apparatus. The defect inspection method for printed matter includes an imaging data acquisition step (step S1), a print data acquisition step (step S2), a preprocessing step (step S3), an inspection processing step (step S4), a statistical processing step (step S5) and.

In step S<b>1 , the captured data acquisition unit 12 acquires captured data based on a captured image of a printed matter printed by a printing device captured by an imaging device. The imaged data acquisition unit 12 may generate imaged data by performing predetermined processing on the acquired imaged image.

In step S2, the print data acquisition unit 14 acquires print data based on the print source image of the printed matter. The print data acquisition unit 14 may generate print data by performing predetermined processing on the acquired print source image.

In step S3, the preprocessing unit 16 performs preprocessing on the imaging data and the print data. Here, the imaging data division processing unit 24 divides the pixels according to the filter size of the convolution layer 40 of the first network 30 of the deep learning model 28, the pooling size of the pooling layer 42, and the numbers of the convolution layers 40 and pooling layers 42. The imaging data is divided by overlapping by the number. In addition, the print data division processing unit 26 divides the number of pixels according to the filter size of the convolution layer 50, the pooling size of the pooling layer 52, and the numbers of the convolution layers 50 and pooling layers 52 of the second network 32 of the deep learning model 28. Divide the print data by overlapping only.

The preprocessing unit 16 may reduce the data size in at least one direction of the imaging data and the print data before the division process. Here, the printed matter is printed by a single-pass printing device. A streak-like defect in the single-pass method is a defect extending in the transport direction during printing. Therefore, by averaging the imaging data in the conveying direction during printing, the data can be reduced while the streak information remains. Therefore, the preprocessing unit 16 can reduce the data size with respect to the conveying direction of the base material at least during printing.

In step S4, the inspection processing unit 18 uses the deep learning model 28 to obtain defect information for each divided area of the printed matter by inputting corresponding areas of the imaging data and the print data divided in step S3. Perform inspection processing for output.

In step S<b>5 , the statistical processing unit 20 determines defects by statistically processing the defect information group output from the inspection processing unit 18 . Here, the statistical processing unit 20 processes M or more continuous defects, which is a natural number equal to or less than N, in the conveying direction of the base material during printing for the target defect information and the N adjacent defect information groups output from the inspection processing unit 18. If it is determined that there is a defect, a complete match determination process is performed to determine that there is a defect. According to the complete match determination process, it is possible to distinguish between erroneous detection that occurs stochastically and correct detection when detection is performed correctly.

At least one of M and N is set to a value according to the type of substrate of the printed matter. Also, at least one of M and N may be set to a value according to the screen type of the printed matter. Furthermore, it is preferred that M and N are equal. In this case, the statistical processing unit 20 may acquire the base material type of the printed matter and the net type of the printed matter from an input unit (not shown).

It is preferable that at least one of M and N is set to a different value between a printing element where an abnormality has occurred and a printing element where an abnormality has not occurred in the printing device that printed the printed material. Also, at least one of M and N has a different value between a portion compensated and a portion not compensated for by a compensation function that compensates for an abnormality in the printing element of the printing device that printed the printed matter by correcting the print data. May be set. In this case, the statistical processing unit 20 may acquire information on the portion printed by the printing element in which the abnormality occurred in the imaging data and information on the portion compensated by the compensation function from the input unit (not shown).

In step S<b>6 (an example of the output process), the output unit 22 outputs quality of the printed matter based on the statistical processing result of the statistical processing unit 20 .

Thus, the processing of the defect inspection method for printed matter is completed. If the inspection of the printed matter is to be continued, the process returns to step S1 and similar processing is performed. may be omitted.

<Number of Overlapping Pixels in Division Processing>
The filter processing in the convolution layers 40A and 50A collects the data around the pixel of interest to capture the characteristics of the pixel of interest. For this reason, when a pixel at the edge becomes a pixel of interest, since there is no data around it, filtering cannot be performed correctly, and basically the reliability of the data at the edge is remarkably lowered. Therefore, edge pixels may be discarded (not used for further processing) after filtering in order to handle reliable data. For example, in the filtering results of the convolutional layers 40A and 50A with a size of 3×3, one pixel at the edge becomes an unreliable region of data, Two pixels are the data unreliable area.

FIG. 4 is a diagram for explaining an unreliable area of data resulting from convolution processing.

The image IM1 shown in FIG. 4 is an image of 10 pixels in the X direction (horizontal direction)×10 pixels in the Y direction (vertical direction). If convolution processing is performed using the edge of the image as the pixel of interest, the reliability is reduced because the calculation includes an area where data does not exist. For example, the pixel PX1 of the image IM1 has only three pixels on the right side, the lower right side, and the lower side, and no pixels exist on the upper side and the left side. Therefore, the data of the result of convolution processing with a size of 3×3 with the pixel PX1 as the target pixel is unreliable. On the other hand, the pixel PX2 of the image IM1 has all nine adjacent pixels. Therefore, the data of the convolution processing result of the size of 3×3 with the pixel PX2 as the target pixel is reliable.

The image IM2 shown in FIG. 4 is the result of convolution processing with a stride of 1 having a size of 3×3 on the image IM1. The hatched area R1 in the image IM2 is the unreliable area of the data. The region R1 includes pixels in one top row, one bottom row, one leftmost column, and one rightmost column of the image IM2.

An image IM3 shown in FIG. 4 is an image similar to the image IM2. The pixel PX3 of the image IM3 has the pixel in the uppermost row and the pixel in the leftmost column among the 25 pixels around it, which is the pixel in the data unreliable region R1. Therefore, the data of the result of convolution processing with a size of 5×5 with the pixel PX3 as the target pixel is unreliable. On the other hand, for the pixel PX4 of the image IM3, the data of all 25 pixels around it are reliable. Therefore, the data of the result of convolution processing with a size of 5×5 with the pixel PX4 as the target pixel is reliable.

The image IM4 shown in FIG. 4 is the result of convolution processing with stride 1 having a size of 5×5 on the image IM3. A hatched area R2 in the image IM4 is an area where the data is unreliable. The region R2 includes pixels in the top three rows, the bottom three rows, the left three columns, and the right three columns of the image IM4.

Image IM5 shown in FIG. 4 is an image similar to image IM4. The pixel PX5 of the image IM5 is located in the data unreliable region R2 of the pixels in the upper row and the left column among the nine adjacent pixels. Therefore, the data of the result of convolution processing with a size of 3×3 with the pixel PX5 as the target pixel is unreliable. On the other hand, for the pixel PX6 of the image IM5, the data of all nine adjacent pixels are reliable. Therefore, the data of the convolution processing result of the size of 3×3 with the pixel PX6 as the target pixel is reliable.

An image IM6 shown in FIG. 4 is the result of convolution processing with a stride of 1 using a 3×3 size filter on the image IM5. A hatched area R3 in the image IM6 is an area where the data is unreliable. Region R3 includes pixels in the top 4 rows, bottom 4 rows, left 4 columns, and right 4 columns of image IM6.

In this way, when convolution processing with stride 1 with sizes of 3×3, 5×5, and 3×3 is sequentially performed on the image IM1 of 10 pixels×10 pixels, the pixel whose data is finally reliable is the central pixel. It can be seen that there are only 4 pixels in the part.

In addition, when pooling processing is performed to reduce the total number of data by combining adjacent data into one data, for example, when pooling processing with a size of 2 × 2 is performed, one pixel after pooling processing is the original size corresponds to data for two pixels each in the vertical and horizontal directions. Therefore, if convolution processing with a size of 3×3 is performed after pooling processing with a size of 2×2, two pixels at the edge before pooling processing become an unreliable region of data.

FIG. 5 is a diagram for explaining an area where data obtained by pooling processing is unreliable.

The image IM11 shown in FIG. 5 is an image having 10 pixels in the X direction×10 pixels in the Y direction. When the image IM11 is pooled with a size of 2×2, the data of the pixels PX11, PX12, PX13, and PX14 of the image IM11 are combined into one pixel. For example, in the case of maxpooling processing, the maximum value among the data of pixels PX11, PX12, PX13, and PX14 is selected.

The image IM12 shown in FIG. 5 is the result of performing the pooling process of the stride 2 with the size of 2×2 on the image IM11. The data of the pixel PX15 of the image IM12 is data obtained by combining the data of the pixels PX11, PX12, PX13, and PX14 of the image IM11.

The image IM13 shown in FIG. 5 is similar to the image IM12. An image IM14 shown in FIG. 5 is the result of convolution processing with stride 1 having a size of 3×3 on the image IM13. A hatched area R11 in image IM14 is an unreliable area of data. The region R11 includes pixels in one row at the top, one row at the bottom, one column at the left, and one column at the right of the image IM14.

Image IM15 shown in FIG. 5 is similar to image IM11. A hatched region R12 in the image IM15 is a region traced back to the image IM11 with respect to the region R11 of the image IM14, and is a region in which the data in the image IM11 becomes unreliable after processing. That is, the data of the pixels in the region R12 of the image IM11 (image IM15) becomes unreliable due to the 2×2 size pooling processing and the 3×3 size convolution processing. The region R12 includes pixels in two rows from the top, two rows from the bottom, two columns from the left, and two columns from the right of the image IM15.

In this way, when the image IM1 of 10 pixels×10 pixels is subjected to the pooling process with stride 2 of size 2×2 and the convolution process of stride 1 of size 3×3, the pixels whose data are reliable are those in the central part. It can be seen that there are only 36 pixels.

Therefore, the imaged data division processing unit 24 needs to divide the imaged data by overlapping pixels equal to or greater than the number of unreliable pixels of the data so that calculation can be performed using only the reliable pixels of the data. That is, the imaging data division processing unit 24 may divide the imaging data by providing overlapping regions so that all pixels of the calculation result output from the first intermediate layer 38 are reliable pixels of the data.

On the other hand, if the number of overlapping pixels is large, the amount of calculation increases. Therefore, it is preferable to divide the imaging data by overlapping the unreliable number of pixels of the data.

Therefore, the imaging data division processing unit 24 uses the filter size of the convolution layer 40 of the first network 30 of the deep learning model 28, the pooling size of the pooling layer 42, and the number of convolution layers 40 and pooling layers 42 to calculate the data after processing. The width of the unreliable region is calculated, and the regions are overlapped by the calculated width and divided. As a result, the overlapped area can be regarded as an unreliable area of data, and a desired area can be operated as a reliable pixel of data. Therefore, even if the imaging data is divided, it is possible to improve the reliability of inspection over the entire surface.

Similarly, the print data division processing unit 26 needs to divide the imaging data by overlapping pixels equal to or greater than the number of unreliable pixels of the data so that calculations can be performed using only the reliable pixels of the data. That is, the print data division processing unit 26 may divide the imaging data by providing an overlap region so that all pixels of the calculation result output from the second intermediate layer 48 are reliable pixels of the data. Further, since the amount of calculation increases when the number of overlapping pixels is large, it is preferable to divide the imaging data by overlapping only the number of unreliable pixels of the data.

Therefore, the print data division processing unit 26, from the filter size of the convolution layer 50 of the second network 32 of the deep learning model 28, the pooling size of the pooling layer 52, and the number of the convolution layers 50 and pooling layers 52, the data after processing The width of the unreliable region is calculated, and the regions are overlapped by the calculated width and divided. As a result, the overlapped area can be regarded as an unreliable area of data, and a desired area can be operated as a reliable pixel of data. Therefore, even when the print data is divided, it is possible to improve the reliability of the inspection of the entire print data.

For example, if the data unreliable area consists of 5 pixels, by dividing the adjacent areas so that 10 pixels overlap each other, the entire surface of each divided area can be made the data reliable area. be.

The number of overlapping pixels is generalized. The total number of convolution layers 40 and pooling layers 42 of the first network 30 is n, and the convolution layers 40 and pooling layers 42 are numbered 1 to n in order of processing. If Sk is the filter size of the k-th convolutional layer after performing pooling with a pooling size of Sj j times, the number of pixels Nm1 to be overlapped when the imaged data division processing unit 24 divides the imaged data is

is greater than or equal to the number represented by

Let Sj be the pooling size of the pooling layer 42 that performs the pooling process of size Sj×Sj, and let Si be the pooling size of the i-th pooling layer out of j times. The stride of the pooling process is Sj. Also, let Sk be the filter size of the convolution layer 40 that performs convolution processing of size Sk×Sk. Sk is an odd number greater than or equal to 3, the filter is square, and the convolution stride is one.

Also, "floor" is the floor function. A floor function is a function that outputs the largest integer among integers smaller than an input value.

Similarly, the total number of convolutional layers 50 and pooling layers 52 of the second network 32 is n, and the convolutional layers 50 and pooling layers 52 are numbered 1 to n in order of processing. If Sk is the filter size of the k-th convolutional layer 50 after performing pooling j times with a pooling size of Sj, the number of pixels Nm2 to be overlapped when the print data division processing unit 26 divides the print data is It is equal to or greater than the number represented by Formula 1.

<Division processing>
6 and 7 are diagrams for explaining the division processing of imaging data. The image IM21 shown in FIGS. 6 and 7 is imaging data based on a captured image of a printed matter printed by a single-pass printing apparatus, and is printed on a base material conveyed in the Y direction. image data.

FIG. 6 shows an example in which the captured image data division processing unit 24 divides an area of a pixels in the X direction and b pixels in the Y direction without overlapping. On the other hand, FIG. 7 shows an example in which the imaging data division processing unit 24 divides a region of a pixels in the X direction and b pixels in the Y direction while overlapping by d pixels in both the X direction and the Y direction. Here, d=Nm1.

In this way, the imaging data division processing unit 24 divides the imaging data so that the number of pixels Nm1 or more overlaps, thereby making it possible to insert reliable pixels into the entire surface of the desired region. It is possible to maintain the inspection performance in the lost edge data.

Similarly, the print data division processing unit 26 divides the print data so that the number of pixels Nm2 or more overlaps, thereby making it possible to insert reliable pixels into the entire surface of the desired area. It is possible to maintain the inspection performance for the edge data that cannot be obtained.

<Deep learning>
In defect inspection of printed matter, there are countless picture patterns on printed matter, and furthermore, it is unknown where defects will occur. For this reason, if learning is performed using only imaging data, it is difficult to distinguish between a pattern portion having a structure very similar to that of the defect and the defect. Therefore, it is desirable to use both imaging data and print data for learning and recognition of the deep learning model 28 .

In constructing the network of the deep learning model 28 with imaging data and printing data as input, there are two methods: a method of combining imaging data and printing data into one input layer; A possible method is to provide an input layer for inputting data and combine them in an intermediate layer.

In the former method, since the correlation between image data and print data is generated from a shallow layer of the network, there is a high degree of freedom in the feature extraction method, and overlearning is likely to occur when the data used for learning is small. Therefore, a large amount of data is required to prevent overfitting, and learning takes time.

On the other hand, the latter method is equivalent to setting a constraint condition between the image data and the print data, which reduces the degree of freedom of the network, but reduces the amount of learning data required and shortens the time required for learning. leads to

Deep learning is a machine learning method that automatically extracts feature values. It is considered useful if it is reasonable. In that respect, the latter method is a very useful method because it reproduces a general comparative inspection method in which defect candidates are extracted from image data and print data and compared. be.

Although there are various types of combining, the position of combining does not matter as long as the imaging data and the print data are input separately.

FIGS. 8-10 are diagrams showing other aspects of the connection between the first network 30 and the second network 32 of the deep learning model 28. FIG. In the example shown in FIG. 8, the first network 30 and the second network 32 are coupled by the output of the first hidden layer 38 being input to the second hidden layer 48 .

In this case, the number of pixels Nm1 to be overlapped when the imaged data division processing unit 24 divides the imaged data is the convolution layer 40 and the pooling layer 42 of the first intermediate layer 38 of the first network 30 involved in the processing of the imaged data. and the convolutional layer 50 and the pooling layer 52 after being combined with the first network 30 in the second intermediate layer 48 of the second network 32 . That is, the imaging data division processing unit 24 sets the total number of the convolution layers 40 and the pooling layers 42 of the first network 30 and the convolution layers 50 and the pooling layers 52 after being combined with the first network 30 to n, The number of pixels Nm1 is calculated with the convolution layer 40, the pooling layer 42, the convolution layer 50, and the pooling layer 52 as the first to n-th processing order.

In the example shown in FIG. 9, contrary to the example shown in FIG. is combined with

In this case, the number of pixels Nm2 to be overlapped when the print data division processing unit 26 divides the print data is the convolution layer 50 and the pooling layer 52 of the second intermediate layer 48 of the second network 32 involved in print data processing. and the convolutional layer 40 and the pooling layer 42 after being combined with the second network 32 in the first intermediate layer 38 of the first network 30 . That is, the print data division processing unit 26 sets the total number of the convolution layers 50 and the pooling layers 52 of the second network 32 and the convolution layers 40 and the pooling layers 42 after being coupled to the second network 32 to n, The number of pixels Nm2 is calculated with the convolutional layer 50, the pooling layer 52, the convolutional layer 40, and the pooling layer 42 as the first to n-th processing order.

Further, in the example shown in FIG. 10, the output of the first intermediate layer 38 and the output of the second intermediate layer 48 are input to the third intermediate layer 56, thereby forming the first network 30 and the second network. A network 32 is also coupled. The output of the third hidden layer 56 is input to the first output layer 34 .

In this case, the number of pixels Nm1 to be overlapped when the imaged data division processing unit 24 divides the imaged data is the convolution layer 40 and the pooling layer 42 of the first intermediate layer 38 of the first network 30 involved in the processing of the imaged data. , and a convolution layer and a pooling layer (not shown) of the third intermediate layer 56 are taken into consideration. That is, the imaging data division processing unit 24 sets the total number of the convolution layer 40 and the pooling layer 42 of the first network 30 and the convolution layer and the pooling layer of the third intermediate layer to n, and sets the convolution layer 40 and the pooling layer 42 to n. , the convolution layer of the third intermediate layer, and the pooling layer of the third intermediate layer from the first to the n-th in order of processing, and the number of pixels Nm1 is calculated.

Similarly, the number of pixels Nm2 to be overlapped when the print data division processing unit 26 divides the print data is the convolution layer 50 and the pooling layer 52 of the second intermediate layer 48 of the second network 32 involved in print data processing. , and a convolution layer and a pooling layer (not shown) of the third intermediate layer 56 are taken into consideration. That is, the print data division processing unit 26 sets the total number of the convolution layer 50 and the pooling layer 52 of the second network 32 and the convolution layer and the pooling layer of the third intermediate layer to n, and sets the convolution layer 50 and the pooling layer 52 to n. , the convolution layer of the third intermediate layer, and the pooling layer of the third intermediate layer from the first to the n-th in order of processing, and the number of pixels Nm2 is calculated.

If deep learning can input print data and image data and output defect information, various types such as autoencoder, U-Net, VGG (Visual Geometry Group) 16 etc. can be used inside the model. . For example, if the defect information to be output is only the presence or absence of defects, a classification model such as VGG16 may be used.

Also, if the defect information is to calculate the recognition strength value of the defect, a regression model, and if the position of the defect is to be calculated, an autoencoder, U-Net, etc. may be selected according to the desired result, or may be combined. You can also let it learn.

Deep learning generally requires a large amount of high-quality data. In particular, in the case of printed matter, there are an infinite number of picture patterns to be printed, so a huge amount of data is necessary to obtain better performance. However, deep learning, which uses a large amount of data, has problems such as insufficient computer memory specifications when reading data, and it takes several days for learning.

On the other hand, in many cases, the shape of defects in printed matter inspected by the defect inspection apparatus 10 is substantially fixed. For example, defects such as missing ink, ink dripping, blocking, and the like are almost circular in shape, although the degree varies. Further, defects such as streaks and scratches are often linear. Therefore, it is possible to reduce the amount of data during learning by performing filtering so that the surrounding normal portions are blurred while the defect information is kept as it is.

<Nearby defect information>
As shown in FIG. 7, the divided area DI2 is adjacent to the divided area DI1 of the image IM21 in the X direction. Therefore, the defect information of the divided area DI1 and the defect information of the divided area DI2 are spatially adjacent to each other. Similarly, the divided area DI3 is adjacent to the divided area DI1 of the image IM21 in the Y direction. Therefore, the defect information of the divided area DI1 and the defect information of the divided area DI3 are spatially adjacent to each other and are neighbor defect information.

For example, when the defect to be determined is ink missing, the statistical processing unit 20 performs statistical processing using defect information that exists spatially nearby. That is, when the defect information of the divided area DI1, which is the defect information of interest, indicates that there is a defect, the statistical processing unit 20 performs statistical processing including the defect information of the divided areas DI2 and DI3 that are spatially adjacent to each other.

It should be noted that the fact that the defect information of two divided areas exist spatially adjacent to each other does not necessarily mean that the divided areas are adjacent to each other. Depending on the defect to be detected, the neighboring spatial distance may be varied.

Further, as shown in FIG. 7, the divided area DI4 is not adjacent to the divided area DI1 of the image IM21, and the spatial distance between them is not close. However, the divided area DI1 and the divided area DI4 exist at positions along the Y direction, respectively, and exist in the printing direction of the printed material. Therefore, the defect information of the divided area DI1 and the defect information of the divided area DI4 are treated as neighboring defect information.

If the defect to be determined is a streak in the printing direction, the statistical processing unit 20 performs statistical processing using defect information present in the printing direction. That is, the statistical processing unit 20 performs statistical processing including the defect information of the divided areas DI3 and DI4 positioned in the Y direction when the defect information of the divided area DI1, which is the defect information of interest, indicates that there is a defect. As a result, even when defect information is generated intermittently due to the same printing element, the statistical processing section 20 can appropriately perform statistical processing.

<Statistical processing>
Machine learning methods such as deep learning generally obtain statistical parameters, and have the problem of making an error in inspection results with a certain probability. Especially when dealing with large data such as commercial stamp printing, the data is basically divided into a certain size, so if the denominator of the probability of occurrence is smaller than the total number of trimmed data, it is judged as a defect even though there is no defect. False detection occurs almost every page, and most of them are judged to be waste paper (defective printed matter).

According to this embodiment, with respect to the defect information output from the inspection processing unit 18 by inputting divided data, statistical processing is performed on neighboring defect information groups that are spatially continuous with certain target defect information. By judging defects, it is possible to distinguish between erroneous detection that occurs stochastically and correct detection when detecting correctly.

In this embodiment, the all-match process is used as an example of the statistical process, but other processes may be used. For example, when the recognition intensity value is calculated as defect information, the average value of neighboring defect information groups may be obtained, and if the average value exceeds a threshold, it may be regarded as a defect. In addition, it is of course possible to use majority statistical processing to determine that a defect exists when a certain percentage or more of the samples are defective, instead of the case where all the samples are defective as in the all-match processing.

<Match all processing>
The data numbers N and M for determining whether or not there is a defect by the full match process may be variable. For example, the color tone and image structure of the captured data change as the type of base material and the type of netting of the printed matter change. This change makes it difficult to distinguish between a defect and a normal part from imaging data, and it is conceivable that the inspection performance will change. Therefore, performance can be maintained by changing N and M depending on the type of base material of the printed matter and the type of mesh.

Specifically, the coarser the graininess of the imaging data read by the scanner 132, the more likely it is that an erroneous detection will occur. For example, high-quality paper itself is rough and is read with poorer graininess than coated paper. Therefore, when the substrate is woodfree paper, it is preferable to make N relatively larger than in the case of coated paper. Further, since the graininess varies depending on the net type, it is preferable to make N relatively large for the net type with relatively poor graininess.

Further, N and M may be changed between the portion compensated by the compensation function for compensating for the abnormality of the printing element and the portion not compensated. For example, when a correction process for correcting streak unevenness is performed in a printing apparatus, the image structure at the corrected portion may change. Therefore, it is possible to make the inspection performance uniform over the entire surface by changing N and M only at the correction points in the data.

<Printing device>
A printing apparatus to which the defect inspection apparatus 10 is applied will be described.

[Configuration of inkjet printing device]
FIG. 11 is an overall configuration diagram of an inkjet printing apparatus 100 (an example of a printing system) to which the defect inspection apparatus 10 is applied. The inkjet printing apparatus 100 prints a color image by ejecting four colors of ink, cyan (C), magenta (M), yellow (Y), and black (K), onto a sheet of paper P that is a base material. machine.

As the paper P, general-purpose printing paper is used. The general-purpose printing paper is not the so-called ink-jet paper, but paper mainly composed of cellulose, such as coated paper used for general offset printing and the like. Also, water-based ink is used as the ink. Aqueous ink refers to an ink in which coloring materials such as dyes and pigments are dissolved or dispersed in water and a water-soluble solvent.

As shown in FIG. 11, the inkjet printing apparatus 100 includes a conveying section 110, a printing section 120, an imaging section 130, a drying section 140, a sorting section 150, and a paper discharging section 160. .

[Conveyor]
The transport unit 110 transports the paper P fed from a paper feeding unit (not shown) in the transport direction (Y direction). The transport section 110 includes an upstream pulley 112 , a downstream pulley 114 and a transport belt 116 .

The upstream pulley 112 has a rotating shaft (not shown) extending horizontally, and the rotating shaft is rotatably supported. The downstream pulley 114 has a rotating shaft (not shown) parallel to the rotating shaft of the upstream pulley 112, and the rotating shaft is rotatably supported.

The conveyor belt 116 is an endless belt made of stainless steel. The conveying belt 116 is stretched over the upstream pulley 112 and the downstream pulley 114 . By using the conveying belt 116 made of stainless steel, the flatness of the paper P can be maintained well.

The downstream pulley 114 has a motor (not shown) as a driving means. When the motor is driven, the downstream pulley 114 rotates counterclockwise in FIG. The upstream pulley 112 rotates counterclockwise in FIG. 11 following the rotation of the downstream pulley 114 . Due to the rotation of the upstream pulley 112 and the downstream pulley 114 , the conveyor belt 116 runs along the running path between the upstream pulley 112 and the downstream pulley 114 .

A sheet P supplied from a paper feeding unit (not shown) is placed on the transport surface of the transport belt 116 . The transport unit 110 transports the paper P placed on the transport belt 116 along the transport path from the upstream pulley 112 to the downstream pulley 114 and delivers it to the paper discharge unit 160 . The paper P is conveyed with its printing surface held horizontally at positions facing the printing unit 120, the imaging unit 130, the drying unit 140, and the sorting unit 150 on the conveying path.

A plurality of suction holes (not shown) are provided in the transport belt 116, and the suction holes of the transport belt 116 are sucked by a pump (not shown), thereby sucking and holding the paper P placed on the transport surface of the transport belt 116 on the transport surface. You may

[Printing department]
The printing unit 120 forms (prints) an image on the paper P based on the print source image. The printing unit 120 includes inkjet heads 122C, 122M, 122Y, and 122K. The inkjet head 122C ejects cyan ink droplets by an inkjet method. Similarly, inkjet heads 122M, 122Y, and 122K eject ink droplets of magenta, yellow, and black, respectively, by an inkjet method.

The inkjet heads 122C, 122M, 122Y, and 122K are arranged at regular intervals along the transport path of the paper P by the transport belt 116 . Each of the inkjet heads 122C, 122M, 122Y, and 122K is composed of a line head and formed with a length corresponding to the maximum paper width. The inkjet heads 122C, 122M, 122Y, and 122K are arranged such that the nozzle surfaces (the surfaces on which the nozzles are arranged) face the conveyor belt 116 .

The inkjet heads 122C, 122M, 122Y, and 122K print on the paper P by ejecting ink droplets from nozzles (an example of printing elements) formed on the nozzle surfaces toward the paper P conveyed by the conveying belt 116. An image is formed on the plane with a predetermined halftone type.

In this manner, the printing unit 120 generates a printed matter by scanning the paper P conveyed in the Y direction by the conveying belt 116 once, using a so-called single-pass method. That is, the printing direction of printed matter printed by the inkjet printing apparatus 100 is the Y direction.

[Imaging part]
The imaging unit 130 acquires an image of the printing surface of the paper P. FIG. The imaging unit 130 is arranged downstream of the printing unit 120 in the direction in which the paper P is transported. The imaging unit 130 includes a scanner 132 (an example of an imaging device).

The scanner 132 is a device that optically reads an image formed on the paper P using the inkjet heads 122C, 122M, 122Y, and 122K, and generates image data representing the read image (an example of a captured image). The scanner 132 includes an imaging device that captures an image printed on the paper P and converts it into an electrical signal. A color CCD (charge coupled device) linear image sensor can be used as the imaging device. A color CMOS (complementary metal oxide semiconductor) linear image sensor may be used instead of the color CCD linear image sensor.

The scanner 132 may include, in addition to the imaging device, an illumination optical system for illuminating the object to be read and a signal processing circuit for processing signals obtained from the imaging device to generate digital image data.

[Drying section]
The drying section 140 dries the ink on the paper P. FIG. The drying section 140 is arranged on the downstream side of the imaging section 130 in the direction in which the paper P is conveyed.

The drying section 140 has a heater 142 . At least one of a halogen heater and an infrared heater is used as the heater 142, for example. The heater 142 heats the printing surface of the paper P to dry the ink on the paper P. FIG. The drying section 140 may include air blowing means such as a fan or a blower.

[Sorting part]
The sorting unit 150 sorts printed matter according to the quality determination of the paper P conveyed by the conveying belt 116 . The sorting section 150 is arranged on the downstream side of the drying section 140 with respect to the sheet P transport direction. The sorting unit 150 has a stamper 152 .

The stamper 152 performs a stamping process of applying ink to the leading edge of the paper P that has been determined to be a defective printed matter according to the quality determination of the paper P transported by the transport belt 116 .

[Paper output section]
The paper discharge unit 160 collects the paper P (printed matter) on which the image is formed and dried. The paper discharge unit 160 is arranged downstream of the sorting unit 150 in the paper P transport direction and at the end point of the transport path of the transport unit 110 . The paper discharge section 160 includes a paper discharge tray 162 .

The paper discharge table 162 stacks and collects the paper P conveyed by the conveying belt 116 . The paper discharge tray 162 is provided with a front paper rest, a rear paper rest, and a lateral paper rest (not shown) to stack the papers P in an orderly manner.

In addition, the paper discharge table 162 is provided so as to be vertically movable by a lifting device (not shown). The driving of the lifting device is controlled in conjunction with the increase or decrease of the number of sheets P stacked on the sheet discharge table 162 . As a result, the highest sheet P among the sheets P stacked on the sheet discharge table 162 always has a constant height.

[Control system of inkjet printer]
FIG. 12 is a block diagram showing the internal configuration of the inkjet printer 100. As shown in FIG. The inkjet printing apparatus 100 includes a user interface 170, a storage unit 172, an integrated control unit 174, a transport control unit 176, a print control unit 178, an imaging control unit 180, a drying control unit 182, and a sorting control unit 184. , and a paper ejection control unit 186 .

The user interface 170 includes an input section (not shown) and a display section (not shown) for the user to operate the inkjet printer 100 . The input unit is, for example, an operation panel that receives input from the user. The display unit is, for example, a display that displays image data and various types of information. A user can cause the inkjet printing apparatus 100 to print a desired image by operating the user interface 170 .

The storage unit 172 stores programs for controlling the inkjet printing apparatus 100 and information necessary for executing the programs. The storage unit 172 is configured by a non-temporary recording medium such as a hard disk (not shown) or various semiconductor memories. The storage unit 172 may store imaging data and print data.

The overall control unit 174 performs various processes according to the programs stored in the storage unit 172 and performs overall control of the entire operation of the inkjet printing apparatus 100 . The defect inspection apparatus 10 is also centrally controlled by the general control unit 174 .

The transport control unit 176 causes the transport unit 110 to transport the paper P in the transport direction by controlling the motor (not shown) of the transport unit 110 . As a result, the paper P supplied from a paper feeding unit (not shown) passes through positions facing the printing unit 120, the imaging unit 130, the drying unit 140, and the sorting unit 150, and finally passes through the paper discharging unit. 160 is discharged.

The print control unit 178 controls ejection of ink from the inkjet heads 122C, 122M, 122Y, and 122K based on the print source image. The print control unit 178 uses the inkjet heads 122C, 122M, 122Y, and 122K to direct cyan, magenta, yellow, and black ink droplets toward the paper P at timings when the paper P passes positions facing the respective nozzle surfaces. Let it spit out. As a result, a color image is formed on the printing surface of the paper P, and the paper P becomes a "printed matter."

Note that the print control unit 178 may output information on the location of the ejection failure nozzle (an example of a printing element in which an abnormality has occurred) to the overall control unit 174 . The defect inspection apparatus 10 acquires information on the location of the nozzle with the ejection failure from the print control unit 178 .

In addition, the print control unit 178 may have a compensation function of compensating for printing by nozzles with ejection failures by correcting the printing source image. As an example, there is a compensation function that compensates for an ejection failure nozzle by increasing the volume of ink droplets of a plurality of adjacent nozzles. The print control unit 178 outputs information on the portion compensated by the printed matter compensation function to the general control unit 174 . The defect inspection apparatus 10 acquires information on the portion compensated by the compensation function of the printed matter from the print control unit 178 .

The imaging control unit 180 controls imaging by the scanner 132 to cause the imaging unit 130 to read the image of the paper P (printed matter). The imaging control unit 180 causes the scanner 132 to read the image formed on the sheet P at the timing when the sheet P passes the position facing the scanner 132 .

The imaging data acquisition unit 12 of the defect inspection apparatus 10 acquires the image read by the scanner 132 as imaging data. Further, the print data acquisition unit 14 of the defect inspection apparatus 10 acquires, as print data, a captured image obtained by capturing a reference printed matter. The print data acquisition unit 14 of the defect inspection apparatus 10 may acquire the print source image as print data.

The drying control unit 182 controls heating by the heater 142 to dry the paper P by the drying unit 140 . The heater 142 heats the paper P when the paper P passes through a position facing the heater 142 .

The sorting control unit 184 sorts the sheets P by the sorting unit 150 by controlling stamp processing by the stamper 152 . The sorting control unit 184 sorts the printed products into good printed products and defective printed products according to the sorting result output from the output unit 22 of the defect inspection apparatus 10 . The sorting control unit 184 stamps the stamper 152 when the sheet P passing through the position facing the stamper 152 is determined to be a defective printed matter.

The paper ejection control unit 186 controls stacking of the paper P on the paper ejection table 162 . The paper P is discharged onto the paper discharge table 162 and stacked. Ink adheres to the leading edge of the paper P of the defective printed matter. Therefore, the user can identify the defective printed matter from among the sheets P stacked on the sheet discharge tray 162 .

<Others>
In the present embodiment, an example of applying a defect inspection apparatus to a printing apparatus and inspecting a printed matter has been described, but the defect inspection apparatus can be introduced to general apparatuses having similar conditions regarding measurement objects, such as film surface inspection. It is possible.

The above defect inspection method may be configured as a program for realizing each step on a computer, and may be configured as a non-temporary recording medium such as a CD-ROM (Compact Disk-Read Only Memory) storing this program. It is possible.

In the embodiments described so far, for example, the hardware structure of the processing unit that executes various processes of the defect inspection apparatus 10 is various processors as shown below. Various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a GPU (Graphics Processing Unit), which is a processor specialized for image processing, Programmable Logic Device (PLD), which is a processor whose circuit configuration can be changed after manufacturing such as FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), etc. Dedicated to execute specific processing It includes a dedicated electrical circuit, which is a processor with designed circuitry, and the like.

One processing unit may be composed of one of these various processors, or two or more processors of the same or different type (for example, a plurality of FPGAs, a combination of CPU and FPGA, or a combination of CPU and GPU). Also, a plurality of processing units may be configured by one processor. As an example of configuring a plurality of processing units with a single processor, first, as represented by computers such as servers and clients, a single processor is configured with a combination of one or more CPUs and software. There is a form in which a processor functions as multiple processing units. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including a plurality of processing units with a single IC (Integrated Circuit) chip. be. In this way, various processing units are configured using one or more various processors as a hardware structure.

Further, the hardware structure of these various processors is, more specifically, an electric circuit combining circuit elements such as semiconductor elements.

The technical scope of the present invention is not limited to the scope described in the above embodiments. Configurations and the like in each embodiment can be appropriately combined between each embodiment without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10... Defect inspection apparatus 12... Imaging data acquisition part 14... Print data acquisition part 16... Pre-processing part 18... Inspection processing part 20... Statistical processing part 22... Output part 24... Imaging data division|segmentation processing part 26... Print data division|segmentation processing part 28 Deep learning model 30 First network 32 Second network 34 First output layer 36 First input layer 38 First intermediate layer 40 Convolution layer 40A Convolution layer 40B Convolution layer 42 Pooling layer 42A Pooling layer 42B Pooling layer 46 Second input layer 48 Second intermediate layer 50 Convolution layer 50A Convolution layer 50B Convolution layer 52 Pooling layer 52A Pooling layer 52B Pooling layer 56 Third intermediate layer 100 Inkjet printer 110 Conveyor 112 Upstream pulley 114 Downstream pulley 116 Conveyor belt 120 Printing unit 122C Inkjet head 122K Inkjet head 122M Inkjet head 122Y Inkjet head DESCRIPTION OF SYMBOLS 130... Imaging part 132... Scanner 140... Drying part 142... Heater 150... Sorting part 152... Stamper 160... Paper discharge part 162... Paper discharge tray 170... User interface 172... Storage part 174... Integrated control part 176... Conveyance control part 178 Print control unit 180 Imaging control unit 182 Drying control unit 184 Sorting control unit 186 Discharge control units DI1 to DI4 Divided areas IM1 to IM21 Images PX1 to PX15 Pixels R1 to R12 Areas S1 to S6 Defect Inspection Method Processing Steps

Claims (24)

an imaged data acquisition unit that acquires imaged data based on an imaged image obtained by imaging a printed matter;
a print data acquisition unit that acquires print data based on a print source image of the printed matter;
an inspection processing unit that uses at least one deep learning model, receives the imaging data and the print data, and outputs defect information of the printed matter;
division processing for dividing the image data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model; a pretreatment unit that performs
with
The inspection processing unit receives the divided imaging data and the printing data,
When n is the total number of the convolutional layers and the pooling layers, and Sk is the filter size of the k-th convolutional layer after j times of pooling with a pooling size of Sj,

The defect inspection apparatus for printed matter as described above. With respect to the defect information group output by the inspection processing unit, adjacent defect information consisting of N adjacent defect information, which is a natural number of 2 or more, exists spatially adjacent to the target defect information or exists in the printing direction of the printed matter. 2. A defect inspection apparatus for printed matter according to claim 1, further comprising a statistical processing unit for determining defects by statistically processing information groups. 3. The printed matter defect inspection apparatus according to claim 2 , wherein the defect information includes presence/absence of defects. 4. The defect of the printed matter according to claim 3 , wherein the statistical processing unit includes statistical processing for judging a defect when the number of pieces of defect information having a defect in the adjacent defect information group is equal to or greater than M, which is a natural number equal to or less than the N. inspection equipment. an imaged data acquisition unit that acquires imaged data based on an imaged image obtained by imaging a printed matter;
a print data acquisition unit that acquires print data based on a print source image of the printed matter;
an inspection processing unit that uses at least one deep learning model, receives the imaging data and the print data, and outputs defect information of the printed matter;
division processing for dividing the image data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model; a pretreatment unit that performs
with
The inspection processing unit receives the divided imaging data and the printing data,
With respect to the defect information group output by the inspection processing unit, adjacent defect information consisting of N adjacent defect information, which is a natural number of 2 or more, exists spatially adjacent to the target defect information or exists in the printing direction of the printed matter. Equipped with a statistical processing unit that determines defects by statistically processing the information group,
The defect information includes the presence or absence of defects,
The statistical processing unit includes statistical processing for determining a defect when the number of pieces of defect information having a defect among the neighboring defect information group is M or more, which is a natural number equal to or less than the N, and
At least one of said M and said N is a defect inspection apparatus for printed matter, wherein a value according to the type of base material of said printed matter. 6. The printed matter defect inspection apparatus according to claim 2, wherein said M and said N are equal. 7. The printed matter defect inspection apparatus according to claim 5 , wherein at least one of said M and said N is a value corresponding to a net type of said printed matter. 6. From claim 5, wherein at least one of said M and said N has a different value between a portion of a printing element in which an abnormality has occurred and a portion of said printing element in which said abnormality has not occurred in a printing device that printed said printed matter. 8. The printed matter defect inspection apparatus according to any one of 7 . At least one of said M and said N is a value in a portion compensated and a portion not compensated by a compensation function for compensating for an abnormality of a printing element of a printing device that printed said printed matter by correcting said print data. 9. The printed matter defect inspection apparatus according to any one of claims 5 to 8, wherein the defects are different from each other. 10. The printed matter defect inspection apparatus according to claim 2 , wherein the defect information includes a defect recognition strength value. 11. The printed matter defect inspection apparatus according to claim 10 , wherein the statistical processing unit includes statistical processing for judging whether or not there is a defect based on an average value of recognition intensity values of neighboring defect information groups. an imaged data acquisition unit that acquires imaged data based on an imaged image obtained by imaging a printed matter;
a print data acquisition unit that acquires print data based on a print source image of the printed matter;
an inspection processing unit that uses at least one deep learning model, receives the imaging data and the print data, and outputs defect information of the printed matter;
division processing for dividing the image data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model; a pretreatment unit that performs
with
The inspection processing unit receives the divided imaging data and the printing data,
With respect to the defect information group output by the inspection processing unit, adjacent defect information consisting of N adjacent defect information, which is a natural number of 2 or more, exists spatially adjacent to the target defect information or exists in the printing direction of the printed matter. Equipped with a statistical processing unit that determines defects by statistically processing the information group,
the defect information includes a recognition strength value of the defect;
The statistical processing unit is a printed matter defect inspection apparatus that includes statistical processing for determining whether or not there is a defect based on an average value of recognition strength values of neighboring defect information groups. an imaged data acquisition unit that acquires imaged data based on an imaged image obtained by imaging a printed matter;
a print data acquisition unit that acquires print data based on a print source image of the printed matter;
an inspection processing unit that uses at least one deep learning model, receives the imaging data and the print data, and outputs defect information of the printed matter;
division processing for dividing the image data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model; a pretreatment unit that performs
with
The inspection processing unit receives the divided imaging data and the printing data,
The deep learning model is
a first network including a first input layer having the imaging data as an input and a first intermediate layer having the first input layer as an input;
a second network including a second input layer that receives the print data and a second intermediate layer that receives the second input layer;
a first output layer that outputs the defect information;
with
A print defect inspection apparatus in which the first network and the second network are combined at least once prior to the first output layer. The preprocessing unit divides the imaging data by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of the convolutional layers and the pooling layers of the first intermediate layer. 14. The printing data according to claim 13 , wherein the print data is divided by overlapping the number of pixels according to the filter size of the convolutional layer, the pooling size of the pooling layer, and the number of the convolutional layers and the pooling layers of the second intermediate layer. defect inspection equipment for printed matter. 15. The printed matter defect inspection apparatus according to claim 1, wherein the defect information includes a position of the defect. The printed matter defect inspection apparatus according to any one of claims 1 to 15 , wherein the preprocessing unit performs processing for reducing the data size in at least one direction of the imaging data and the print data. 17. The printed matter defect inspection apparatus according to claim 16 , wherein the preprocessing unit reduces the data size in at least the conveying direction of the base material during printing. 18. The deep learning model according to any one of claims 1 to 17 , wherein the imaging data and the print data are input, and at least one of the presence or absence of a defect, the position of the defect, and the recognition strength value of the defect is output. Defect inspection apparatus for the printed matter described. a printing device that prints a printed matter using a printing element based on the print data;
an imaging device that captures the printed matter and acquires the captured image;
A printed matter defect inspection apparatus according to any one of claims 1 to 18;
printing system with an imaged data acquisition step of acquiring imaged data based on an imaged image obtained by imaging a print;
a print data acquisition step of acquiring print data based on a print source image of the printed matter;
an inspection processing step of using at least one deep learning model, inputting the imaging data and the printing data, and outputting defect information of the printed matter;
A division process is performed to divide the imaging data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model. a pretreatment step;
with
The inspection processing step receives the divided imaging data and the printing data ,
When n is the total number of the convolutional layers and the pooling layers, and Sk is the filter size of the k-th convolutional layer after j times of pooling with a pooling size of Sj,

The defect inspection method for printed matter as described above . an imaged data acquisition step of acquiring imaged data based on an imaged image obtained by imaging a print;
a print data acquisition step of acquiring print data based on a print source image of the printed matter;
an inspection processing step of using at least one deep learning model, inputting the imaging data and the printing data, and outputting defect information of the printed matter;
A division process is performed to divide the imaging data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model. a pretreatment step;
with
The inspection processing step receives the divided imaging data and the printing data ,
With respect to the defect information group output in the inspection processing step, adjacent defect information consisting of N adjacent defect information which is a natural number of 2 or more existing spatially adjacent to the attention defect information or existing in the printing direction of the printed matter Equipped with a statistical processing step of determining defects by statistically processing the information group,
The defect information includes the presence or absence of defects,
The statistical processing step includes statistical processing of determining a defect when the number of pieces of defect information having a defect among the neighboring defect information group is M or more, which is a natural number equal to or less than the N, and
At least one of the M and the N is a defect inspection method for printed matter, wherein a value according to the type of base material of the printed matter . an imaged data acquisition step of acquiring imaged data based on an imaged image obtained by imaging a print;
a print data acquisition step of acquiring print data based on a print source image of the printed matter;
an inspection processing step of using at least one deep learning model, inputting the imaging data and the printing data, and outputting defect information of the printed matter;
A division process is performed to divide the imaging data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model. a pretreatment step;
with
The inspection processing step receives the divided imaging data and the printing data ,
With respect to the defect information group output in the inspection processing step, adjacent defect information consisting of N adjacent defect information which is a natural number of 2 or more existing spatially adjacent to the attention defect information or existing in the printing direction of the printed matter Equipped with a statistical processing step of determining defects by statistically processing the information group,
the defect information includes a recognition strength value of the defect;
The statistical processing step includes statistical processing for judging whether or not there is a defect based on the average value of the recognition strength values of the adjacent defect information group. an imaged data acquisition step of acquiring imaged data based on an imaged image obtained by imaging a print;
a print data acquisition step of acquiring print data based on a print source image of the printed matter;
an inspection processing step of using at least one deep learning model, inputting the imaging data and the printing data, and outputting defect information of the printed matter;
A division process is performed to divide the imaging data and the print data by overlapping the number of pixels according to the filter size of the convolution layer, the pooling size of the pooling layer, and the number of the convolution layers and the pooling layers of the deep learning model. a pretreatment step;
with
The inspection processing step receives the divided imaging data and the printing data ,
The deep learning model is
a first network including a first input layer having the imaging data as an input and a first intermediate layer having the first input layer as an input;
a second network including a second input layer that receives the print data and a second intermediate layer that receives the second input layer;
a first output layer that outputs the defect information;
with
A method for defect inspection of printed matter, wherein the first network and the second network are combined at least once prior to the first output layer. A program for causing a computer to execute the printed matter defect inspection method according to any one of claims 20 to 23 .

Images