JP7729147B2 - Inspection equipment and program - Google Patents

Description

The present invention relates to an inspection device and a program.

2. Description of the Related Art Some printing systems include a process for inspecting printed matter for defects as a post-printing process.
In this type of printing system, an image used as a standard for inspection (hereinafter referred to as a "reference image") may be generated from print data corresponding to the printed material. This type of printing system compares an image optically read from the printed material (hereinafter referred to as an "inspection image") with the reference image, and determines that any areas where the pixel values of the different parts exceed a threshold value are defective. Examples of defects include stains and whiteouts.

Japanese Patent Application Laid-Open No. 2018-4276

However, even for the same stain or other defect, human evaluation of the defect is comprehensive, and even if the features that appear in the difference image generated from the reference image and the inspection image are quantified, the human evaluation may not match. For example, even if the magnitude of the numerical value calculated from the features that appear in the difference image is the same, the sensory evaluation may or may not determine that it is a defect.

The present invention aims to improve the accuracy of defect inspection compared to when defects are inspected by quantifying features that appear in differential images.

The invention described in claim 1 is an inspection device having a processor that inspects printed matter for defects based on a relationship in which a sensory value representing the degree of defect determined from visual sensory evaluation is linked to a combination of first color information of a reference image used as the inspection standard and second color information of an inspection image read from the printed matter.
The invention described in claim 2 is the inspection device described in claim 1, wherein the processor calculates a sensory value of the target area based on the first color information and the second color information corresponding to the target area of inspection.
A third aspect of the present invention is the inspection device according to the second aspect, wherein the processor extracts the target region from a difference image between the reference image and the inspection image.
The invention described in claim 4 is the inspection device described in claim 1, wherein the relationship provides a link between the first color information, the second color information, the characteristics of the inspection target area, and the sensory value.
A fifth aspect of the present invention is the inspection device according to the fourth aspect, wherein the features include both or either of the size of the target region and variance information of the target region.
A sixth aspect of the present invention is the inspection device according to the first aspect, wherein a plurality of the relationships are prepared according to the contents of the print job.
A seventh aspect of the present invention is the inspection device according to the sixth aspect, wherein the processor selects the relationship to be used for inspection in accordance with the contents of a print job.
An eighth aspect of the present invention is the inspection device of the first aspect, wherein the processor determines the content of post-processing depending on the magnitude of the sensory value obtained for the inspection image.
The invention described in claim 9 is an inspection device described in claim 8, in which the processor determines the content of post-processing for the printed material based on the content of multiple post-processing processes determined for each of multiple target areas of the inspection image.
The invention described in claim 10 is an inspection device described in claim 8, in which the processor determines the content of post-processing for the printed matter based on multiple sensory values obtained for multiple target areas of the reference image.
The invention described in claim 11 is a program for enabling a computer to realize a function of inspecting defects in printed matter based on a relationship in which a sensory value representing the degree of defect determined from visual sensory evaluation is linked to a combination of first color information of a reference image used as the inspection standard and second color information of an inspection image read from the printed matter.

According to the first aspect of the present invention, the accuracy of defect inspection can be improved compared to when defects are inspected by quantifying features appearing in a differential image.
According to the second aspect of the present invention, it is possible to obtain a sensory evaluation of a specific region.
According to the third aspect of the present invention, a sensory evaluation can be obtained for each target area that may have a defect.
According to the fourth aspect of the present invention, it is possible to approximate the sensory evaluation of a human.
According to the fifth aspect of the present invention, it is possible to approximate the sensory evaluation of a human.
According to the sixth aspect of the present invention, it is possible to perform defect inspection according to the content of the print job.
According to the seventh aspect of the present invention, it is possible to perform defect inspection according to the content of the print job.
According to the eighth aspect of the present invention, post-processing can be performed according to the results of the inspection.
According to the ninth aspect of the present invention, the content of post-processing for the printed matter can be determined comprehensively.
According to the tenth aspect of the present invention, the content of post-processing for the printed matter can be determined comprehensively.
According to the eleventh aspect of the present invention, the accuracy of defect inspection can be improved compared to when defects are inspected by quantifying features appearing in a differential image.

FIG. 1 is a diagram illustrating an example of the configuration of an image printing system used in a first embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of an inspection device. FIG. 10 is a diagram illustrating an example of data in a threshold table. 10 is a diagram illustrating an example of the configuration of a functional value LUT. FIG. 2 is a diagram illustrating an example of a functional configuration of an inspection device used in the first embodiment. 10 is a flowchart illustrating a part of an example of an inspection operation for defects by the inspection device used in the first embodiment. 10 is a flowchart illustrating the remaining part of the example of the defect inspection operation by the inspection device used in the first embodiment. 10 is a flowchart illustrating an example of an inspection operation for defects by the inspection device used in the second embodiment, where the inspection operation differs from the first embodiment. 10 is a flowchart illustrating an example of an inspection operation for defects by an inspection device used in a third embodiment, the difference from the first embodiment. FIG. 10 is a diagram illustrating a machine learning device that learns the relationship between a combination of color information and features of a pixel extracted as a defect candidate and a sensory value obtained from a visual sensory evaluation. FIG. 13 is a diagram illustrating an example of the functional configuration of an inspection device used in a fourth embodiment. 1A and 1B are diagrams illustrating an example of a user designating an area to be inspected for defects, in which (A) shows an example of displaying the positions of groups made up of pixels extracted as defect candidates, and (B) shows an example of a user designating a group. 10A and 10B are diagrams illustrating an example of inspecting defects in an arbitrary area designated by a user.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
First Embodiment
<System Configuration>
FIG. 1 is a diagram illustrating an example of the configuration of an image printing system 1 used in the first embodiment.
The image printing system 1 shown in FIG. 1 is made up of a print data generating device 20 and a printing system 30 connected to a network 10 .
The network 10 may be, for example, a wired LAN (Local Area Network) or a wireless LAN. However, the network 10 may also be the Internet or a mobile communication system such as 5G.

The print data generating device 20 is a processing device that converts document data into print data. The print data generating device 20 is also called, for example, a DFE (Digital Front End) or a RIP (Raster Image Processor).
The print data in this embodiment is output as raster format data (hereinafter referred to as "raster data"). Note that document data is text and images, and is provided from a computer operated by a user.

The printing system 30 comprises a printing device 310 that prints an image corresponding to the printing data on the surface of paper, an inspection device 320 that receives printed paper (hereinafter also referred to as "printed matter") from the printing device 310 and inspects the printing defects, and a post-processing device 330 that processes the printed matter according to a pre-specified output format.
The printing device 310 has a processing unit that processes print data and a printing unit that prints images on paper. The printing device 310 prints images on the surface of paper using, for example, an electrophotographic method or an inkjet method. In this embodiment, the paper is assumed to be cut paper. However, the paper may also be roll paper.

The inspection device 320 optically reads the printed matter discharged from the printing device 310 to obtain an inspection image, and inspects the printed matter for defects in real time.
Defects to be detected include, for example, stains and white spots.
The inspection device 320 in this embodiment generates a reference image from print data to be used as a reference when inspecting for print defects.
The post-processing device 330 is a so-called finisher, and performs, for example, sorting, stapling, folding, and tape binding.

<Configuration of inspection device>
FIG. 2 is a diagram illustrating an example of the hardware configuration of the inspection device 320. As shown in FIG.
2 includes a processor 321 that controls the operation of the entire apparatus, a ROM (Read Only Memory) 322 that stores a BIOS (Basic Input Output System) and the like, a RAM (Random Access Memory) 323 that is used as a work area for the processor 321, a display 324 that displays information related to the inspection, an operation reception device 325 that receives user operations, a scanner 326 that reads inspection images from printed materials, a hard disk drive 327 that stores reference images and inspection images, and a communication module 328 that is used for communication with the outside. The processor 321 and each component are connected via a signal line 329 such as a bus.

The processor 321, the ROM 322, and the RAM 323 function as a so-called computer. The processor 321 realizes various functions by executing programs. For example, the processor 321 executes a process for detecting defects in printed matter.
The display 324 is, for example, a liquid crystal display or an organic EL display. In the present embodiment, the display 324 is integrally provided on the main body of the device, but may be a monitor connected to the inspection device 320.

The operation reception device 325 is composed of a capacitance type film sensor arranged on the surface of the display 324, and switches, buttons, etc. arranged on the housing.
A device that integrates the display 324 and the operation reception device 325 is called a touch panel. The touch panel is used to receive user operations on keys displayed in software (hereinafter also referred to as "soft keys").
The scanner 326 includes a light source that irradiates the surface of the printed material being conveyed with linear illumination light, a line sensor that receives the light reflected by the surface of the printed material, and an optical system that forms an image of the reflected light on the line sensor. In this embodiment, an image of the entire printed material to be inspected is called an "inspection image." The inspection image is stored in the RAM 323, for example.

The hard disk drive 327 stores an inspection threshold table 327A, a sensory value LUT 327B, an inspection log 327C, and the like.
The inspection threshold table 327A is a table that stores thresholds according to the characteristics of the image. For example, different thresholds are prepared for bright image parts where dirt is easily noticeable and dark image parts where white areas are easily noticeable. In addition, different thresholds are prepared for edge regions and non-edge regions.

The threshold value is set assuming that the print data will be printed on white paper, i.e., that the background color of the test image is white. The white color assumed by the threshold value means, for example, a whiteness of 100%. Hereinafter, paper with a whiteness of 100% will also be referred to as white paper. It is desirable to change the threshold value depending on the color of the paper on which the print corresponding to the test image will be printed.

FIG. 3 is a diagram illustrating an example of data in the threshold table 327A.
3, threshold values are set according to the combination of the presence or absence of edges in the reference image and the inspection image. In this embodiment, multiple threshold values tables 327A are prepared according to differences in the brightness of the reference image.

For example, "100" is set as the threshold for an area where no edge exists in both the reference image and the inspection image, and "150" is set as the threshold for an area where an edge exists in both the reference image and the inspection image.
Furthermore, a threshold value of "50" is set for an area where an edge does not exist in the reference image but does exist in the inspection image, and a threshold value of "150" is set for an area where an edge exists in the reference image but does not exist in the inspection image. The former area is what is known as a stain, and the latter area is what is known as a white area.

Returning to the description of FIG.
The sensory value LUT (=Lookup table) 327B is a correspondence table that links a combination of color information of a reference image of a part detected as a defect candidate and color information of an inspection image to a sensory value that represents the degree to which a person visually inspecting the target area perceives it as a defect.
The color information of the reference image is an example of "first color information," and the color information of the inspection image is an example of "second color information." The sensory value is a numerical value that represents the result of the sensory evaluation. In this embodiment, the larger the numerical value of the sensory value, the more likely a person is to recognize it as a defect.

Fig. 4 is a diagram illustrating an example of the configuration of the functional value LUT 327B. In Fig. 4, the color information of the reference image and the test image is expressed as coordinate values in the Lab color space. For example, brightness L is given as a value between "0" and "100," and complementary color dimensions a and b are given as values between "0" and "128."
In the functional value LUT 327B, a functional value corresponding to the intersection position of the Lab values of the reference image and the Lab values of the test image is stored. For example, the functional value corresponding to the combination of (L, a, b) = (90, 96, 50) of the reference image and (L, a, b) = (30, 15, 10) of the test image is "55."

In this embodiment, the sensory values in the sensory value LUT 327B are the results of a sensory evaluation of whether a difference in color information in an area corresponding to a defect candidate is judged to be a defect when visually observed. In other words, the intersection of the Lab values of the reference image and the Lab values of the inspection image is linked to the result of a sensory evaluation of the difference in Lab values.
In other words, a value that numerically represents the degree to which the human eye would recognize a defect when the Lab values of the reference image are replaced with the Lab values of the inspection image is linked as a sensory value.
In this sense, the sensory value in this embodiment differs from the evaluation value calculated from the magnitude of the color difference.

Returning to the description of FIG.
The inspection log 327C records the results of the inspections performed for each printed material.
Additionally, the hard disk drive 327 may store a reference image generated from print data.
As described above, the reference image in this embodiment is digitally generated from the print data by the processor 321 .
In addition, firmware and application programs used for defect inspection are also stored in the hard disk drive 327. Hereinafter, firmware and application programs will be collectively referred to as "programs."

In this embodiment, the hard disk drive 327 is assumed to be the auxiliary storage device, but a semiconductor memory may also be used.
The communication module 328 is configured with a communication module that complies with wired or wireless communication standards, and may be, for example, an Ethernet (registered trademark) module, a USB (Universal Serial Bus) module, or a wireless LAN module.

Fig. 5 is a diagram illustrating an example of the functional configuration of the inspection device 320 used in the first embodiment. Fig. 5 shows functions related to defect inspection. The functions shown in Fig. 5 are realized through the execution of a program by the processor 321 (see Fig. 2).
The inspection device 320 shown in FIG. 5 has, as inspection-related functions, a reference image generation unit 351, an inspection image generation unit 352, an intra-image edge extraction unit 353, a difference image generation unit 354, a threshold setting unit 355, a defect candidate extraction unit 356, a sensory value determination unit 357, and a post-processing control unit 358.

The reference image generating unit 351 is a functional unit that generates a reference image based on print data provided from the print data generating device 20 (see FIG. 1) to the printing device 310. The reference image in this embodiment is generated assuming that an image is printed on white paper.
The reference image is provided to an intra-image edge extraction unit 353 , a difference image generation unit 354 , a threshold setting unit 355 , and a functional value determination unit 357 .

The inspection image generating unit 352 is a functional unit that generates an inspection image in real time from an image read in a line by the scanner 326 .
The inspection image is also provided to an intra-image edge extraction unit 353 , a difference image generation unit 354 , a threshold setting unit 355 , and a functional value determination unit 357 .
The printed matter that has passed through the scanner 326 is discharged to a post-processing device 330 (see FIG. 1) at the subsequent stage.

The intra-image edge extraction unit 353 is a functional unit that extracts edges from both the reference image and the inspection image. Edges are, for example, the contours or features of a subject, and can be extracted using discontinuities in brightness values. The edge image extracted from the reference image and the edge image extracted from the inspection image are provided to a difference image generation unit 354 and a threshold setting unit 355.
The difference image generating unit 354 has a registration unit 354A as a sub-function, and is a functional unit that generates a difference image between the reference image and the inspection image after registration.
The alignment unit 354A collates, for example, an edge image of the reference image with an edge image of the inspection image, and aligns the reference image with the inspection image. The difference image generation unit 354 outputs the generated difference image to the defect candidate extraction unit 356.

The threshold setting unit 355 is a functional unit that sets the threshold used for defect inspection. In this embodiment, the threshold setting unit 355 reads a threshold corresponding to a combination of the brightness of the reference image, edge information of the reference image, edge information of the inspection image, etc. from the threshold table 327A (see Figure 3), and outputs it to the defect candidate extraction unit 356.

The defect candidate extraction unit 356 is a functional unit that compares the pixel values of the differential image provided by the differential image generation unit 354 with a threshold value and extracts defect candidates contained in the inspection image. The defect candidate extraction unit 356 determines that pixels whose pixel values are less than the threshold value are normal, and determines that pixels whose pixel values are equal to or greater than the threshold value are defect candidates.
Even if a pixel exceeds the threshold, it may or may not be evaluated by a person as a defect, and therefore in this embodiment, it is referred to as a "defect candidate."
The defect candidate extraction unit 356 outputs the extracted defect candidates to the sensory value determination unit 357 .

The functional value determining unit 357 groups the pixels extracted by the defect candidate extracting unit 356 and determines whether or not each group is defective. The grouped collection of pixels is an example of a target region.
In the present embodiment, the functional value determining unit 357 groups, for example, adjacent pixels among the pixels extracted as candidates. One group may be made up of only one pixel.

However, even if a pixel extracted as a candidate is not adjacent to another candidate pixel or group, it may be included in the same group if the distance between the pixels or the shortest distance between the pixel and a pixel that makes up the group is below a predetermined threshold. Incidentally, if there are multiple groups with pixel distances below a threshold for a pixel extracted as a candidate, the pixel will be included in the group with the shorter distance.

After the grouping is completed, the functional value determination unit 357 determines a representative value of the color information of the pixels belonging to the same group for each of the reference image and the test image. The representative value is calculated as, for example, the average value of the color information of each pixel belonging to the same group.
Once the representative values for the groups to be processed have been determined, the sensory value determination unit 357 reads out the sensory values corresponding to the combination of the determined representative values from the sensory value LUT 327 B. The read-out sensory values are output from the sensory value determination unit 357 to the post-processing control unit 358 as the sensory values of the corresponding groups.

When the sensory values have been determined for all groups of defect candidates associated with the same printed matter, the sensory value determination unit 357 notifies the post-processing control unit 358 of this fact.
However, the sensory value determination unit 357 may notify the post-processing control unit 358 of the change in the print material to be processed together with the sensory value.

Note that if the post-processing control unit 358 can distinguish whether the sensory values provided by the sensory value determination unit 357 are linked to the same printed matter or different printed matters, there is no need to notify that the process of determining the sensory values for each printed matter described above has ended. For example, if each sensory value is accompanied by information identifying the printed matter being inspected, the post-processing control unit 358 can identify the sensory values associated with the same printed matter from the information accompanied by the sensory values.

The post-processing control unit 358 is a functional unit that determines the post-processing for each printed item that is the subject of inspection. In this embodiment, the post-processing control unit 358 determines the content of post-processing for each printed item based on the magnitude of one or more sensory values determined for each printed item that is the subject of inspection. In this embodiment, printed items are printed in page units. For this reason, post-processing control is also said to be performed on a page-by-page basis.

In this embodiment, the post-processing control unit 358 determines the content of post-processing for the corresponding printed matter based on the maximum value of multiple sensory values associated with the same printed matter. If there are three defect candidates extracted from the same printed matter, and the sensory value of group A is "70," the sensory value of group B is "50," and the sensory value of group C is "30," the largest of these, "70," is determined to be the representative sensory value.

In this embodiment, the post-processing control unit 358 provides three types of post-processing: "stop printing,""recordlog," and "ignore."
"Stop printing" is a post-processing method intended for printed material with high-severity defects. "Record log" is a post-processing method intended for printed material with medium-severity defects. "Ignore" is a post-processing method intended for printed material with low-severity defects.

Two thresholds, A and B, are used to determine the post-processing corresponding to these three types. For example, if the sensory value is greater than threshold A, "stop printing" is determined as the post-processing; if the sensory value is less than threshold A but greater than threshold B, "record log" is determined as the post-processing; and if the sensory value is less than threshold B, "record log" is determined as the post-processing.

In addition to these, the contents of post-processing may include, for example, "changing the discharge destination of the printed material" and "instructions for stamping, punching, etc."
The "change of print destination" is provided, for example, to distinguish prints with highly severe defects from prints with minor or medium defects detected.
"Instructions for stamping, punching, etc." are provided for the purpose of eliminating printed matter with serious defects.

<Inspection operation>
The defect inspection operation will be described below with reference to FIGS.
FIG. 6 is a flowchart illustrating a part of an example of the defect inspection operation by the inspection device 320 (see FIG. 1) used in the first embodiment.
FIG. 7 is a flowchart illustrating the remaining steps of the defect inspection operation example performed by the inspection device 320 used in the first embodiment.
As described above, the processing operations shown in FIGS. 6 and 7 are realized through the execution of a program by the processor 321 (see FIG. 2).

When the processor 321 receives a new print from the printing device 310 (see FIG. 1), it starts the processing operation shown in FIG.
First, the processor 321 acquires a reference image (step 1), and then acquires an inspection image (step 2). If the same image is to be printed continuously, the reference image may be acquired only once.

Next, processor 321 extracts edges of the reference image (step 3), and then extracts edges of the inspection image (step 4).
Processor 321 then uses the two edge images to align the reference image and the inspection image (step 5), although the alignment of the reference image and the inspection image may be performed before extracting the edge images.

Next, the processor 321 determines the brightness of the reference image for the pixel to be processed (step 6), and then determines a threshold value using the brightness of the reference image and information on the edge image (step 7). For example, the processor 321 determines a threshold value to be used for defect determination depending on the presence or absence of an edge, etc.
Furthermore, the processor 321 calculates the difference between the pixel values of the reference image and the inspection image for the pixel to be processed (step 8), and determines whether the calculated difference exceeds a threshold value (step 9). This process corresponds to the process of extracting pixels that are defect candidates.

Thereafter, the processor 321 determines whether or not the determination of all pixels has been completed (step 10).
If a negative result is obtained in step 10, processor 321 returns to step 6 and repeats the processes of steps 6 to 9 described above for the remaining pixels that have not yet been inspected.
If a positive result is obtained in step 10, the processor 321 groups the defect candidates (step 11), and then obtains the sensory values corresponding to the combinations of each color information of the reference image and the inspection image at the positions corresponding to the groups (step 12).

After obtaining the sensory values, the processor 321 determines whether or not processing of all groups has been completed (step 13).
If there are any unprocessed groups remaining, the processor 321 obtains a negative result in step 13. If a negative result is obtained in step 13, the processor 321 returns to step 12 and executes the processing of step 12 for one of the unprocessed groups.
If there are no more groups left to be processed, processor 321 obtains a positive result in step 13 .

If a positive result is obtained in step 13, the processor 321 determines the content of post-processing for the entire page (step 14). In other words, the content of post-processing for the printed material to be inspected is determined.
Thereafter, the processor 321 outputs the result of the inspection (step 15). The processor 321 also performs post-processing of the determined content.

<Summary>
In this embodiment, defects in printed matter are inspected based on the relationship between sensory values that represent the results of how people evaluate color differences as defects for combinations of color information of a reference image and an inspection image corresponding to areas extracted as defect candidates.
Therefore, the defect inspection performed by the inspection device 320 (see FIG. 1) can be closer to human sensitivity than when calculation is performed using a general formula defined by color difference.

<Second Embodiment>
In this embodiment, another example of the inspection operation will be described. Therefore, other than the inspection operation, the present embodiment is the same as the first embodiment. That is, in the present embodiment, the configuration of the image printing system 1 (see FIG. 1) and the hardware configuration and functional configuration of the inspection device 320 (see FIG. 1) are the same as those in the first embodiment.
8 is a flowchart illustrating an example of an inspection operation for defects by the inspection device 320 used in the second embodiment, with reference to differences from the first embodiment. In Fig. 8, parts corresponding to those in Fig. 7 are assigned the same reference numerals.

In this embodiment, once the sensory values for each group are acquired in step 12, the processor 321 (see FIG. 2) determines the post-processing content for the group to be processed (step 21). For example, the processor 321 determines one of the following processes for each group: "stop printing,""record a log," or "ignore."
After executing step 21, processor 321 determines whether or not processing of all groups has been completed (step 13).

In this embodiment, if a positive result is obtained in step 13, the processor 321 determines the content of post-processing for the entire page of the printed material based on the content of post-processing for each group determined in step 21 (step 14A).
For example, if there is one group determined to be "stop printing," and there are multiple groups determined to be "record a log" and "ignore," the processor 321 determines "stop printing," which is the post-processing for the most severe defect, as the post-processing content for the entire page. In other words, the processor 321 prioritizes the post-processing content of the higher order as the post-processing content for the entire page.
In the case of the inspection operation according to this embodiment, it is possible to obtain the same inspection results as in the first embodiment.

<Third Embodiment>
Next, another example of the inspection operation will be described. Therefore, other than the inspection operation, the present embodiment is the same as the first embodiment. That is, in the present embodiment, the configuration of the image printing system 1 (see FIG. 1) and the hardware configuration and functional configuration of the inspection device 320 (see FIG. 1) are the same as those in the first embodiment.
9 is a flowchart illustrating an example of an inspection operation for defects by the inspection device 320 used in the third embodiment, with reference to differences from the first embodiment. In FIG. 9, parts corresponding to those in FIG. 7 are denoted by the same reference numerals.

In this embodiment, when defect candidates are grouped in step 11, processor 321 (see Figure 2) obtains sensory values corresponding to the combination of each color information and each feature of the reference image and inspection image at the position corresponding to the group (step 31).
The features here include, for example, the number of defect candidate pixels that make up the group and the degree of dispersion of the defect candidate pixels that make up the group. Incidentally, the features may also include the shape formed by the set of pixels that make up the group. The shape may be, for example, linear or elliptical.

The number of pixels in the defect candidate that make up the group affects the size of the defect candidate: the larger the number of pixels in the defect candidate that make up the group, the higher the possibility that it will be recognized as a defect.
However, even if the number of defect candidate pixels that make up a group is large, if the difference in color information is difficult to perceive, it is not necessarily the case that a person will recognize them as defects.Also, even if the number of defect candidate pixels that make up a group is small, if the difference in color information is easy to perceive, there is a high possibility that a person will recognize them as defects.

The degree of dispersion of the defect candidate pixels that make up a group is a criterion for determining the range of pixels that are considered to be one group. If the degree of dispersion is large, the range of candidate pixels included in one group will be wide. On the other hand, if the degree of dispersion is small, the range of candidate pixels included in one group will be narrow.
In this embodiment, the sensory value LUT 327B stores a relationship linking the number of pixels of the defect candidates that make up a group, the degree of dispersion of the pixels of the defect candidates that make up a group, and the sensory values obtained when the defect candidates that have been classified based on a combination of color information of the reference image and color information of the inspection image are visually evaluated. The degree of dispersion here is what is known as "variance information."

Therefore, unlike in the first embodiment, the sensory value obtained in step 31 is given as a human sensory evaluation that also includes the characteristics of each group, which is a collection of pixels that are recognized as defect candidates as a whole.
Other processing operations are the same as those in embodiment 1. That is, after acquiring the sensory values for each group in step 31, processor 321 determines whether or not the judgment for all groups has been completed (step 13), and when the judgment for all groups has been completed, determines the content of post-processing for the entire page (step 14).

<Summary>
In this embodiment, a sensory value is given to each group, which is a collection of pixels that are defect candidates recognized as a whole, as a result of visual sensory evaluation of defect candidates classified by a combination of color information and features.
Human defect evaluation is a comprehensive evaluation that takes into account the size and variation of defect candidates. Moreover, in this embodiment, the sensory values, which are the results of visual sensory evaluation of samples that resemble actual defects, are linked to combinations of color information and features, thereby achieving defect inspection that is closer to human sensibilities.

<Fourth Embodiment>
In this embodiment, an example of performing defect inspection based on a learning model that learns the relationship between the combination of color information and features of pixels extracted as defect candidates and the sensory values obtained from visual sensory evaluation will be described.
The configuration of the image printing system 1 (see FIG. 1) and the hardware configuration of the inspection device 320 (see FIG. 1) in this embodiment are the same as those in the first embodiment.
FIG. 10 is a diagram illustrating a machine learning device 400 that learns the relationship between a combination of color information and features of pixels extracted as defect candidates and a sensory value obtained from a visual sensory evaluation.

The machine learning device 400 receives as input a combination of color information of a reference image at a position corresponding to a group of defect candidates, color information of an inspection image at a position corresponding to a group of defect candidates, the number of pixels constituting the group of defect candidates, and the degree of dispersion of the pixels of the defect candidates constituting the group, and is provided with training data that indicates a relationship in which a corresponding sensory value is output.
For the machine learning here, for example, a convolutional neural network is used.

Convolutional neural networks are an example of supervised learning.
A convolutional neural network consists of a convolutional layer that extracts local features, a pooling layer that further emphasizes the spatial features of the output of the convolutional layer, and a fully connected layer that combines the results of repeated iterations of a convolutional layer that takes the output of the pooling layer as input and a pooling layer that takes the output of the pooling layer as input.

By learning using a convolutional neural network, when a combination of color information from a reference image at a position corresponding to a group of defect candidates, color information from an inspection image at a position corresponding to a group of defect candidates, the number of pixels that make up the group of defect candidates, and the degree of dispersion of the pixels of the defect candidates that make up the group is given as input, a learning model 410 is generated that outputs a corresponding sensory value.

Fig. 11 is a diagram illustrating an example of the functional configuration of an inspection device 320 used in the fourth embodiment. In Fig. 11, parts corresponding to those in Fig. 5 are assigned the same reference numerals.
The sensory value determination section 357 in the inspection device 320 shown in FIG. 11 has a learning model 410 instead of the sensory value LUT 327B (see FIG. 5).
The sensory value determination unit 357 differs from embodiment 1 in that it provides the learning model 410 with color information of the reference image at a position corresponding to the group of defect candidates, color information of the inspection image at a position corresponding to the group of defect candidates, the number of pixels constituting the group of defect candidates, and the degree of dispersion of the pixels of the defect candidates constituting the group, and obtains the corresponding sensory value from the learning model 410.

<Summary>
In the case of the method according to this embodiment, as in the first embodiment, it is possible to make the defect inspection performed by the inspection device 320 (see Figure 1) closer to human sensibility than when calculations are made using a general formula defined by color difference.

<Fifth Embodiment>
In the above-described embodiment, defects are inspected for all groups obtained by grouping pixels extracted as defect candidates, but the user may also specify on the screen the area of the inspection image that he or she wishes to inspect.
12A and 12B are diagrams illustrating an example in which a user specifies an area to be inspected for defects. (A) shows an example of a display of the positions of groups G1 to G4, each consisting of pixels extracted as defect candidates, and (B) shows an example of a group G specified by the user.

An operation screen 500A shown in FIG. 12A and an operation screen 500B shown in FIG. 12B are displayed on, for example, the display 324 (see FIG. 2).
The operation screen 500A shown in Figure 12 (A) displays statements 510 prompting the user to operate the device, such as "A defect candidate has been found" and "Do you want to specify the candidate you want to inspect?", an inspection image 520, a button 530 that the user operates if he or she wants to specify a candidate, and a button 540 that the user operates if he or she does not want to specify a candidate.
The positions of the four groups G1 to G4 are indicated by dashed lines in the inspection image 520. In the case of the operation screen 500A, the mouse cursor is positioned over the button 530.

An operation screen 500B shown in FIG. 12B is displayed when a button 530 is clicked on the operation screen 500A.
The operation screen 500B displays a statement 550 saying "Please specify the candidate you wish to test," a button 560 to be operated when instructing the execution of the test once the candidate has been specified, a button 570 to be operated when canceling the execution of the test, and a statement 580 prompting the user to operate "Do you wish to test?"
12B, the group G1 is selected with the mouse cursor, and when the button 560 is clicked in this state, a defect inspection is executed.

Sixth Embodiment
In the case of embodiment 5, the user specifies the group to be inspected from among the areas corresponding to the groups extracted as defect candidates by the inspection device 320 (see Figure 1), but in this embodiment, we will explain the case where the user specifies any area.
FIG. 13 is a diagram for explaining an example of inspecting defects in an arbitrary area portion designated by the user.
The operation screen 600 shown in FIG. 13 is also displayed on, for example, the display 324 (see FIG. 2).

On the operation screen 600, a statement 610 saying "Please specify the area you wish to inspect," a reference image 620, and an area portion 630 specified by the mouse cursor are displayed.
When the user specifies an arbitrary area portion to be used for defect inspection, information specifying the specified area portion is directly provided to the sensory value determination unit 357 .
In this case, the sensory value determination unit 357 outputs a sensory value for the area portion designated by the user using the method described in the first embodiment and the like.

<Other embodiments>
(1) Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope of the above-described embodiments. It is clear from the claims that various modifications and improvements to the above-described embodiments are also included in the technical scope of the present invention.

(2) In the above-described embodiment, an image generated from print data is used as the reference image, but an optically read image may also be used as the reference image.

(3) In the above-described embodiment, the relationship between the color information of the reference image, the color information of the test image, and the sensory values obtained from the visual sensory evaluation is described as the relationship between the combination of coordinate values in the Lab color space and the sensory values, but other color spaces such as the L ^* a ^* b ^* color space, the HSV color space, or the HSL color space may also be used.

(4) In the above-described embodiment, the sensory value LUT 327B has been described as defining the relationship between the combination of color information of the inspection target area and the sensory value, or storing the relationship between the combination of color information and features of the inspection target area and the sensory value. However, the relationship used for the inspection may be selected depending on the content of the print job. For example, white spaces are easily noticeable in areas filled with black, and in printed forms, there is a high need to detect stains that may be mistaken for decimal points. For this reason, the sensory value LUT 327B used for defect inspection may be selected depending on the content of the print job.
Furthermore, when training the training model 410 (see FIG. 10), multiple training models 410 may be trained according to the content of the print job.

(5) The processor in each of the above-described embodiments refers to a processor in a broad sense, and includes general-purpose processors (e.g., CPUs, etc.) as well as dedicated processors (e.g., GPUs (Graphical Processing Units), ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), programmable logic devices, etc.).
Furthermore, the operations of the processors in each of the above-described embodiments may be performed by a single processor alone, or may be performed by multiple processors located in physically separate locations in cooperation with each other. Furthermore, the order in which the operations of the processors are performed is not limited to the order described in each of the above-described embodiments, and may be changed individually.

1...Image printing system, 10...Network, 20...Printing data generation device, 30...Printing system, 310...Printing device, 320...Inspection device, 327A...Threshold value table, 327B...Sensory value LUT, 327C...Inspection log, 330...Post-processing device, 351...Reference image generation unit, 352...Inspection image generation unit, 353...Intra-image edge extraction unit, 354...Difference image generation unit, 354A...Alignment unit, 355...Threshold value setting unit, 356...Defect candidate extraction unit, 357...Sensory value determination unit, 358...Post-processing control unit, 400...Machine learning device, 410...Learning model

Claims (11)

a processor;
The processor:
Inspecting defects in the printed matter based on a relationship in which a sensory value representing the degree of defect obtained from a visual sensory evaluation is linked to a combination of first color information of a reference image used as an inspection standard and second color information of an inspection image read from the printed matter.
Inspection equipment. The processor:
determining a sensory value of the target area based on the first color information and the second color information corresponding to the target area of the inspection;
The inspection device according to claim 1 . The processor:
extracting the target region from a difference image between the reference image and the inspection image;
The inspection device according to claim 2 . the relationship provides a link between the first color information, the second color information, features of the target area of inspection, and the sensory value;
The inspection device according to claim 1 . The features include both or either of the size of the target region and variance information of the target region.
The inspection device according to claim 4. A plurality of the relationships are prepared according to the contents of the print job.
The inspection device according to claim 1 . the processor selects the relationship to be used for inspection according to the content of the print job;
The inspection device according to claim 6. The processor:
determining the content of post-processing according to the magnitude of the sensory value acquired for the inspection image;
The inspection device according to claim 1 . The processor:
determining the content of post-processing for the printed matter based on the content of the plurality of post-processing processes determined for each of the plurality of target regions of the inspection image;
The inspection device according to claim 8. The processor:
determining the content of post-processing for the print based on the plurality of sensory values determined for the plurality of target regions of the reference image;
The inspection device according to claim 8. On the computer,
a function of inspecting defects in printed matter based on a relationship in which a sensory value representing the degree of defect obtained from a visual sensory evaluation is linked to a combination of first color information of a reference image used as an inspection standard and second color information of an inspection image read from the printed matter;
A program to achieve this.