JP7699429B2 - Inspection system and inspection method - Google Patents

Description

The present invention generally relates to inspection of objects.

In recent years, there has been a demand for automating work support in order to reduce the number of workers and prevent mistakes. As part of this, there is an increasing need for technology that can automatically assess the situation from images captured by wearable cameras worn by workers or fixed cameras installed on-site.

One example of a system that determines the work situation from video is an inspection system that determines whether an object at the site is normal by comparing it with a pre-registered image. This is used, for example, to find defective products at product inspection sites, or to check whether lamps, meters, etc. of parts being worked on are functioning normally at maintenance work sites.

An example of prior art related to the above is a technology for product inspection (see Patent Document 1). The technology described in Patent Document 1 aligns (corrects tilt and misalignment) an image registered in a database with an image captured by a wearable camera to check whether the product being inspected is defective.

JP 2018-54438 A

The technology described in Patent Document 1 compares the product being inspected with the registered image over the entire image, so if this is used directly for visual inspection, there is a problem that it can lead to erroneous judgments when changes are made to parts of the equipment that are unrelated to the judgment of completion of work, such as color changes due to aging, or the attachment of stickers or notes.

The present invention was made in consideration of the above points, and aims to propose an inspection system etc. that can inspect objects more appropriately.

In order to solve this problem, the present invention provides a storage unit that stores an attention spot mask that indicates which part of an object to be inspected should be focused on, a calculation unit that calculates the distance between an image of the attention spot mask of the object in an image of the object photographed by the photographing unit at a first time point and the image of the attention spot mask in an image of the object photographed by the photographing unit at a second time point after the first time point, and the distance between the image and the image of the attention spot mask in a target image, which is an image of the object when it is in a normal state, and a determination unit that compares the two distances calculated by the calculation unit to determine whether the object is in a normal state or not.

In the above configuration, the distance between the image of the attention spot mask in the image of the object at a first time point and the image of the attention spot mask in the image of the object at a second time point, and the distance between said image and the image of the attention spot mask in the target image are compared to determine whether the object is in a normal state or not, so that the object can be inspected more accurately than, for example, when the entire image is compared.

The present invention allows for more appropriate inspection of objects.

1 is a diagram showing an example of a configuration of an inspection system according to a first embodiment; FIG. 2 is a diagram showing an example of a process related to the inspection system according to the first embodiment; 4 is a diagram showing an example of a data configuration stored in a DB unit according to the first embodiment; FIG. FIG. 4 is a diagram showing an example of the configuration of a target portion mask creation unit according to the first embodiment; FIG. 4 is a diagram illustrating an example of a configuration of a combination ratio formulation unit according to the first embodiment. FIG. 4 illustrates an example of a configuration of an image comparison unit according to the first embodiment; FIG. 2 is a diagram illustrating an example of a configuration of a GUI unit according to the first embodiment. FIG. 2 is a diagram showing an example of a GUI screen according to the first embodiment; FIG. 11 is a diagram illustrating an example of a process performed by an inspection system according to a second embodiment. FIG. 13 is a diagram illustrating an example of a configuration of an inspection system according to a third embodiment. FIG. 13 is a diagram illustrating an example of a process performed after an operation determination according to the third embodiment. FIG. 13 is a diagram illustrating an example of a configuration of a GUI unit according to a third embodiment. FIG. 13 is a diagram showing an example of a GUI screen according to the third embodiment. FIG. 13 is a diagram showing an example of a GUI screen according to the third embodiment.

(I) First embodiment Hereinafter, one embodiment of the present invention will be described in detail. In this embodiment, an inspection technique for automatically determining whether the appearance of an object matches a registered image will be described. The inspection system of this embodiment uses an image from which parts not related to the inspection are excluded to determine whether an object at the site is normal or not. The object is an object related to the subject of inspection, and may be a product, a machine used in the manufacture of the product (a conveyor belt, a manufacturing device, etc.), or a device for managing the manufacture of the product (a magnet showing the progress of the manufacture, a whiteboard, etc.).

In order to eliminate areas that are not relevant to the inspection, it is possible to manually add points of interest to the panoramic image in advance and register them in a database (DB). However, when performing maintenance work or other tasks on a large number of devices, it is necessary to manually register points of interest for each device, which can be problematic in terms of the amount of work required.

In this regard, the present embodiment also describes a configuration that automatically determines points of interest, making it possible to determine whether an object is normal or not without having to manually register points of interest.

Next, an embodiment of the present invention will be described with reference to the drawings. However, the present invention is not limited to the embodiment. The following description and drawings are examples for explaining the present invention, and some parts have been omitted or simplified as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following explanation, various types of information may be explained using expressions such as "table" and "list", but the various types of information may be expressed in other data structures. To indicate independence from data structure, "XX table", "XX list", etc. may be referred to as "XX information". When explaining identification information, expressions such as "identification information", "identifier", "name", "ID", and "number" are used, but these are interchangeable.

In the following description, the same elements in the drawings are given the same numbers, and descriptions may be omitted as appropriate. Furthermore, when describing similar elements without distinguishing between them, the common portion (excluding the branch number) of the reference number including the branch number may be used, and when describing similar elements with distinction, the reference number including the branch number may be used. For example, when describing the image pair selection unit without making a distinction between them, it may be written as "image pair selection unit 101," and when describing the individual image pair selection units with distinction between them, it may be written as "interesting area mask image pair selection unit 101-1," "image comparison image pair selection unit 101-2," and so on.

In FIG. 1, 100 generally indicates an inspection system according to the first embodiment.

Figure 1 shows an example of the configuration of the inspection system 100.

The inspection system 100 includes an imaging unit 110, a DB unit 120, an alignment unit 130, a mask creation unit 140, a cropping unit 150, a combination ratio determination unit 160, an image comparison unit 170, an appearance normality determination unit 180, and a GUI unit 190.

The photographing unit 110 is, for example, a camera, and captures an image of the equipment (work target part) on which work is to be performed. The DB unit 120 is a DB that holds images (target images) of the work target part in a normal state that have been photographed in advance. The alignment unit 130 aligns corresponding parts between the image of the work target part photographed by the photographing unit 110 and the image held in the DB unit 120 in pixel units. The attention point mask creation unit 140 automatically creates a mask (attention point mask) that indicates which part of the image aligned by the alignment unit 130 is to be used for determining the normality of the appearance. The cropping processing unit 150 performs a process (cropping process) of cutting out the area indicated by the attention point mask from the aligned image. The combination ratio formulation unit 160 formulates the relative importance of the calculation method used to calculate the difference (distance) between the images. The image comparison unit 170 calculates the distance between the images based on the importance formulated by the combination ratio formulation unit 160. The appearance normality determination unit 180 performs an appearance normality determination (task determination) based on the distance between the images to determine whether the task was completed normally. The GUI unit 190 displays the results of the processing to the user. The user may be the worker who performs the task, or may be a person different from the worker.

The inspection system 100 also includes an image pair selection unit 101, which includes an attention area mask image pair selection unit 101-1, an image comparison image pair selection unit 101-2, and a combination ratio image pair selection unit 101-3. Each image pair selection unit 101 selects an appropriate pair of images (two images) from the input image group, and outputs them to subsequent processing. In this embodiment, the attention area mask image pair selection unit 101-1 selects a pair of an image of the part to be worked on at the start of the work (start image) and an image of the part to be worked on at the end of the work (end image). The image comparison image pair selection unit 101-2 selects a pair of a start image and an end image, and a pair of a target image and an end image. The combination ratio image pair selection unit 101-3 selects a pair of a start image and a target image.

Details of the DB unit 120 will be described later using FIG. 3. Details of the attention area mask creation unit 140 will be described later using FIG. 4. Details of the combination ratio formulation unit 160 will be described later using FIG. 5. Details of the image comparison unit 170 will be described later using FIG. 6. Details of the GUI unit 190 will be described later using FIG. 7.

The alignment unit 130 receives as input a target image, a start image, and an end image. These images all show the same object (the part to be worked on), but the image capturing conditions, such as the angle and the distance from the subject, are different, so the part to be worked on looks different in the images. In order to compare the images, it is preferable that the appearance of the part to be worked on is consistent at the pixel coordinate level, so the alignment unit 130 geometrically transforms (aligns) the other images so that they look the same as the part to be worked on in the target image.

The registration calculation method may be an existing method such as template matching, AKAZE, or SURF, which is a feature matching method. By performing the registration process, the correspondence between pixels capturing the same part of the input image can be determined, making it possible to perform image comparison downstream of the process. Furthermore, if the DB unit 120 holds a mask of interest, the registration unit 130 also receives that image as an input. In this case, there is no need to align the interest mask with the target image (geometric transformation), and the registration unit 130 outputs the interest mask as is.

The cropping processor 150 performs a process called cropping, which cuts out a part of the input image. To specify the area to be cropped, the cropping processor 150 also receives an area of interest mask as input.

The appearance normality determination unit 180 receives as input from the image comparison unit 170 the distance between the start image and the end image (distance 1) and the distance between the end image and the target image (distance 2), and determines whether the appearance of the work target part shown in the end image is normal by comparing these distances. One example of the determination logic is that if distance 2 is smaller than distance 1, the end image is more similar to the target image than the start image, and therefore the appearance is determined to be normal.

Figure 2 shows an example of processing related to the inspection system 100.

In S200, when starting work, the user photographs the part to be worked on using the photographing unit 110, and the photographing unit 110 acquires (takes) an image (starting image) of the part to be worked on.

In S201, the worker performs a task. For example, the worker performs the task according to a predetermined procedure or waits until the operation of the equipment is completed, and as time passes, the appearance of the part being worked on changes.

In S202, when the user finishes the work, the image capturing unit 110 captures an image of the work target part, which then captures an image of the work target part (finishing image). The starting image and the finishing image are input to the alignment unit 130.

In S203, the start image and end image that have been aligned by the alignment unit 130 are input to the attention area mask creation unit 140, which creates an attention area mask.

In S204, the starting image and the target image that have been aligned by the alignment unit 130 are input to the combination ratio formulation unit 160, which determines the importance of the calculation method used to calculate the distance between the images.

In S205, the image comparison unit 170 calculates the distance between the start image and the end image (distance 1) and the distance between the end image and the target image (distance 2). In addition, the importance of the calculation method used by the image comparison unit 170 at this time conforms to that determined in S204.

In S206, the appearance normality determination unit 180 compares the distance between the images calculated in S205 and performs a work determination as to whether the work target part is in a normal state (visual inspection as to whether the appearance of the work target part shown in the end image is normal). If the appearance normality determination unit 180 determines that the work target part is not in a normal state, it notifies the GUI unit 190 of information indicating that the work target part is not in a normal state (is in an abnormal state) (S207). If the appearance normality determination unit 180 determines that the work target part is in a normal state, it notifies the GUI unit 190 of information indicating that the work target part is in a normal state (is in a normal state) (S208).

Figure 3 is a diagram showing an example of the data structure stored in DB unit 120. DB unit 120 stores data in a form in which a work process is linked to an image used for work judgment. DB unit 120 stores information in which a work ID capable of identifying a work, a target image, a reference image, and a focus area mask are associated with each other. Of this information, the reference image and focus area mask may or may not actually store data.

Figure 4 shows an example of the configuration of the attention area mask creation unit 140 and its relationship with other units.

The interest area mask creation unit 140 receives a pair of images from the interest area mask image pair selection unit 101-1 as input, and creates an interest area mask from the difference between those images. In addition, if the DB unit 120 holds an interest area mask, it receives the interest area mask from the alignment unit 130 as input.

The attention area mask creation unit 140 includes a pixel difference unit 400, a threshold processing unit 410, a mask area determination unit 420, and an attention area mask integration unit 430.

First, the pixel difference unit 400 calculates the difference between each pixel of the two input images. Next, the threshold processing unit 410 performs threshold processing on the obtained difference using a preset threshold. Areas with large differences between the two images are areas that changed during work, so these areas are calculated through processing by the pixel difference unit 400 and the threshold processing unit 410. In this embodiment, the output of the threshold processing unit 410 is referred to as a mask candidate image, and pixels in the mask candidate image that are equal to or greater than the threshold are referred to as mask candidate pixels.

Finally, the mask location determination unit 420 determines the attention location mask. The mask location determination unit 420 includes a region creation unit 421, a noise removal unit 422, a nearby region integration unit 423, and a DB information output unit 424. The region creation unit 421 groups mask candidate pixels and recognizes them as mask candidate regions. The noise removal unit 422 removes mask candidate regions that are smaller than a preset threshold (area). If two mask candidate regions are nearby (if they are smaller than a preset threshold (number of pixels)), the nearby region integration unit 423 integrates them and treats them as a single mask candidate region. If the DB unit 120 holds an attention location mask, the DB information output unit 424 outputs the attention location mask.

As explained using FIG. 1, in this embodiment, the attention area mask image pair selection unit 101-1 selects one pair (a pair of a start image and an end image) and inputs it to the attention area mask creation unit 140, but in other embodiments, multiple pairs may be input. In this case, an attention area mask is created for each image pair, so that the attention area mask integration unit 430 integrates (e.g., performs an OR operation) multiple mask candidate regions into a single attention area mask and outputs it.

Figure 5 shows an example of the configuration of the combination ratio formulation unit 160 and its relationship with other units.

The combination ratio formulation unit 160 receives a pair of images from the combination ratio image pair selection unit 101-3 as input, and outputs a value representing the importance of the calculation method (distance calculation unit 600 described below) used by the image comparison unit 170 from those images. The combination ratio formulation unit 160 includes, as weighting units, a fixed weighting unit 500, a distance proportional weighting unit 510, a distance inverse proportional weighting unit 520, and a weight setting unit 530. The combination ratio formulation unit 160 may use any weighting unit to formulate the importance.

The fixed weighting unit 500 determines that the importance of all distance calculation units 600 is the same, and outputs the I-th importance I _I =1/(the number of distance calculation units 600).

The distance proportional weighting unit 510 calculates the distance between the input image pairs using the distance calculation unit 600, and determines the importance in proportion to the distance. In other words, if the distance output of the I-th distance calculation unit 600 is D _I , the I-th importance I _I =D _I /Σ _J (D _J ) is output.

The distance inverse proportional weighting unit 520 calculates the distance between the input image pairs using the distance calculation unit 600, and determines the importance inversely proportional to the distance. In other words, if the distance output of the I-th distance calculation unit 600 is D _I , the I-th importance I _I = (1/D _I )/Σ _J (1/D _J ) is output.

The weight setting unit 530 outputs the pre-set importance as is.

The combination ratio formulation unit 160 may also include a combination ratio integration unit 540.

As described with reference to FIG. 1, in this embodiment, the combination ratio image pair selection unit 101-3 selects one pair (a pair of a starting image and a target image) and inputs it to the combination ratio formulation unit 160, but in other embodiments, multiple pairs may be input. In this case, the importance is formulated for each image pair, and is integrated (e.g., an average value is calculated) in the combination ratio integration unit 540.

Figure 6 shows an example of the configuration of the image comparison unit 170 and its relationship with other units.

The image comparison unit 170 calculates the distance (degree of difference) between the image group, which is output from the cropping processing unit 150 and in which the parts that are important for the task judgment have been cropped out. Since the distance between images is calculated for any two images, the image comparison image pair selection unit 101-2 selects an image pair from the image group output from the cropping processing unit 150.

As explained with reference to FIG. 1, the image pairs to be selected are a pair of a start image and an end image, as well as a pair of an end image and a target image. Each pair selected by the image comparison image pair selection unit 101-2 is input to the image comparison unit 170, so the output of the image comparison unit 170 in this embodiment is two distances corresponding to the two pairs.

Various distance calculation units 600 are available as processing units that calculate the distance between images. Each distance calculation unit 600 receives two images as input and outputs one value that represents the distance between the images. In addition, the image comparison unit 170 may perform preprocessing on the image to be input to the distance calculation unit 600, such as converting the image to a black and white image or to another color space such as HSV or HSL, before inputting the image.

The image comparison unit 170 includes a distance calculation unit 600, which includes a histogram distance calculation unit 600-1, a pixel average distance calculation unit 600-2, a pixel median distance calculation unit 600-3, a pixel difference sum distance calculation unit 600-4, and a learning model type distance calculation unit 600-5.

The histogram distance calculation unit 600-1 creates a histogram of pixel values for each image and calculates the distance between them. The method for calculating the histogram distance may be a comparison of absolute values for each bin of the histogram, or a method such as EARTH MOVER'S DISTANCE.

The pixel average distance calculation unit 600-2 calculates the average pixel value for each image and calculates the distance between them. The distance calculation method may be a comparison of absolute values or a comparison such as the L2 distance.

The pixel median distance calculation unit 600-3 calculates the median pixel value for each image and calculates the distance between them. The distance calculation method may be a comparison of absolute values or a comparison such as the L2 distance.

The pixel difference sum distance calculation unit 600-4 calculates the difference between pixels that are in the same position between the two images, and uses the sum of these difference values as the distance. The pixel difference may be calculated by comparing absolute values or by a comparison such as the L2 distance.

The learning model distance calculation unit 600-5 calculates the distance between two images using a model (machine learning model, AI model, etc.) adjusted based on learning data.

The image comparison unit 170 includes a distance result combination unit 610 for aggregating the output of the distance calculation unit 600. The distance result combination unit 610 refers to the importance of the distance calculation unit 600 received from the combination ratio formulation unit 160 and aggregates the distance outputs of the distance calculation unit 600 into one value. More specifically, the distance result combination unit 610 includes a weighted multiplication unit 611 and a threshold processing unit 612. The weighted multiplication unit 611 multiplies the importance by the corresponding distance output and calculates the sum. The threshold processing unit 612 sets a threshold in advance for each distance calculation unit 600, and if the distance output exceeds that value, it sets it to "1", otherwise it sets it to "0", and the results are aggregated by importance as explained for the weighted multiplication unit 611.

Figure 7 shows an example of the configuration of the GUI unit 190 and its relationship with other units.

The GUI unit 190 receives as input the results of the work judgment output from the appearance normality judgment unit 180, in addition to the group of images output from the alignment unit 130. The GUI unit 190 includes a judgment basis GUI unit 700. The judgment basis GUI unit 700 takes the form of a GUI (Graphical User Interface) and includes the following processing units (1) to (3) to enable a person (user) operating the GUI to efficiently view the basis of the work judgment.

(1) The image display unit 701 displays the start image, the end image, the target image, the attention area mask, and the image cropped by the cropping processing unit 150.
(2) The assessment result display unit 702 displays the results of the work assessment output from the appearance normality assessment unit 180.
(3) The judgment approval unit 703 confirms with the user whether the result of the work judgment is correct.

Figure 8 is a diagram showing an example of a GUI screen visualized by the judgment basis GUI unit 700. First, the GUI screen is defined to include a component 800 that displays the image used in the work judgment, a component 810 that displays the result of the work judgment, and a component 820 that prompts the user to confirm whether the result of the work judgment is correct.

In this way, by performing the processing of this embodiment, it is possible to perform work judgment that is robust against deterioration of the parts being worked on and changes in areas unrelated to the work judgment. It is also possible to automatically determine points of interest, making it possible to perform work judgment without having to manually register points of interest.

(II) Second embodiment The configuration of the inspection system 100 of this embodiment is the same as that shown in Fig. 1, and therefore illustration and description thereof will be omitted. A difference from the first embodiment is that the DB unit 120 holds, in addition to the target image, a reference image, which is an image when the external appearance of the part to be worked on is abnormal, and outputs it to the alignment unit 130. Note that there may be multiple reference images.

In addition to the image pairs selected by the attention area mask image pair selection unit 101-1, image comparison image pair selection unit 101-2, and combination ratio image pair selection unit 101-3 in the first embodiment, in this embodiment, additional image pairs are also selected. The attention area mask image pair selection unit 101-1 selects a pair of a target image and a reference image. The image comparison image pair selection unit 101-2 selects a pair of an end image and a reference image and a pair of an end image and a target image. The combination ratio image pair selection unit 101-3 selects a pair of a target image and a reference image. Furthermore, if there are multiple reference images, all combinations are considered when creating pairs.

In this embodiment, the appearance normality determination unit 180 receives as input from the image comparison unit 170 the distance between the start image and the end image (distance 1), the distance between the end image and the target image (distance 2), and the distance between the end image and the reference image (distance 3...N), and performs work determination by comparing these distances. Examples of determination logic include a method in which the part being worked on is determined to be normal when distance 2 is smaller than distance 1 as in the first embodiment, and a method in which the part being worked on is determined to be in a normal state when distance 2 is the smallest compared to distance 3...N, since the end image is more similar to the target image than the reference image.

Figure 9 shows an example of processing related to the inspection system 100 in this embodiment.

In S900, when starting work, the user photographs the part to be worked on using the photographing unit 110, and the photographing unit 110 acquires (takes) an image (starting image) of the part to be worked on.

In S901, the worker performs a task. For example, the worker performs the task according to a predetermined procedure or waits until the operation of the equipment is completed, and as time passes, the appearance of the part being worked on changes.

In S902, when the user finishes the work, the image capturing unit 110 captures an image of the work target part, which then captures an image of the work target part (finishing image). The starting image and the finishing image are input to the alignment unit 130.

In S903, the start image and end image aligned by the alignment unit 130, the target image, and the reference image are input to the attention spot mask creation unit 140, which creates an attention spot mask. Note that the process of creating an attention spot mask from the target image and the reference image is similar to the process of creating an attention spot mask from the start image and the end image, so a description thereof will be omitted.

In S904, the start image and end image aligned by the alignment unit 130, the target image, and the reference image are input to the combination ratio formulation unit 160, which determines the importance of the calculation method used to calculate the distance between the images.

In S905, the image comparison unit 170 calculates the distance between the start image and the end image (distance 1), the distance between the end image and the target image (distance 2), and the distance between the end image and the reference image (distance 3...N). The importance of the calculation method used by the image comparison unit 170 at this time conforms to that determined in S904.

In S906, the appearance normality determination unit 180 compares the distance between the images calculated in S905 and performs a work determination as to whether the work target part is in a normal state (visual inspection as to whether the appearance of the work target part shown in the end image is normal). If the appearance normality determination unit 180 determines that the work target part is not in a normal state, it notifies the GUI unit 190 of information indicating that the work target part is not in a normal state (is in an abnormal state) (S907). If the appearance normality determination unit 180 determines that the work target part is in a normal state, it notifies the GUI unit 190 of information indicating that the work target part is in a normal state (is in a normal state) (S908).

The GUI in this embodiment is largely similar to the GUI in the first embodiment, so illustrations and explanations are omitted. The difference is that in addition to the start image, end image, and target image, a reference image is also displayed in the part 800.

In this way, by performing the processing of this embodiment, it is possible to perform work judgment using not only the normal appearance (target image) of the part being worked on, but also the appearance when it is abnormal (reference image), making it possible to perform work judgment more robustly.

(III) Third embodiment Fig. 10 is a diagram showing an example of the configuration of the inspection system 100 in this embodiment. In this embodiment, a configuration for performing feedback from the GUI unit 190 to the DB unit 120 is added to the inspection system 100 described in the first embodiment. Also, in this embodiment, not only the target image but also a reference image may be used as in the second embodiment.

The feedback mechanism added in this embodiment is explained using the flowchart in Figure 11.

Figure 11 shows an example of the processing that is performed after the appearance normality determination unit 180 performs an operation determination (S207, S208 in Figure 2, S907, S908 in Figure 9).

In S1101, the GUI unit 190 asks the user whether the basis for the work assessment should be presented.

In S1102, the GUI unit 190 presents a GUI screen showing the basis for the work determination as shown in FIG. 13.

In S1103, the GUI unit 190 edits the current work content and asks the user whether to update the DB unit 120.

In S1104, the GUI unit 190 presents a GUI screen on which the attention area mask can be edited, as shown in FIG. 14, and allows the user to edit the attention area mask.

In S1105, the GUI unit 190 updates the DB unit 120 based on the attention area mask edited in S1104 and the end image.

Figure 12 shows an example of the configuration of the GUI unit 190 in this embodiment and its relationship with other units.

The GUI unit 190 receives as input the image group output from the alignment unit 130, as well as the result of the work judgment output from the appearance normality judgment unit 180. The GUI unit 190 includes a judgment basis GUI unit 700 and an editing GUI unit 1200.

First, the judgment basis GUI unit 700 includes the following processing units (1) to (4) for enabling a user operating the GUI to efficiently view the basis for an activity judgment.
(1) The image display unit 701 displays the start image, the end image, the target image, the attention area mask, and the image cropped by the cropping processing unit 150.
(2) The assessment result display unit 702 displays the results of the work assessment output from the appearance normality assessment unit 180.
(3) The judgment approval unit 703 confirms with the user whether the result of the work judgment is correct.
(4) The edit confirmation unit 1201 edits the target portion mask and confirms with the user whether or not to update the DB unit 120.

The editing GUI unit 1200 includes the following processing units (1) to (11) for enabling a person (user) operating the GUI to efficiently edit the attention area mask.
(1) The image display unit 1211 displays the start image, the end image, the target image, the attention area mask, and the image cropped by the cropping processing unit 150.
(2) The region selection unit 1212 defines, as a region of interest, adjacent pixels among pixels detected as a region of interest in the region of interest mask. The region selection unit 1212 selects a region of interest in response to a user input.
(3) The region enlargement/reduction unit 1213 changes the size of the region of interest selected by the region selection unit 1212 in accordance with a user input.
(4) The region moving unit 1214 translates the region of interest selected by the region selecting unit 1212 in accordance with a user input.
(5) The region rotation unit 1215 rotates the region of interest selected by the region selection unit 1212 in response to a user input.
(6) The region combining unit 1216 combines the multiple regions of interest selected by the region selecting unit 1212 in response to a user input.
(7) The region deletion unit 1217 deletes the region of interest selected by the region selection unit 1212 in response to a user input.
(8) A pixel addition/deletion unit 1218 adds or deletes any pixel in the target location mask in response to a user input.
(9) The distance result combination ratio change unit 1219 sets the output value of the weight setting unit 530 according to a user input.
(10) The provisional judgment result display unit 1220 displays the result of the work judgment when using the attention area mask being edited.
(11) The editing completion unit 1221 ends the editing work.

Figure 13 shows an example of a GUI screen visualized by the judgment basis GUI unit 700.

The GUI screen is defined by a component 800 that displays the image used in the work assessment, a component 810 that displays the results of the work assessment, a component 820 that asks the user whether the results of the work assessment are correct, and a component 1300 that edits the attention area mask and asks the user whether to update the DB section 120.

Figure 14 shows an example of a GUI screen visualized by the editing GUI unit 1200.

The GUI screen is defined by a component 1400 for displaying an image used in the work judgment, a component 1401 for selecting an area of interest, a component 1402 for changing the size of the selected area of interest, a component 1403 for translating the selected area of interest, a component 1404 for rotating the selected area of interest, a component 1405 for combining multiple selected areas of interest, a component 1406 for deleting the selected area of interest from the mask, a component 1407 for adding/deleting an arbitrary pixel to/from the area of interest mask, a component 1408 for setting the output value of the weight setting unit 530, a component 1409 for displaying the results of the work judgment when the area of interest mask being edited is used, and a component 1410 for ending the editing work.

In this way, by performing the processing of this embodiment, even if automatic generation of a mask of interest fails, manual editing is possible, enabling more robust task determination.

The program according to the present invention is a program that is incorporated into a computer and causes the computer to operate as an inspection system 100. By incorporating the program according to the present invention into a computer, the inspection system 100 shown in the block diagram of FIG. 1 etc. is configured.

(IV) Supplementary Notes The above-described embodiment includes, for example, the following contents.

In the above embodiment, the present invention has been described as being applied to an inspection system, but the present invention is not limited to this and can be widely applied to various other systems, devices, methods, and programs.

The program is executed by a processor (e.g., CPU, GPU) to perform a specified process using storage resources (e.g., memory) and/or interface devices (e.g., communication ports) as appropriate, so the subject of the process may be the processor. Similarly, the subject of the process performed by executing the program may be a computer, controller, device, system, calculator, or node having a processor. The subject of the process performed by executing the program may be a calculation unit, and may include a dedicated circuit (e.g., FPGA, ASIC) that performs a specific process.

A program may be installed in a device such as a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and a storage resource that stores the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. Also, two or more programs may be realized as one program, and one program may be realized as two or more programs.

The inspection system 100 may be an inspection device such as a personal computer, and the functions of the inspection device (DB section 120, alignment section 130, attention area mask creation section 140, cropping processing section 150, combination ratio formulation section 160, image comparison section 170, appearance normality determination section 180, GUI section 190, etc.) may be realized, for example, by a CPU reading a program stored in a ROM into a RAM and executing it (software), or may be realized by hardware such as a dedicated circuit, or may be realized by a combination of software and hardware. In addition, some of the functions of the inspection device may be realized by another computer capable of communicating with the inspection device.

In addition, one function of the inspection device may be divided into multiple functions, or multiple functions may be combined into one function. Also, some of the functions of the inspection device may be provided as a separate function, or may be included in another function. Also, some of the functions of the inspection device may be realized by another computer that can communicate with the inspection device.

In addition, the screens shown and described in the above embodiment are merely examples, and any design may be used as long as the information received is the same.

In the above embodiment, the average value is used in various calculations, but it is not limited to the average value, and may be the maximum value, the minimum value, the difference between the maximum value and the minimum value, the mode, the median, the standard deviation, etc.

In addition, in the above-described embodiment, the output of information is not limited to display on a display. The output of information may be audio output from a speaker, output to a file, printing on paper media using a printing device, projection on a screen using a projector, or other forms.

In addition, in the above description, information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

The above-described embodiment has the following characteristic configurations:

The inspection system (e.g., inspection system 100) includes a memory unit (e.g., DB unit 120 and attention point mask creation unit 140, computer, circuit) that stores (e.g., stores in a storage device) an attention point mask that indicates which part of an object to be inspected (e.g., a part to be worked on, a component) should be focused on, and an image (e.g., a part image) of the attention point mask of the object in an image (e.g., an overall image) captured by a photographing unit (e.g., photographing unit 110) at a first time point (which may be the start of work or when the component is installed), and an image (e.g., a part image) of the attention point mask of the object in an image (e.g., an overall image) captured by a photographing unit (e.g., photographing unit 110) at a second time point after the first time point (which may be the end of work, or when the component is inspected or inspected, etc.). 10, which may be the same as or different from the one used for photographing at the first point in time. ) and the distance between the image of the attention area mask in the image photographed by the image comparison unit 170, and the distance between the image of the attention area mask in the target image, which is an image when the object is in a normal state, and a determination unit (e.g., an appearance normality determination unit 180, a computer, a circuit) that compares the two distances calculated by the calculation unit to determine whether the object is in a normal state or not (this may be a work determination as to whether the work was performed correctly or not, or a defective product determination as to whether the part is in a normal state or not).

In the above configuration, the distance between the image of the attention spot mask in the image of the object at a first time point and the image of the attention spot mask in the image of the object at a second time point, and the distance between said image and the image of the attention spot mask in the target image are compared to determine whether the object is in a normal state or not, so that the object can be inspected more accurately than, for example, when the entire image is compared.

The inspection system includes an alignment unit (e.g., alignment unit 130, computer, circuit) that aligns an image of an object captured by an imaging unit with a target image of the object, and the alignment unit aligns an image of the object captured by the imaging unit before work related to the object is started with a first image aligned with the target image of the object, and an image of the object captured by the imaging unit after the work is completed with a second image aligned with the target image, and the storage unit generates and stores a mask of a point of interest of the object from the difference between the first image and the second image.

In the above configuration, a mask of the target area of the object is automatically generated and stored from an image of the object before the work and an image of the object after the work, which can reduce, for example, the labor required for registering the mask of the target area.

The inspection system includes a processing unit (cropping processing unit 150, computer, circuit) that performs processing to cut out the portion of the attention area mask from a predetermined image, the processing unit cuts out the portion of the attention area mask from the first image to obtain a first area image (e.g., the attention area image of the start image), cuts out the portion of the attention area mask from the second image to obtain a second area image (e.g., the attention area image of the end image), and cuts out the portion of the attention area mask from the target image to obtain a target area image (e.g., the attention area image of the target image), the calculation unit calculates a first distance between the first area image and the second area image and a second distance between the second area image and the target area image, and the determination unit compares the first distance and the second distance calculated by the calculation unit to determine whether the object is in a normal state.

The inspection system includes a determination unit (e.g., a combination ratio formulation unit 160, a computer, a circuit) that determines the importance of each of the multiple calculation methods used by the calculation unit to calculate the first distance and the second distance.

According to the above configuration, even if the calculation method that can correctly calculate the distance between images varies depending on the image pair, for example, it is possible to automatically select a calculation method that can correctly calculate the distance between images.

The calculation unit compares histograms created from the first location image and the second location image to calculate a distance (for example, histogram distance calculation unit 600-1), compares the average values of pixel values between the first location image and the second location image to calculate a distance (for example, pixel average distance calculation unit 600-2), compares the medians of pixel values between the first location image and the second location image to calculate a distance (for example, pixel median distance calculation unit 600-3), calculates the sum of difference values obtained by calculating the difference between pixels at the same position between the first location image and the second location image as a distance (for example, pixel difference sum distance calculation unit 600-4), calculates the distance between the first location image and the second location image using a model adjusted based on learning data (for example, learning model type distance calculation unit 600-5), and multiplies each calculated distance by the importance determined by the determination unit and takes the sum, and sets the first distance (for example, distance result combination unit 610), The histograms created from the second location image and the target location image are compared to calculate the distance (for example, histogram distance calculation unit 600-1), the average pixel values of the second location image and the target location image are compared to calculate the distance (for example, pixel average distance calculation unit 600-2), the median pixel values of the second location image and the target location image are compared to calculate the distance (for example, pixel median distance calculation unit 600-3), the sum of the difference values calculated for the difference between pixels at the same position between the second location image and the target location image is calculated as the distance (for example, pixel difference sum distance calculation unit 600-4), the distance between the second location image and the target location image is calculated using a model adjusted based on learning data (for example, learning model type distance calculation unit 600-5), and the calculated distances are multiplied by the importance determined by the determination unit and the sum is taken as the second distance (for example, distance result combination unit 610).

The inspection system includes a user interface unit (e.g., GUI unit 190, computer, circuit) that outputs the first image, the second image, the target image, the attention area mask, the first area image, the second area image (e.g., part 800), the result of the judgment by the judgment unit (e.g., part 810), and part information (e.g., part 820) for receiving the result of confirmation of whether the result is correct (e.g., GUI screen shown in FIG. 8).

According to the above configuration, for example, the user can judge whether the attention area mask is correct or not based on each displayed image, and can determine whether the judgment result is correct.

The storage unit stores a target image, which is an image of the object when it is in a normal state, and one or more reference images, which are images of the object when it is not in a normal state, and generates and stores a mask of an area of interest of the object from the difference between the target image and the reference image (see, for example, FIG. 9).

In the above configuration, a mask of areas of interest is generated from a target image and a reference image. For example, the more reference images there are, the more noise there can be in the mask of areas of interest, and the more accurate the inspection can be.

The inspection system includes a user interface unit (e.g., GUI unit 190, computer, circuit) that enables editing of the attention area mask stored in the memory unit, and the memory unit stores the attention area mask edited by the user interface unit.

According to the above configuration, for example, a user can manually edit the mask of the area of interest to make it correct, thereby reducing erroneous judgments during the inspection.

The inspection system includes an alignment unit (e.g., alignment unit 130, computer, circuit) that aligns an image of an object captured by an imaging unit with a target image of the object, and a processing unit (cropping processing unit 150, computer, circuit) that performs processing to cut out a portion of a mask of an area of interest from a predetermined image, the alignment unit aligns an image of an object captured by an imaging unit at a first time point with the target image of the object, and a second image aligns an image of an object captured by an imaging unit at a second time point after the first time point with the target image, and the processing unit aligns an image of the object captured by an imaging unit at a second time point after the first time point with the target image, and the processing unit aligns the portion of the mask of the area of interest from the first image. the portion of the attention portion mask is cut out from the second image to obtain a first portion image (e.g., an attention portion image of a start image), the portion of the attention portion mask is cut out from the second image to obtain a second portion image (e.g., an attention portion image of an end image), and the portion of the attention portion mask is cut out from the target image to obtain a target portion image (e.g., an attention portion image of a target image), and the user interface unit receives the first image, the second image, the target image, the attention portion mask, the first portion image, and the second portion image (e.g., component 800), a result of the judgment by the judgment unit (e.g., component 810), and a result of confirmation of whether the result is correct. 13 ), and when editing of the attention place mask is accepted from the second part information, the processing unit 1213 performs a process of changing the size of the area of the attention place mask (for example, a process performed by the area enlargement/reduction unit 1213), a process of changing the location of the area of the attention place mask (for example, a process performed by the area movement unit 1214), a process of changing the angle of the area of the attention place mask (for example, a process performed by the area rotation unit 1215), and a process of combining the areas when there are multiple areas of the attention place mask (for example, a process performed by the area rotation unit 1215). At least one of the following processes is performed: a process of deleting a region of the attention point mask (for example, a process performed by the region deleting unit 1217); a process of adding or deleting an arbitrary pixel in the attention point mask (for example, a process performed by the pixel adding/deleting unit 1218); a process of changing the importance of a plurality of calculation methods used by the calculation unit to calculate the distance (for example, a process performed by the distance result combination ratio changing unit 1219); a process of displaying the result of the judgment made by the judgment unit when using the attention point mask being edited (for example, a process performed by the provisional judgment result display unit 1220); and a process of ending editing (for example, a process performed by the editing ending unit 1221).

Furthermore, the above-mentioned configurations may be modified, rearranged, combined, or omitted as appropriate without departing from the spirit and scope of the present invention.

It should be understood that an item in a list in the format "at least one of A, B, and C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Similarly, an item in a list in the format "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

100...Inspection system.

Claims (5)

an alignment unit that generates a first image by aligning an image of an object to be inspected taken by an imaging unit at a first time point with a target image that is an image of the object when the object is in a normal state, and a second image by aligning an image of the object to be inspected by an imaging unit at a second time point after the first time point with the target image;
A storage unit that stores a focus area mask that indicates which area of the object should be focused on;
a calculation unit that calculates a first distance representing a similarity between an image of the attention spot mask of the object in the first image and an image of the attention spot mask in the second image, and a second distance representing a similarity between the image of the attention spot mask in the second image and the image of the attention spot mask in the target image;
a determination unit that compares the two distances calculated by the calculation unit to determine whether the object is in a normal state;
a user interface unit that enables editing of the attention area mask stored in the storage unit;
A processing unit that performs processing to cut out the portion of the attention area mask from a predetermined image,
the storage unit generates and stores the attention area mask based on a difference between the first image and the second image, and stores the attention area mask edited by the user interface unit;
the processing unit cuts out a portion of the attention portion mask from the first image to obtain a first portion image, cuts out a portion of the attention portion mask from the second image to obtain a second portion image, and cuts out a portion of the attention portion mask from the target image to obtain a target portion image,
The user interface unit includes:
outputting the first image, the second image, the target image, the attention spot mask, the first spot image, the second spot image, a result of the determination by the determination unit, first component information for receiving a result of confirmation of whether the result is correct, and second component information for receiving editing of the attention spot mask;
When editing of the attention point mask is accepted from the second component information, at least one of the following processes is performed: a process of changing the size of the attention point mask area; a process of changing the location of the attention point mask area; a process of changing the angle of the attention point mask area; a process of combining the areas when there are multiple areas of the attention point mask; a process of deleting the area of the attention point mask; a process of adding or deleting an arbitrary pixel in the attention point mask; a process of changing the importance of multiple calculation methods used by the calculation unit to calculate distances; a process of displaying a result of judgment by the judgment unit when using the attention point mask being edited; and a process of ending editing.
Inspection system. The first time point is before a work related to the object is started,
The inspection system of claim 1 , wherein the second point in time is after the task is completed. a determination unit that determines an importance of each of a plurality of calculation methods used by the calculation unit to calculate the first distance and the second distance;
The inspection system of claim 1 . The calculation unit is
The histogram of pixel values of the first location image and the histogram of pixel values of the second location image are compared to calculate the distance, the average values of pixel values of the first location image and the second location image are compared to calculate the distance, the median values of pixel values of the first location image and the second location image are compared to calculate the distance, the sum of difference values calculated by calculating the difference between pixels at the same position between the first location image and the second location image is calculated as the distance, and the distance between the first location image and the second location image is calculated using a model adjusted based on learning data, and the calculated distances are multiplied by the importance determined by the determination unit, and the sum is set as the first distance,
The histogram of pixel values of the second location image and the histogram of pixel values of the target location image are compared to calculate a distance, the average pixel values of the second location image and the target location image are compared to calculate a distance, the median pixel values of the second location image and the target location image are compared to calculate a distance, the sum of difference values calculated for pixels at the same position between the second location image and the target location image is calculated as the distance, and the distance between the second location image and the target location image is calculated using a model adjusted based on learning data, and the second distance is calculated by multiplying each calculated distance by the importance determined by the determination unit and taking the sum of the multiplied values.
The inspection system of claim 3 . a registration unit generates a first image by registering an image of an object to be inspected photographed by the photographing unit at a first time point with a target image which is an image of the object when the object is in a normal state, and a second image by registering an image of the object to be inspected photographed by the photographing unit at a second time point after the first time point with the target image;
A storage unit stores a target area mask indicating which area of the object should be focused on;
a calculation unit calculates a distance representing a similarity between an image of the attention spot mask of the object in the first image and an image of the attention spot mask in the second image, and a distance representing a similarity between the image of the attention spot mask in the second image and the image of the attention spot mask in the target image;
a determination unit comparing the two distances calculated by the calculation unit to determine whether the object is in a normal state;
A user interface unit enables editing of the attention area mask stored in the storage unit;
A processing unit performs a process of cutting out the portion of the attention area mask from a predetermined image;
Including,
the storage unit generates and stores the attention area mask based on a difference between the first image and the second image, and stores the attention area mask edited by the user interface unit;
the processing unit cuts out a portion of the attention portion mask from the first image to obtain a first portion image, cuts out a portion of the attention portion mask from the second image to obtain a second portion image, and cuts out a portion of the attention portion mask from the target image to obtain a target portion image;
The user interface unit,
outputting the first image, the second image, the target image, the attention spot mask, the first spot image, the second spot image, a result of the determination by the determination unit, first component information for receiving a result of confirmation of whether the result is correct, and second component information for receiving editing of the attention spot mask;
When editing of the attention point mask is accepted from the second component information, at least one of the following processes is performed: a process of changing the size of the attention point mask area; a process of changing the location of the attention point mask area; a process of changing the angle of the attention point mask area; a process of combining the areas when there are multiple areas of the attention point mask; a process of deleting the area of the attention point mask; a process of adding or deleting an arbitrary pixel in the attention point mask; a process of changing the importance of multiple calculation methods used by the calculation unit to calculate distances; a process of displaying a result of judgment by the judgment unit when using the attention point mask being edited; and a process of ending editing.
Testing method.

Images