JP7494652B2 - Apparatus and method for preparing for visual inspection - Google Patents

Description

The present invention relates to an appearance inspection preparation device and an appearance inspection preparation method for setting inspection conditions before visual inspection of a product.

Visual inspection, which checks for scratches and chips on the surface of products on the production line, relies on a large amount of manpower and is a skilled job that requires a high level of skill. In recent years, the labor shortage has become more serious due to the decline in the working-age population, and there is a demand for the automation of visual inspection.

In relation to this background, in relation to automated visual inspection technology, as disclosed in Patent Document 1, for example, there have been proposals for a visual inspection device that uses an articulated robot to change the positions of the camera and lighting relative to the product in various ways, illuminates the product surface from a specified direction, and captures the light reflected from the product surface, as well as a technology that detects areas in the captured image that are clearly different from the surrounding area in terms of brightness, contrast, color, etc. as defects.

JP 2007-248241 A

In general, to accurately detect defects in visual inspection, it is necessary to appropriately adjust the imaging direction and lighting direction according to the orientation of the product surface and the target defect so as to obtain an image with good contrast between the defect and its surroundings.

In this regard, Patent Document 1 discloses a technology that determines multiple inspection points, which are the positions and orientations of the camera and lighting that capture the entire surface of the product, based on the inspection conditions, which are the imaging direction and lighting direction conditions that make it easier to inspect for defects. With the technology disclosed in Patent Document 1, the inspection conditions are not determined automatically, but are determined by the user.

A typical method for determining inspection conditions is the bright field method, in which the inspection conditions are set as conditions that the imaging direction and lighting direction must satisfy for the range in which the brightness of defects in an image of the product is above a certain level. A series of inspection points is generated, which is a combination of the joint angles of the manipulator that drives the camera, lighting, etc. so as to satisfy the imaging direction and lighting direction of the inspection conditions, and then an appearance inspection of the product is prepared. This series of inspection points makes it possible to carry out an appearance inspection of the product in an optimal manner.

When determining inspection conditions using the bright field method, the brightness threshold for determining whether a defect can be inspected is often the brightness at the point where the product appears clearly bright. In such cases, the range that satisfies the inspection conditions in a single captured image will be narrower than when the threshold is the brightness at the boundary between whether or not a defect can be inspected. In other words, more images are required to inspect the entire product, and the inspection time will be longer. Inspection time and the number of images taken are costs associated with inspection. It is desirable to have a shorter inspection time. It is desirable to have fewer images taken.

The present invention was made with the above problems in mind, and aims to generate a sequence of inspection points that allows products to be inspected at the lowest possible inspection cost.

To solve the above-mentioned problems, the visual inspection preparation device of the present invention is characterized by having an acquisition means for acquiring the positions of defects of multiple defective products related to a product, an inspection condition calculation means for outputting the imaging direction of a camera and the lighting direction of lighting, which are the inspection conditions of a robot that inspects the defects, based on the positions of the defects, and an inspection point sequence calculation means for calculating, based on the inspection conditions and the shape data of the product, an inspection point sequence which is a combination of the joint angles of a manipulator that drives the camera so as to satisfy the imaging direction of the inspection conditions, and the joint angles of a manipulator that drives the lighting so as to satisfy the lighting direction of the inspection conditions.

The visual inspection preparation method of the present invention is characterized by comprising the steps of acquiring the positions of defects in a plurality of defective products related to a product; outputting, based on the positions of the defects, an imaging direction by a camera and an illumination direction by a light, which are inspection conditions for a robot that inspects the defects; and calculating, based on the inspection conditions and shape data of the product, a sequence of inspection points, which is a combination of joint angles of a manipulator that drives the camera so as to satisfy the imaging direction of the inspection conditions, and joint angles of a manipulator that drives the light so as to satisfy the illumination direction of the inspection conditions.
Other means will be described in the description of the embodiment of the invention.

The present invention makes it possible to generate a sequence of inspection points that allows a product to be inspected at minimal inspection cost.

1 is a diagram showing an example of the configuration of a visual inspection preparation system according to an embodiment of the present invention; FIG. 2 is a diagram illustrating an example of a device configuration of an imaging device that images a defect. FIG. 13 is a diagram illustrating an example of a defective product information table. FIG. 2 is a diagram showing an example of a display operation screen in the user interface device. 13 is a flowchart showing a process of photographing a product based on the position and orientation of a defective product in an inspection condition calculation device and calculating inspection conditions. FIG. 13 is a diagram showing combinations of the imaging direction angle and the lighting direction angle. 13A and 13B are diagrams illustrating the following combinations of imaging direction angles and lighting direction angles. FIG. 13 is a diagram for explaining the inspection conditions for defective product A. FIG. 13 is a diagram for explaining the inspection conditions for defective product B. FIG. 13 is a diagram for explaining a product set of inspection conditions. FIG. 1 is a diagram showing the quasi-kinematics of an imaging device. 11 is a flowchart (part 1) showing a process of optimizing inspection points based on inspection conditions in an inspection point calculation device. 13 is a flowchart (part 2) showing a process of optimizing inspection points based on inspection conditions in the inspection point calculation device. 10A and 10B are diagrams illustrating parameters in a process for determining an attitude related to an inspection point and a camera attitude at the inspection point in an inspection point calculation device. 1 is a diagram for explaining an imaging direction and an illumination direction in an inspection point calculation device. FIG. FIG. 13 is a diagram for explaining an area that can be inspected from the 0th inspection point. FIG. 13 is a diagram for explaining an area that can be inspected from a first inspection point. FIG. 13 is a diagram for explaining an area that can be inspected from a second inspection point. FIG. 13 is a diagram for explaining the logical sum of areas that can be inspected from two inspection points.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.
FIG. 1 is a diagram showing an example of the configuration of a visual inspection preparation system 1 according to an embodiment of the present invention.
The visual inspection preparation system 1 includes a user interface device 11, an inspection condition calculation device 12, an inspection point sequence calculation device 13, an image capture device 14, and a storage device 15. This visual inspection preparation system 1 generates an inspection point sequence that can inspect the entire surface of a product at the minimum inspection cost. Here, the inspection cost includes the inspection time, the number of images captured, the cost of changing the posture of a manipulator, etc.

FIG. 2 is a diagram showing an example of the configuration of the imaging device 14 that images a defect.
The photographing device 14 is a device capable of photographing a product 5 placed on an inspection stage 145 by holding a camera 141 with a manipulator 143 to change the photographing direction and holding a light 142 with a manipulator 144 to change the illumination direction.

The camera 141 is a device that captures an optical image of the product 5 placed on the inspection stage 145. The manipulator 143 holds the camera 141 and changes the imaging direction. The light 142 irradiates the product 5 with visible light or the like. The manipulator 144 holds the light 142 and changes the lighting direction.

Returning to FIG. 1 for further explanation, the inspection condition calculation device 12 is composed of known elements such as a personal computer, and includes an inspection condition calculation unit 121. Based on each posture of the robot and the defect position input in advance, the inspection condition calculation unit 121 uses the imaging device 14 to capture images of defects in the defective product under various imaging and lighting conditions. The inspection condition calculation unit 121 then calculates imaging and lighting conditions under which the defect can be inspected from the images obtained by imaging. In other words, the inspection condition calculation device 12 functions as an inspection condition calculation means that outputs, based on the position of the defect in the defective product, the imaging direction by the camera and the illumination direction by the light, which are the inspection conditions of the robot that inspects the defect.

The inspection point sequence calculation device 13 is composed of known elements such as a personal computer. Based on the product shape and inspection conditions, the inspection point sequence calculation device 13 calculates an inspection point sequence for photographing the entire surface of the product 5 in the minimum number of times. Based on the inspection conditions and product shape data, the inspection point sequence calculation device 13 functions as an inspection point sequence calculation means for calculating an inspection point sequence which is a combination of the joint angles of the manipulator that drives the camera 141 so as to satisfy the imaging direction of the inspection conditions, and the joint angles of the manipulator that drives the lighting 142 so as to satisfy the lighting direction of the inspection conditions.

The storage device 15 is configured with a known large-capacity storage device such as a hard disk drive or a solid-state drive. The storage device 15 includes an inspection target storage unit 151, a defective product information storage unit 152, an inspection condition storage unit 153, and an inspection point sequence storage unit 154.

The inspection target memory unit 151 stores product shape data linked to product numbers. The product shape data is defined in a known format, such as STL (Stereolithography).

The defective product information storage unit 152 is a section that stores information described in the defective product information table 1521 shown in FIG. 3 described below, and stores individual number information for identifying defective products, product number information relating to the defective products, and the location of defects found in the defective products.

FIG. 3 is a diagram showing an example of the defective product information table 1521. As shown in FIG.
The defective product information table 1521 has a defective product individual number column, a product number column, and a defect position/posture information column.
The defective product individual number column is a column for storing an identification number for identifying a defective product.
The product number column is a column for storing a product number for identifying the version number of a product, etc.
The defect position/orientation information field contains three-dimensional position/orientation information of defects in defective products. The information stored in this field is expressed as a set of a three-dimensional vector representing a position and its orientation expressed by roll, pitch, yaw, etc. at that time, or as an expansion matrix used for affine transformation.

Returning to Fig. 1, the description will continue. The inspection condition storage unit 153 stores the conditions of the imaging direction and the illumination direction under which defects can be detected.
The inspection point sequence storage unit 154 stores, linked to the product number, an inspection point sequence set, which is a set of the positions and orientations of the camera 141 and the lighting 142 with respect to the product 5. The positions and orientations of the camera 141 and the lighting 142 are expressed as a pair of a three-dimensional vector representing the position and an orientation represented by roll, pitch, yaw, etc., or as an enlargement matrix used in affine transformation.

FIG. 4 is a diagram showing an example of the display operation screen 6 in the user interface device 11. As shown in FIG.
The user interface device 11 displays a display operation screen 6 including an inspection target input section 111 , a defect position input section 112 , an inspection condition display section 113 , and an inspection point display section 114 .

The inspection target input section 111 is a section for inputting product information for the inspection target, and is displayed as a pane on the display operation screen 6. A pane here refers to each divided display area in a display method in which the screen is divided into multiple display areas. The inspection target input section 111 displays a defective product individual number text box 1111 and a product number text box 1112. "A-000001" has been entered in the defective product individual number text box 1111. "p00001" has been entered in the product number text box 1112. The information entered here is stored in the defective product information table 1521 shown in FIG. 3.

The defect position input section 112 is a section for inputting the position of a defect in a defective product. By clicking anywhere on the surface of the product model displayed in the defect position input section 112, the operator can specify the position of a defect that frequently occurs in this product.

The inspection condition display unit 113 is a section that displays the inspection conditions in a graph. The vertical axis of this graph indicates the imaging direction, and the horizontal axis indicates the lighting direction.

In the inspection point display unit 114, inspection point positions 1141 to 1144 are displayed around the computer graphic product image. The inspection point display unit 114 generates these inspection point positions 1141 to 1144. Furthermore, when the user clicks on one of the inspection point positions 1141 to 1144, the inspection point display unit 114 displays the product surface area that can be inspected from the corresponding inspection point superimposed on the product image. In this case, since the user clicked on position 1141, the product surface area that can be inspected from the inspection point at position 1141 is displayed with dark hatching.

FIG. 5 is a flow chart showing a process in which the inspection condition calculation device 12 photographs the product 5 based on the position and orientation of the defective product and calculates the inspection conditions.
The inspection condition calculation unit 121 repeats the processes in steps S10 to S18 for all defective products, which include defects that occur in the products.

In step S11, the inspection condition calculation unit 121 acquires the product number j and defective individual number i of this defective product, and the defect position/posture information _A0 from the defective product information storage unit 152. The inspection condition calculation unit 121 functions as an acquisition unit that acquires the positions of defects of multiple defective products related to the product.
The inspection condition calculation unit 121 performs the processes in steps S12 to S16 by independently changing the angle α between the imaging direction and the normal to the defect by an increment Δα from 0 degrees to +90 degrees, and independently changing the angle β between the illumination direction and the normal to the defect by an increment Δβ from 0 degrees to +90 degrees. That is, the inspection condition calculation unit 121 repeats the processes for all combinations of the imaging direction and the illumination direction. Here, the normal to the defect refers to the normal to a facet (surface) that includes the defect when the product is represented by a polygon (polyhedron).

FIG. 6A is a diagram showing combinations of the imaging direction angle α and the illumination direction angle β.
The facet 51 is a facet (surface) when the product is represented as a polyhedron, and is also a facet at the defect position. The normal np is a normal to this facet 51 and also a normal to the defect.

The vector from facet 51 to camera 141 is the imaging direction. Here, each imaging direction is shown as a vector to camera 141. Camera 141 shown with a solid line indicates the current imaging direction. Camera 141 shown with a dashed line indicates a past or future imaging direction. The angle between normal np and the current imaging direction is α. The angular increment of the imaging direction is Δα. The inspection condition calculation unit 121 moves the angle α between the imaging direction and the normal to the defect from 0 degrees to +90 degrees in increments of Δα.

The vector from the facet 51 to the illumination 142 is the illumination direction. Here, each illumination direction is shown as a vector to the illumination 142. The illumination 142 shown by a solid line indicates the current illumination direction. The illumination 142 shown by a dashed line indicates a past or future illumination direction. The angle increment of the illumination direction is Δβ. The angle between the normal np and the current illumination direction is β. The inspection condition calculation unit 121 moves the angle β between the illumination direction and the normal to the defect by an increment of Δβ from -90 degrees to +90 degrees. The inspection condition calculation unit 121 then inspects all combinations of imaging directions and illumination directions.

FIG. 6B is a diagram showing the combination of the imaging direction angle α and the illumination direction angle β.
In FIG. 6B, the inspection condition calculation unit 121 keeps the direction of the illumination 142 fixed, increases the angle α between the imaging direction and the normal line of the defect by an increment Δα, and calculates and combines the next imaging direction.
After that, when the inspection condition calculation unit 121 combines a certain illumination direction with all the imaging directions, it increases the angle β between the illumination direction and the normal line of the defect by the step size Δβ. In this way, the inspection condition calculation unit 121 determines the range in which defect inspection is possible among the combinations of imaging directions and illumination directions.

5 to continue the explanation. In step S13, the inspection condition calculation unit 121 calculates the position and orientation of the camera 141 when the defect is captured at the center of the image. The case where the camera 141 captures the defect at the center of the image is when the camera 141 faces the defect directly. In this step, the position and orientation of the camera 141 facing the defect is defined as P _c (α) and is calculated using the following formula (1).

The position and orientation of the light 142 facing the defect is denoted as P _l (β) and calculated by the following equation (2).

Here, α is the angle between the imaging direction and the normal np of the defect, and is the angle of incidence of the optical axis of the camera 141. β is the angle between the lighting direction and the normal np of the defect, and is the angle of incidence of the light beam of the lighting 142. _Dc is the distance from the camera 141 to the product 5, and is defined as a specification of the camera 141. _Dl is the distance from the lighting 142 to the product 5, and is defined as a specification of the lighting 142.

In step S14, the inspection condition calculation unit 121 derives the joint angles of the manipulators 143 and 144 in this posture. That is, the inspection condition calculation unit 121 derives the joint angle of the manipulator 143 when the camera 141 has a given imaging direction, and derives the joint angle of the manipulator 144 when the lighting 142 has a given lighting direction.

When each joint angle of the manipulator 143 shown in FIG. 8 is θ (=[θ ₀ , θ ₁ , . . . , θ _n ]), the forward kinematics of the manipulator 143 is described by equation (3).

Here, J _n (Θ) is an augmented matrix representing the posture transformation caused by the rotation of each joint, and W is an augmented matrix representing the posture of the product 5 in the coordinate system set in the manipulator 143. Note that the thick dashed arrows in Fig. 8 represent each posture transformation.

By converting this forward kinematics into inverse kinematics using a common technique, it is possible to determine the joint angles of the manipulator 143 for photographing the product 5 in a given imaging direction. The joint angles of the manipulator 144 for illuminating the product 5 can be determined in a similar manner.

In step S15, the inspection condition calculation unit 121 operates the manipulators 143 and 144 so that the joint angles are as determined in step S13 to achieve the above posture, and photographs the product 5 with the defect at its center.

The inspection condition calculation unit 121 controls the image capture device 14 to obtain a group of images of the product 5, focusing on the defect in various imaging and lighting directions.

In step S16, if the inspection condition calculation unit 121 has not repeated the process for all combinations of imaging directions and illumination directions, the process returns to step S12. If the inspection condition calculation unit 121 has repeated the process for all combinations of imaging directions and illumination directions, the process proceeds to step S17.

In step S17, the inspection condition calculation unit 121 determines whether or not defect inspection is possible (inspection result) based on each captured image, and determines the inspection conditions by linking this defect inspection possible or not with information on the combination of the imaging direction and the illumination direction at the time of shooting. Details of the procedure are shown below. Note that in the following procedure, the inspection condition _Cji is expressed by two values, that is, defect inspection possible and defect inspection impossible, and the horizontal pixel position w and vertical pixel position h corresponding to each imaging direction are determined by the following formulas (4) and (5).

The inspection condition calculation unit 121 compares the brightness b _d0 of the defect at the center of the image with the brightness b s0 of the vicinity of the defect in an image captured at an angle α ₀ between the imaging direction and the normal to the defect, and an angle β ₀ between the illumination direction and the _normal to the defect. If this difference is equal to or greater than a certain value b _c , the inspection condition calculation unit 121 determines that inspection is possible, and if this difference is less than a certain value b _c , it determines that inspection is impossible. The inspection condition calculation unit 121 can determine whether or not a combination of imaging direction and illumination direction at the time of capture is capable of inspection based on the brightness difference between the images of the defect and its vicinity, and can obtain the inspection conditions.
If the defect is inspectable, the inspection condition C _ji (w ₀ , h ₀ ) corresponding to α ₀ and β ₀ is determined as shown in equation (6).

In step S18, the inspection condition calculation unit 121 determines whether the above process has been repeated for all defective individual products (individual product number i) belonging to product number j. If there are any defective individual products that have not been processed, the inspection condition calculation unit 121 returns to step S10, and if all defective individual products have been processed, the inspection condition calculation unit 121 proceeds to step S19.

In step S19, the inspection condition calculation unit 121 processes all defective individual products (individual product number i) belonging to product number j to obtain inspection conditions _Cji for defective products. Then, the inspection condition calculation unit 121 sets the intersection condition of the inspection conditions _Cji for defective products belonging to product number j as the inspection condition _Cj for product number j. The intersection condition is the logical product of a set of multiple inspection conditions, and is defined by the following formula (7).

Figures 7A to 7C show an explanation of the intersection condition. In the examples shown in Figures 7A to 7C, the dotted box represents the investigation range of the inspection condition, i.e., the boundary in the intersection image, and the gray hatched area represents the inspection condition.

The rectangle in Figure 7A indicates the space 3 of the imaging direction and lighting direction, and the hatched portion indicates the inspection condition 3a for defective product A of a certain product. The hatched portion in Figure 7B indicates the inspection condition 3b for another defective product B of the same product. In this case, the intersection condition 3c of defective products A and B shown in the gray hatched portion in Figure 7C represents the imaging direction and lighting direction conditions that allow inspection of either the defects present in defective product A or the defects present in defective product B. Inspection condition 3d surrounded by a dashed line in Figure 7C represents the imaging direction and lighting direction conditions that allow inspection of one or more of the defects present in defective product A or the defects present in defective product B.

The inspection condition calculation unit 121 outputs the inspection condition _Cj thus obtained to the inspection condition storage unit 153. By limiting the number of times the camera 141 takes pictures and the number of times the manipulator operates to the intersection set condition 3c among the space 3 consisting of all combinations of the image capturing directions and the illumination directions, the inspection condition calculation unit 121 can quickly inspect defects in known defective products.

Returning to Fig. 5, the description will continue. When the inspection condition calculation unit 121 finishes the process of step S19, the process of Fig. 5 ends.
5, the inspection condition calculation unit 121 sets the luminance at the boundary between whether or not a defect can be inspected as the threshold value of the luminance of the defect. This makes it possible to widen the range in which a single captured image satisfies the inspection conditions compared to the case where the luminance at a location where the product is clearly bright is set as the threshold value of the luminance of the defect.

In the process of FIG. 5, the inspection condition calculation unit 121 actually images the defective product and calculates the inspection conditions based on the defect position, but it may also be possible to virtually simulate the imaging of the defective product and calculate the inspection conditions based on the input defect position.

9A and 9B are flowcharts of the process in the inspection point sequence calculation device 13. In FIG.
In step S30, the inspection point sequence calculation device 13 selects one product number j stored in the inspection target storage unit 151, and obtains the product shape data M and the inspection condition _Cj for that product number j.

In step S31, the inspection point sequence calculation device 13 arranges cameras and lighting in various ways on a sphere that is centered on the center of the product shape data M and contains the product shape data M, and calculates a set of inspection points. The center of the sphere, that is, the center of the product shape data M, is found as the center of an axis-aligned bounding box (AABB) that contains the vertices that make up the product shape data M. The method of finding the AABB for a set of points is described below. The inspection point sequence calculation device 13 first finds the maximum and minimum values of each axis in the set of points. Next, it finds a rectangular parallelepiped whose vertex coordinates are the maximum or minimum values of each axis. The sphere that contains the product shape data M is found as a sphere with its center at the center of the AABB found in this way and its radius being the default working distance of the camera 141 and lighting 142.

The process of determining the position and orientation of the camera 141 and the lighting 142 relative to the product shape data M will be described using Figure 10. In Figure 10, a and b are angle parameters, and in a given Cartesian coordinate system, when the three axes are the x-axis, y-axis, and z-axis, a represents the rotation angle (pitch angle) around the y-axis, and b represents the rotation angle (yaw angle) around the z-axis.

The enlarged matrix P _c (a, b) expressing the position and orientation of the camera 141 in FIG. 10 is defined by equation (8).

A 3×3 matrix T _c (a, b) expressing the orientation of the camera 141 is defined by equation (9).

A three-dimensional vector p _c (a, b) expressing the position of the camera 141 is defined by equation (10).

Similar to the above-mentioned setting of the position and orientation of the camera 141, an enlarged matrix P _l (c, d) expressing the position and orientation of the light 142 is defined by equation (11) using a pitch angle c and a yaw angle d.

A 3×3 matrix T _l (c, d) representing the orientation of the illumination 142 is defined by equation (12).

A three-dimensional vector p _l (c, d) representing the position of the light 142 is defined by the following equation (13).

According to the above formula, the pitch angle a of the camera 141 and the pitch angle c of the lighting 142 are given in constant increments from 0 degrees to 90 degrees, and the yaw angle b of the camera 141 and the yaw angle d of the lighting 142 are given in constant increments from 0 degrees to 360 degrees. This makes it possible to automatically calculate multiple inspection points on a sphere that contains the product 5.

Returning to Fig. 9A, the inspection point sequence calculation device 13 repeats steps S32 to S35 for the calculated inspection points.
In step S33, the inspection point sequence calculation device 13 sets the position and orientation of the camera 141 as _Pc ( _a0 , _b0 ) and the position and orientation of the illumination 142 as _Pl ( _c0 , _d0 ) for an inspection point randomly selected from the calculated inspection points. In this process, the imaging direction and illumination direction are determined for the inspection point defined by _Pc and _Pl , and then an inspectable facet is determined. Here, a facet refers to each surface when the product shape data M is represented as a polygon (polyhedron).

Here, the process of determining an inspectable facet will be described. The inspection point sequence calculation device 13 selects a facet f from the product shape data M. The imaging direction of this facet f is a vector from the position of the camera 141 to the center of gravity of the facet f. The illumination direction of the facet f is a vector from the position of the illumination 142 to the center of gravity of the facet f. In this way, the inspection point sequence calculation device 13 determines the incidence angle α _f of the optical axis of the camera 141 and the incidence angle β _f of the light beam of the illumination 142. The relationship between the imaging direction, illumination direction, and facet is shown in FIG. 11. At this time, the incidence angle α _f of the optical axis of the camera 141 is defined by equation (14).

The angle of incidence _βf of the light ray of the illumination 142 is defined by equation (15).

In addition, in formula (14) and formula (15), n(f) is a unit normal vector of f. g(f) is a position vector of the center of gravity of f. N(v) is a unit vector of vector v. The inspection point sequence calculation device 13 obtains pixel coordinates _wf and _hf in the inspection conditions for the incidence angle _αf of the optical axis of the camera 141 and the incidence angle _βf of the light ray of the illumination 142. At this time, if the inspection condition _Cj ( _wf , _hf ) obtained in step S30 is 1, the inspection point sequence calculation device 13 determines that the facet can be inspected. The inspection point sequence calculation device 13 repeats the above process for all facets to obtain whether the facet can be inspected. In this way, by determining whether the facet can be inspected by the inspection condition _Cj ( _wf , _hf ), the set of facets to be inspected is calculated and narrowed down. The reason for narrowing down the set of facets to be inspected to inspectable facets here is to extract those facets that are not obstructed by the manipulators 143 and 144 in step S34.

9A , in step S34, the inspection point sequence calculation device 13 calculates the orientation of the manipulator 143 that matches the position and orientation _Pc of the camera 141, and calculates the orientation of the manipulator 144 that matches the position and orientation _Pl of the light 142.

For the postures of the manipulators 143 and 144 obtained here, if the manipulators 143 and 144 have an intersection with a line segment connecting the camera 141 or lighting 142 and an inspectable facet of the product shape data M, that inspection point is excluded from the inspection points because the inspectable facet of the product 5 is blocked by the manipulators 143 and 144. That is, the inspection point sequence calculation means extracts inspection points on the sphere that contains the product 5 at which the manipulators 143 and 144 do not block the inspectable facet of the product 5. Note that even if the manipulators 143 and 144 block a facet of the product 5 when they take a specified robot posture, if that facet is an uninspectable facet, that robot posture does not pose a problem. This improves the degree of freedom of the robot posture and makes it possible to further reduce the cost of changing the posture of the robot.

When the inspection point sequence calculation device 13 has repeated the processing of all the calculated inspection points in step S35, it proceeds to step S36, and if there are any unprocessed inspection points, it returns to step S32.

In step S36, the inspection point sequence calculation device 13 determines the facet set S as an area that can be inspected from all the inspection points. The area that can be inspected from all the inspection points is defined by the following formula (16) using a facet set Fu _i that can be inspected at a certain inspection point u _i (i=0, 1, 2, ...).

The following processing steps S37 to S41 will be described with reference to Figures 12A to 12D. For a certain inspection point u _i , a facet set F'u _i that can be inspected among facets contained in facet set S is obtained by the following formula (17).

In step S37, the inspection point sequence calculation device 13 obtains this for all inspection points u _i (i=0, 1, 2, ...). The facet set F'u _i with the largest total area is added to the inspection point sequence. The initial value of the inspection point sequence is empty.
FIG. 12A discloses a facet set _F'u0 at an inspection point _u0 when a facet set S is an area that can be inspected from all inspection points.
FIG. 12B discloses a facet set F′u ₁ at an inspection point u ₁ when the facet set S is an area that can be inspected from all inspection points.
FIG. 12C discloses a facet set F′u ₂ at an inspection point u ₂ when the facet set S is an area that can be inspected from all inspection points.
As shown in FIG. 12B, in the example disclosed in FIGS. 12A to 12C, since F′u ₁ has the largest number of elements, in this example, check point u ₁ is added to the check point string.

The inspection point sequence calculation device 13 repeats the processes of steps S38 to S41 until the facet set S becomes an empty set.
In step S39, the inspection point sequence calculation device 13 finds an inspection point sequence that minimizes the posture conversion cost f( _uj , _uk ) when moving from the inspection point _uj at the end of the inspection point sequence to another inspection point _uk . Here, when the total area of the facet set _F'ui is A( _F'ui ), the posture conversion cost f can be found by equation (18).

where ω( _uj , _uk ) is the maximum value of the rotation amount of each joint when the manipulator is operated from the inspection point _uj to _uk . Assuming that the rotation speed of each joint is a predetermined value, the rotation amount of each joint is proportional to the time for the posture change of the robot.
In step S40, the inspection point sequence calculation device 13 updates the facet set S in accordance with the following formula (19).

An example of the updated facet set S is shown in Fig. 12D. The facet set S in Fig. 12D is obtained by calculating the difference between the set S of all facets that can be inspected from the inspection points disclosed in Fig. 12A to Fig. 12C and the set included in the set _F'u1 of facets that can be inspected from the inspection point _u1 disclosed in Fig. 12B. Also, _F'u0 in Fig. 12D is obtained by calculating the logical product of the facet set S in Fig. 12D and the facet set _F'u0 that can be inspected from the inspection point _u0 .

Returning to the explanation of FIG. 9B, in step S41, if the inspection point sequence calculation device 13 repeats the process until the facet set S becomes an empty set, it proceeds to step S42, and if the facet set S is not an empty set, it returns to step S38.

In step S41, the inspection point sequence calculation device 13 optimizes the robot operation through the inspection point sequence determined by the loop processing of steps S38 to S41 using a rapidly-exploring random tree (RRT) method or the like. That is, the inspection point sequence calculation device 13 creates a set of facets that constitute the shape data of the product 5 that can be inspected from the imaging direction and lighting direction that satisfy the inspection conditions, and calculates an inspection point sequence that can inspect the set with the minimum posture conversion cost.
Thereafter, the inspection point sequence calculation device 13 outputs the obtained inspection point sequence and robot operation, and ends the process of FIG. 9B.

As disclosed above, the visual inspection preparation system 1 sets inspection conditions that enable inspection of defects without overlooking them based on actual inspection data of defective products, and automatically generates an inspection point sequence based on this inspection condition. In this way, the visual inspection preparation system 1 automatically generates an inspection point sequence that enables inspection of the product 5 with the minimum number of photographs, and prepares for visual inspection of the product. This inspection point sequence makes it possible to optimally carry out visual inspection of the product.

(Modification)
The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to those having all of the configurations described. It is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

In each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are interconnected.

As modified examples of the present invention, for example, the following (a) and (b) are available.
(a) The imaging device may be configured to change the imaging direction and lighting direction relative to the product by fixing the camera, gripping and driving the lighting with a first manipulator, and gripping/carrying the product with a second manipulator. In such a modified example, a combination of the joint angle of the first manipulator that drives the lighting and the joint angle of the second manipulator that grips/carries the product is the inspection point sequence.
(b) The imaging device may be configured to change the imaging direction and the illumination direction relative to the product by fixing the illumination, gripping and driving the camera with a first manipulator, and gripping/carrying the product with a second manipulator. In such a modified example, the combination of the joint angle of the first manipulator that drives the camera and the joint angle of the second manipulator that grips/carries the product is the inspection point sequence.

1. Visual inspection preparation system (visual inspection preparation device)
11 User interface device 111 Inspection target input section 1111 Defective product individual number text box 1112 Product number text box 112 Defect position input section 113 Inspection condition display section 114 Inspection point display section 1141 Position 12 Inspection condition calculation device (inspection condition calculation means)
121 Inspection condition calculation unit 13 Inspection point sequence calculation device (inspection point sequence calculation means)
14 Photographing device 141 Camera 142 Lighting 143, 144 Manipulator 145 Inspection stage 15 Storage device 151 Inspection target storage section 152 Defective product information storage section 1521 Defective product information table 153 Inspection condition storage section 154 Inspection point string storage section 5 Product 6 Display operation screen 61 Defective product information input pane 611 Defective product individual number 612 Product number text box 613 Defect position information text box 62 Inspection condition pane 63 Inspection point pane 631 Inspection point

Claims (9)

An acquisition means for acquiring positions of defects of a plurality of defective products related to the product;
an inspection condition calculation means for outputting an imaging direction by a camera and an illumination direction by lighting, which are inspection conditions of a robot that inspects the defect, based on the position of the defect;
an inspection point sequence calculation means for calculating an inspection point sequence which is a combination of joint angles of a manipulator that drives the camera so as to satisfy the imaging direction of the inspection conditions and joint angles of a manipulator that drives the lighting so as to satisfy the lighting direction of the inspection conditions, based on the inspection conditions and the shape data of the product;
A visual inspection preparation device comprising: An acquisition means for acquiring positions of defects of a plurality of defective products related to the product;
an inspection condition calculation means for outputting an imaging direction by a camera and an illumination direction by lighting, which are inspection conditions of a robot that inspects the defect, based on the position of the defect;
an inspection point sequence calculation means for calculating, based on the inspection conditions and the shape data of the product, an inspection point sequence which is a combination of joint angles of a manipulator that grips the product with respect to a fixed camera so as to satisfy the imaging direction of the inspection conditions, and joint angles of a manipulator that drives the lighting so as to satisfy the lighting direction of the inspection conditions;
A visual inspection preparation device comprising: An acquisition means for acquiring positions of defects of a plurality of defective products related to the product;
an inspection condition calculation means for outputting an imaging direction by a camera and an illumination direction by lighting, which are inspection conditions of a robot that inspects the defect, based on the position of the defect;
an inspection point sequence calculation means for calculating, based on the inspection conditions and shape data of the product, an inspection point sequence which is a combination of joint angles of a manipulator that grips the product and satisfies the lighting direction of the inspection conditions with respect to fixed lighting, and joint angles of a manipulator that drives the camera and satisfies the imaging direction of the inspection conditions;
A visual inspection preparation device comprising: the inspection condition calculation means sets a threshold value of defect brightness as a brightness at a boundary between whether or not a defect can be inspected;
4. The visual inspection preparation device according to claim 1, wherein the visual inspection preparation device is a device for performing visual inspection. the inspection condition calculation means calculates, as the inspection condition, a product set of conditions under which the defect can be inspected;
4. The visual inspection preparation device according to claim 1, wherein the visual inspection preparation device is a device for performing visual inspection. the inspection point sequence calculation means calculates inspection points on a sphere including the product that meet the inspection conditions;
4. The visual inspection preparation device according to claim 1, wherein the visual inspection preparation device is a device for performing visual inspection. the inspection point sequence calculation means extracts inspection points on a sphere that encompasses the product, the inspection points being such that the manipulator does not obstruct any inspectable facet of the product;
7. The visual inspection preparation device according to claim 6. the inspection point sequence calculation means creates a set of facets that can be inspected from an imaging direction and an illumination direction that satisfy the inspection conditions, from among the facets that constitute the shape data of the product, and calculates an inspection point sequence that can inspect the set with a minimum posture conversion cost.
4. The visual inspection preparation device according to claim 1, wherein the visual inspection preparation device is a device for performing visual inspection. obtaining defect locations for a plurality of defective items associated with the product;
outputting an imaging direction by a camera and an illumination direction by lighting, which are inspection conditions of a robot that inspects the defect, based on the position of the defect;
calculating an inspection point sequence, which is a combination of joint angles of a manipulator that drives the camera so as to satisfy the imaging direction of the inspection conditions and joint angles of a manipulator that drives the lighting so as to satisfy the lighting direction of the inspection conditions, based on the inspection conditions and shape data of the product;
A method for preparing for visual inspection, comprising:

Images