JP6326243B2 - Image processing apparatus and three-dimensional shape measuring apparatus - Google Patents

Description

  Embodiments described herein relate generally to an image processing apparatus and a three-dimensional shape measurement apparatus that detect a predetermined object from an image.

  Conventionally, various image processing techniques have been developed as techniques for detecting a predetermined object from an image. For example, Japanese Patent Laying-Open No. 2005-29312 (Patent Document 1) discloses a technique for specifying a partial image similar to a registered image (template image) from an input image by pattern matching using a normalized correlation method. Yes.

  A technique for detecting a predetermined object from this type of image can be applied to various scenes. For example, consider a case where an object (work) in which a plurality of object elements (for example, bumps) are arranged based on design information such as CAD information (Computer Aided Design). In this case, on the basis of an object image (work image) that is an image obtained by imaging the manufactured object, the shape and arrangement position of each bump of the manufactured work may be compared and evaluated with design information.

  At this time, the coordinates (hereinafter referred to as element design coordinates) indicating the target arrangement position of each object element in the coordinate system (hereinafter referred to as design coordinate system) for defining the position on the design information, and the object image It is important to associate the coordinates (hereinafter referred to as element actual coordinates) indicating the position of each object element actually disposed in a coordinate system (hereinafter referred to as actual coordinate system) for defining the position of Hereinafter, information for associating element design coordinates and element actual coordinates is referred to as alignment information. The alignment information may be information for associating the element design coordinates with the element actual coordinates, and includes, for example, information on the actual coordinates corresponding to the design coordinates of the reference point (hereinafter referred to as reference coordinates) included in the design information. It is.

  A conventional technique for detecting a predetermined object from an image can be used as a technique for acquiring this alignment information. For example, by setting a predetermined template image as a registered image in advance, alignment information can be obtained by performing pattern matching by a normalized correlation method on the work image.

JP-A-2005-293152

  However, the technique for detecting a predetermined object from the above image is insufficient as a technique for acquiring alignment information.

  For example, when acquiring alignment information using a straight line approximation method, for example, when a plurality of bumps are two-dimensionally arranged, a minimum of 2 straight lines along the upper and lower ends and the left and right ends of the plurality of bumps in the work image are used. It is good to extract by the multiplication method and make an intersection into the position of an alignment mark. However, the linear approximation method is not convenient because of high preconditions, such as being weak against changes in light and darkness of the work image and weak against the presence of dust in the image.

  In addition, the normalized correlation method is robust against changes in light and dark, but is weak against work image hiding (occlusion) and work image rotation. When using the normalized correlation method, it is necessary to prepare a registered image in advance. However, selection of a registered image requires a certain technique and knowledge, and the detection accuracy greatly depends on the skill level of the registered image selector. In addition, since the normalized correlation method calculates the registered image and the work image in comparison with each pixel, a huge amount of calculation is required and the calculation time becomes long. Furthermore, in order to cope with the variety of bump arrangement patterns in the normalized correlation method, it is necessary to take a large registered image, so that the calculation time becomes longer.

  The present invention has been made in consideration of the above-described circumstances, and an object thereof is to provide an image processing apparatus and a three-dimensional shape measurement apparatus that can easily acquire alignment information without using a registered image.

In order to solve the above-described problem, an image processing apparatus according to an embodiment of the present invention defines a position on the design information from design information for producing an object in which a plurality of object elements are arranged. In the design coordinate system for obtaining the element design coordinates, which are the design coordinates indicating the target arrangement position of each object element, and for each object element, the element design coordinates and a plurality of other element design coordinates A design vector generation unit that generates a design vector group composed of relative vectors between them, and a position on the object image based on an object image obtained by imaging the object created based on the design information An actual coordinate acquisition unit for acquiring element actual coordinates, which are actual coordinates indicating the actual arrangement position of each object element, and each element element in the object image. Coordinates and several other The real vector generation unit 23 for generating a real vector group composed of relative vectors between the real coordinates and the design vector group and the real vector group are compared to design the reference point included in the design information. obtain Bei alignment processing unit having, an alignment information acquisition unit for identifying the real coordinates corresponding to the coordinates. The alignment unit further includes an integrated vector generation unit that generates an integrated vector by integrating the design vector group. In addition, the alignment information acquisition unit identifies the real coordinates corresponding to the design coordinates of the reference point by comparing the integrated vector and the real vector group .

  According to the image processing apparatus and the three-dimensional shape measurement apparatus according to the present invention, alignment information can be easily acquired without using a registered image.

1 is a block diagram showing a configuration example of an image processing apparatus according to a first embodiment of the present invention. (A) is explanatory drawing which shows an example of a CAD image, (b) is explanatory drawing which shows an example of the target object image which imaged the target object produced based on CAD information. Explanatory drawing which shows an example of the mode at the time of producing | generating a design vector group by a design vector production | generation part. (A) is explanatory drawing which shows an example of an element design coordinate, (b) is explanatory drawing which shows an example of a design vector group. (A) is explanatory drawing which shows an example of an element real coordinate, (b) is explanatory drawing which shows an example of a real vector group. 3 is a flowchart showing a procedure when the CPU of the image processing apparatus shown in FIG. 1 easily obtains real coordinate information corresponding to reference coordinates without using a registered image. The block diagram which shows the example of 1 structure of the image processing apparatus which concerns on 2nd Embodiment of this invention. (A) is explanatory drawing which shows an example of a design vector group, (b) is explanatory drawing which shows an example of likelihood information for every integration vector and integration element vector, (c) is the integration produced | generated by two design vector groups The figure which shows an example of the likelihood information for every vector 4 and integration element vector, (d) is explanatory drawing which shows an example of two design vector groups used as the origin of the integration vector shown in (c). The figure for demonstrating the mode of the matching process using the integrated vector performed by the alignment information acquisition part. The flowchart which shows the procedure at the time of acquiring easily the information of the real coordinate corresponding to a reference | standard coordinate using an integrated vector, without using a registration image by CPU of the image processing apparatus shown in FIG. FIG. 11 is a subroutine flowchart showing a procedure of matching processing using an integrated vector executed by the alignment information acquisition unit in step S <b> 12 of FIG. 10. FIG. 11 is a subroutine flowchart showing another procedure of matching processing using an integrated vector executed by the alignment information acquisition unit in step S12 of FIG. Explanatory drawing which shows an example of a mode that the real vector group of arbitrary element real coordinates REALj and the integrated vector were compared when the target object in a target object image has a predetermined rotation angle. Explanatory drawing which shows an example of a mode which calculates | requires angle | corner (theta) k = (theta) 1 which real vector k = 1 and the integration element vector corresponding to this real vector. 14A is a diagram for explaining a state in which the integrated vector is rotationally corrected by half of θ1 in the example shown in FIG. 14, and FIG. 15B is a diagram using the integrated vector after the rotational correction shown in FIG. Explanatory drawing which shows an example of a mode which calculates | requires angle | corner (theta) k = (theta) 2 which real vector k = 2 and the integration element vector corresponding to this real vector. The block diagram which shows the example of 1 structure of the image processing apparatus which concerns on 3rd Embodiment of this invention. (A) is explanatory drawing which shows the triangle as an example of an extraction target object, (b) is between the feature point of the extraction target previously memorize | stored in a storage medium according to the shape of the extraction target object, and the reference point of a detection target figure An explanatory view showing an example of a vector, (c) is an explanatory view showing an example of a voting process performed on a reference point based on a feature point of an input image, and (d) is a shape of various extraction objects. Explanatory drawing which shows the example of. Explanatory drawing which shows an example of the mode at the time of extracting the extraction object of circular shape by generalized Hough transformation. The flowchart which shows the procedure of the acquisition process of the element real coordinate performed by the real coordinate acquisition part of the image processing apparatus shown in FIG. 16 by step S3 of FIG. The subroutine flowchart which shows the procedure of the binarization process and labeling process which are performed by the real coordinate acquisition part by FIG.19 S52. The figure for demonstrating the binarization process and labeling process in the procedure shown in FIG. (A) is explanatory drawing which shows an example of a mode that three prediction feature points which should co-occur with respect to one feature point (x, y) are extracted when an object element is circular shape. FIG. 7B is an explanatory diagram illustrating an example of the direction of the normal direction of the edge of the feature point to be co-extracted extracted in the case illustrated in FIG. The figure for demonstrating that co-occurrence GHT is applicable even if a target object element is arbitrary shapes. The flowchart which shows the other procedure of the acquisition process of the element real coordinate performed by the real coordinate acquisition part of the image processing apparatus shown in FIG. 16 by step S3 of FIG. FIG. 23 is a subroutine flowchart showing a procedure of co-occurrence GHT image generation processing executed by a real coordinate acquisition unit in step S80 of FIG. The block diagram which shows the example of 1 structure of the solid-shape measuring apparatus containing the image processing apparatus which concerns on 4th Embodiment of this invention. The perspective view which shows the example of 1 structure of a focus position change part. The block diagram which shows the example of 1 structure of the image processing apparatus which concerns on 4th Embodiment of this invention.

  Embodiments of an image processing apparatus and a three-dimensional shape measurement apparatus according to the present invention will be described with reference to the accompanying drawings.

(1. First embodiment)
(1-1. Configuration)
FIG. 1 is a block diagram showing a configuration example of an image processing apparatus 10 according to the first embodiment of the present invention.

  As illustrated in FIG. 1, the image processing apparatus 10 includes an input unit 11, a display unit 12, a network connection unit 13, a storage unit 14, and a control unit 15.

  The input unit 11 is configured by a general input device such as a keyboard, a trackball, a touch panel, and a numeric keypad, and outputs an operation input signal corresponding to a user operation to the control unit 15.

  The display unit 12 includes a general display output device such as a liquid crystal display or an OLED (Organic Light Emitting Diode) display, and displays various images such as a CAD image under the control of the control unit 15.

  The network connection unit 13 implements various information communication protocols according to the network form. The network connection unit 13 connects the image processing apparatus 10 and other electrical devices according to these various protocols. For this connection, an electrical connection via an electronic network can be applied.

  Here, the electronic network means an information communication network using telecommunications technology in general, and includes a telephone communication line network, an optical fiber communication network, a cable communication network, and a satellite communication network in addition to a wireless / wired LAN and an Internet network. Including.

  The storage unit 14 includes a recording medium that can be read and written by the CPU of the control unit 15 such as a magnetic or optical recording medium or a semiconductor memory. The storage unit 14 stores design information (for example, CAD information) in advance. The CAD information is design information for producing an object (work) on which a plurality of object elements (for example, bumps) are arranged. The design information may be downloaded via the network by the network connection unit 13.

  The control unit 15 includes a storage medium such as a CPU, a RAM, and a ROM, and controls the image processing apparatus 10 according to an alignment processing program stored in the storage medium.

  More specifically, the CPU of the control unit 15 loads an alignment processing program stored in a storage medium such as a ROM and data necessary for the execution of the program into the RAM, and registers the registered image according to the program. A process for easily acquiring information (alignment information) of actual coordinates corresponding to the reference coordinates (design coordinates of the reference points) is executed without using.

  The RAM of the control unit 15 provides a work area for temporarily storing programs and data executed by the CPU. The storage medium such as the ROM of the control unit 15 stores an alignment processing program and various data necessary for executing the program.

  A storage medium such as a ROM has a configuration including a recording medium readable by a CPU, such as a magnetic or optical recording medium or a semiconductor memory, and a part of programs and data in the storage medium. Or you may comprise so that all may be downloaded via an electronic network.

  As shown in FIG. 1, the CPU of the control unit 15 functions as at least the alignment processing unit 20 according to the alignment processing program. The alignment processing unit 20 uses a required work area of the RAM as a temporary storage location for data. In addition, this function realization part may be comprised by hardware logics, such as a circuit, without using CPU.

  FIG. 2A is an explanatory diagram illustrating an example of a CAD image, and FIG. 2B is an explanatory diagram illustrating an example of an object image obtained by capturing an object manufactured based on CAD information.

  The alignment processing unit 20 includes a design vector generation unit 21, a real coordinate acquisition unit 22, a real vector generation unit 23, and an alignment information acquisition unit 24.

  In general, the alignment processing unit 20 associates element design coordinates indicating the target arrangement position of the object element of the design image with element actual coordinates where the object element mounted on the object is actually arranged. Hereinafter, the configuration of the alignment processing unit 20 will be described in detail with reference to FIGS.

  FIG. 3 is an explanatory diagram showing an example of a state when a design vector group is generated by the design vector generation unit 21. FIG. 3 shows an example of the upper left corner of the array when the object elements are arrayed two-dimensionally.

  FIG. 4A is an explanatory diagram showing an example of element design coordinates, and FIG. 4B is an explanatory diagram showing an example of a design vector group.

  The design vector generation unit 21 is an element that is a design coordinate indicating a target arrangement position of each object element in a design coordinate system for defining a position on the design information in advance from the design information stored in the storage unit 14. Get design coordinates. As the element design coordinates of the object element, for example, representative coordinates such as the center and the center of gravity of the area occupied by the object element are used. In addition, the design vector generation unit 21 generates a design vector group including a relative vector (design vector) between its own coordinates and element design coordinates of a plurality of other object elements for each of a plurality of element design coordinates. .

  More specifically, as shown in FIG. 3, the design vector generation unit 21 first acquires the element design coordinates of the object element belonging to the predetermined area 31. Next, for each of the acquired element design coordinates, another element design coordinate located within the predetermined distance 32 is acquired. And the design vector production | generation part 21 produces | generates the design vector group which consists of a relative vector (design vector) between an own coordinate and the acquired other element design coordinate for every some element design coordinate (FIG.4 (b) )reference).

  In the following description, the element design coordinates are appropriately referred to as CADi (where i represents 1, 2,..., L, l represent natural numbers) (see FIGS. 4A and 4B).

  The CAD information is naturally generated before the object is manufactured. For this reason, the design vector generation unit 21 can generate a design vector group from CAD information in advance before an object image is captured.

  FIG. 5A is an explanatory diagram illustrating an example of element real coordinates, and FIG. 5B is an explanatory diagram illustrating an example of a real vector group.

  The real coordinate acquisition unit 22 is a predetermined region 31 in the real coordinate system for defining a position on the target image based on the target image captured by the imaging unit 1 based on the target created based on the design information. Element real coordinates, which are real coordinates indicating the actual arrangement positions of the respective object elements belonging to, are acquired. In the following description, the element actual coordinates are appropriately referred to as REALj (where j is 1, 2,..., M, m are natural numbers and represent the total number of element actual coordinates belonging to the predetermined region 31) (FIG. 5). (Refer to (a) and (b)).

  The imaging unit 1 is composed of, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

  The actual coordinate acquisition unit 22 may acquire the element actual coordinates from the object image (so-called luminance image) itself captured by the imaging unit 1, or pattern matching (for example, normalized correlation method, general Element actual coordinates may be acquired from an image subjected to feature amount extraction processing such as generalized Hough transform (GHT, Generalized Hough Transform, etc.), or an image or target subjected to feature amount extraction processing on an object image Element actual coordinates may be acquired from an image obtained by binarizing the object image itself.

  Moreover, after performing a labeling process with respect to a binary image, the representative coordinate of each label may be acquired as an element actual coordinate, and the distance image of a target object may be sufficient. That is, the actual coordinate acquisition unit 22 can acquire the element actual coordinates based on the object image. When labeling processing is performed, the element actual coordinates can be acquired if there is label information. For this reason, the object image may be deleted from the storage medium after the labeling process is performed.

  For each element actual coordinate, the actual vector generation unit 23 acquires another element actual coordinate located within the predetermined distance 32, and uses a relative vector (real vector) between the own coordinate and the acquired other element actual coordinates. A real vector group is generated (see FIG. 5B). In the following description, real vectors constituting the real vector group are referred to as real vectors k (where k is 1, 2,..., N, n represent natural numbers) as necessary.

  The alignment information acquisition unit 24 compares (collates) the design vector group generated for each of the plurality of object elements with the actual vector group generated for each element actual coordinate. Then, based on the comparison result, the actual coordinates corresponding to the reference coordinates included in the design information are specified.

  The alignment information acquisition unit 24 acquires reference coordinate information from the design information. The reference coordinates may be any of element design coordinates or may be design coordinates that are not element design coordinates. When the reference coordinates are not element design coordinates, the reference coordinates can be defined by a relative position with respect to predetermined element design coordinates, for example.

  When setting element design coordinates as reference coordinates, it is preferable to select a design vector group of reference coordinates that has a characteristic configuration compared to design vector groups of other element design coordinates. For example, when the object elements are two-dimensionally arranged, any of the element design coordinates located at the four corners may be set as the reference coordinates. This is because the design vector group of the element design coordinates located at the four corners is composed of almost one quadrant of the vector, and often has a characteristic configuration compared to the design vector group of the other element design coordinates. In the following description, an example in which the upper left element design coordinate CAD1 is used as the reference coordinate (alignment mark) will be described.

  In this example, the alignment information acquisition unit 24 is targeted to extract REAL1 corresponding to CAD1 from the object image. Here, as is clear by comparing the top of the real vector FIG. 4B and the top of FIG. 5B, the design vector group of CAD1 is similar to the real vector group of REAL1. high. Therefore, the alignment information acquisition unit 24 can easily extract REAL1 corresponding to CAD1 based on the similarity between the design vector group and the real vector group.

(1-2. Operation)
Next, an example of the operation of the image processing apparatus 10 according to the first embodiment will be described.

  FIG. 6 is a flowchart illustrating a procedure when the CPU of the image processing apparatus 10 illustrated in FIG. 1 easily acquires information on real coordinates corresponding to the reference coordinates without using a registered image. In FIG. 6, reference numerals with numbers added to S indicate steps in the flowchart. This procedure starts with CAD information stored in advance in the storage unit 14.

  First, in step S <b> 1, the design vector generation unit 21 acquires element design coordinates of each object element from the design information stored in the storage unit 14. Next, in step S2, the design vector generation unit 21 generates a design vector group for each of a plurality of element design coordinates.

  Note that steps S1 and S2 can be performed in advance before the production of the object.

  Next, in step S <b> 3, the actual coordinate acquisition unit 22 acquires the element actual coordinates of each object element based on the object image. Next, in step S4, the real vector generation unit 23 generates a real vector group for each element real coordinate.

  In step S <b> 5, the alignment information acquisition unit 24 compares the design vector group and the real vector group, and specifies the real coordinates corresponding to the reference coordinates.

  For example, in the example shown in FIGS. 4 and 5, the alignment information acquisition unit 24 includes each design vector group of the element design coordinates CAD1, CAD2,... And each of the actual coordinates of the element actual coordinates REAL1, REAL2,. By comparing the vector group, it can be easily specified that the design vector group of CAD1 and the real vector group of REAL1 correspond, that is, that CAD1 and REAL1 correspond.

  In this case, when the information indicating that the design coordinate of the reference point (alignment mark) is CAD1 is acquired from the design information, the alignment information acquisition unit 24 easily specifies that the actual coordinate corresponding to the reference coordinate is REAL1. can do. Even when information indicating that the reference coordinate is a predetermined relative position from CAD 1 is acquired from the design information, the actual coordinate corresponding to the reference coordinate can be easily specified using the information on the predetermined relative position.

  With the above procedure, it is possible to easily acquire information on real coordinates corresponding to the reference coordinates without using a registered image.

  The image processing apparatus 10 according to the present embodiment can specify the real coordinates corresponding to the reference coordinates by using the design information and comparing the design vector group and the real vector group. For this reason, it is possible to easily obtain the element actual coordinate information corresponding to the reference coordinates without using a registered image. Therefore, according to the image processing apparatus 10 according to the present embodiment, since it is not necessary to prepare a registered image in advance, it is excellent in usability, and it corresponds to the reference coordinates with high accuracy regardless of the level of the registered image selector. Coordinate information can be acquired.

  Further, the image processing apparatus 10 according to the present embodiment can also acquire information on real coordinates corresponding to the reference coordinates without performing calculation for each pixel. For this reason, it is possible to acquire information on real coordinates corresponding to the reference coordinates at a very high speed as compared with a conventional pattern matching technique such as a normalized correlation method that performs calculation for each pixel. Furthermore, when steps S1 and S2 are performed in advance in the procedure shown in FIG. 6, the processing speed can be further increased.

  In addition, the image processing apparatus 10 according to the present embodiment uses two or more reference coordinates, and identifies the actual coordinates corresponding to each of the two or more reference coordinates, thereby rotating the object in the object image. The angle can be easily calculated.

  The image processing apparatus 10 according to the present embodiment uses design information. For this reason, it is not necessary for the plurality of object elements to have the same shape, and it is not necessary to arrange them periodically.

(2. Second Embodiment)
(2-1. Configuration)
Next, a second embodiment of the image processing apparatus according to the present invention will be described.

  FIG. 7 is a block diagram showing a configuration example of an image processing apparatus 10A according to the second embodiment of the present invention.

  The image processing apparatus 10A shown in the second embodiment is different from the image processing apparatus 10 shown in the first embodiment in that an integrated vector obtained by integrating a design vector group is generated. Since other configurations and operations are not substantially different from those of the image processing apparatus 10 illustrated in FIG. 1, the same components are denoted by the same reference numerals and description thereof is omitted.

  The alignment processing unit 20A of the control unit 15A of the image processing apparatus 10A includes a design vector generation unit 21, a real coordinate acquisition unit 22, a real vector generation unit 23, an alignment information acquisition unit 24A, and an integrated vector generation unit 41.

  FIG. 8A is an explanatory diagram illustrating an example of a design vector group, and FIG. 8B is an explanatory diagram illustrating an example of likelihood information 44 for each integrated vector 42 and integrated element vector 43. FIG. 8A is an example of the same design vector group as FIG. 4B, and FIG. 8B is an example of the integrated vector 42 generated by integrating the design vector group.

  FIG. 8C is a diagram showing an example of likelihood information 44 for each of the integrated vector 42 and the integrated element vector 43 generated by two design vector groups, and FIG. 8D is an integrated vector shown in FIG. FIG. 4 is an explanatory diagram showing an example of two design vector groups that are the basis of 42.

  The integrated vector generation unit 41 generates an integrated vector 42 by integrating the design vector group. At this time, the integrated vector generation unit 41 can obtain information on the appearance frequencies of the integrated element vectors 43 constituting the integrated vector 42.

  Based on the appearance frequency of the integrated element vector 43, the integrated vector generation unit 41 generates likelihood information 44 that associates each element design coordinate CADi with the likelihood for each integrated element vector 43. At this time, the integrated vector generation unit 41 has likelihood information 44 so that the element design coordinates that are likely to correspond to the element real coordinates of the real vector group having the same real vector as the integrated element vector 43 have a higher likelihood. Is generated.

  For example, consider a case where two integrated design vector groups shown in FIG. 8D are integrated to generate an integrated vector 42 shown in FIG. 8C, and this integrated vector 42 is compared with a predetermined real vector group. However, it is unknown whether the element real coordinates of the predetermined real vector group correspond to CAD1 or CAD2.

  As is clear from FIG. 8D, both the design vectors CAD1 and CAD2 have downward vectors. Therefore, likelihood information 44 in which the same weight (for example, 50) is associated with each of CAD1 and CAD2 is generated in the downward integrated element vector 43.

  On the other hand, a vector having a leftward component cannot be a CAD1 design vector, but is always a CAD2 design vector. Therefore, likelihood information 44 in which a larger weight than CAD1 is associated with CAD2 is generated for the integrated element vector 43 having a leftward component.

  In particular, a negative likelihood may be assigned to an element design coordinate that cannot correspond to an element real coordinate of a real vector group having the same real vector as the integrated element vector 43. In the example of FIG. 8D, an example in which likelihood information 44 in which a weight of −100 is associated with CAD1 and a weight of +100 is associated with CAD2 is assigned to the integrated element vector 43 having a component facing left. When the negative likelihood is used, a real vector group having the same real vector as the rare integrated element vector 43 can be detected with higher accuracy.

  FIG. 9 is a diagram for explaining the state of the matching process using the integrated vector 42 executed by the alignment information acquisition unit 24.

  The alignment information acquisition unit 24A identifies the actual coordinates corresponding to the reference coordinates by comparing the integrated vector 42 and the actual vector group. Specifically, the alignment information acquisition unit 24A acquires the likelihood information 44 of the nearest integrated element vector 43 for each of the real vectors constituting the real vector group of the element real coordinates. Next, the alignment information acquisition unit 24A identifies the element design coordinates corresponding to the element actual coordinates by summing the plurality of acquired pieces of likelihood information (see FIG. 9).

  By repeating this for each real vector group, the alignment information acquisition unit 24A defines the reference coordinates (the reference coordinates are defined by the relative positions with respect to the predetermined element design coordinates when the element design coordinates are set as the reference coordinates. In this case, the predetermined element design coordinates) can be associated with the actual element coordinates, and the actual coordinates corresponding to the reference coordinates can be specified.

(2-2. Operation)
Next, an example of the operation of the image processing apparatus 10A according to the second embodiment will be described.
(2-2-1. Main flow)
FIG. 10 is a flowchart illustrating a procedure when the CPU of the image processing apparatus 10 </ b> A illustrated in FIG. 7 easily obtains real coordinate information corresponding to the reference coordinates using the integrated vector 42 without using a registered image. . Steps equivalent to those in FIG. 6 are denoted by the same reference numerals, and redundant description is omitted.

  In step S <b> 11, the integrated vector generation unit 41 generates the integrated vector 42 by integrating the design vector group, and at the same time, based on the appearance frequencies of the integrated element vectors 43 constituting the integrated vector 42. Likelihood information 44 in which each of the element design coordinates CADi and the likelihood are associated with each other is generated. Note that step S11 can be executed in advance prior to the production of the object, similarly to steps S1 and S2.

  In step S12, the alignment information acquisition unit 24A specifies a real coordinate corresponding to the reference coordinate by executing a process of comparing the integrated vector 42 and the real vector group (matching process using the integrated vector).

  With the above procedure, it is possible to easily acquire information on real coordinates corresponding to the reference coordinates using the integrated vector 42 without using a registered image.

(2-2-2. Matching process using integrated vector)
FIG. 11 is a subroutine flowchart showing the procedure of the matching process using the integrated vector executed by the alignment information acquisition unit 24A in step S12 of FIG.

  In step S21, the alignment information acquisition unit 24A stores j = 1 in a required work area of the RAM.

  Next, in step S22, the alignment information acquisition unit 24A assigns k = 1, 2,..., N to the real vectors in the real vector group of the element real coordinates REALj in ascending order (where n is This is a natural number and represents the total number of real vectors constituting the real vector group).

  Next, in step S23, the alignment information acquisition unit 24A stores k = 1 in a required work area of the RAM.

  Next, in step S <b> 24, the alignment information acquisition unit 24 </ b> A estimates an integrated element vector corresponding to the real vector k among the integrated element vectors 43 constituting the integrated vector 42. For example, when the magnitude difference between vectors and the angle between the vectors are within a predetermined threshold value, it is estimated that the vectors correspond to each other.

  Next, in step S25, the alignment information acquisition unit 24A acquires the likelihood information 44 of the integrated element vector 43 corresponding to the real vector k.

  Next, in step S26, the alignment information acquisition unit 24A adds 1 to k stored in the required work area of the RAM.

  Next, in step S27, the alignment information acquisition unit 24A determines whether k is n or less. If k is n or less, the process returns to step S24 because there remains a real vector that does not match the integrated vector 42. On the other hand, if k is larger than n, the process proceeds to step S28.

  Next, in step S28, the alignment information acquisition unit 24A sums the likelihood information 44 acquired for each real vector, and estimates an element design coordinate CADi corresponding to the element actual coordinate REALj (see FIG. 9).

  Next, in step S29, the alignment information acquisition unit 24A adds 1 to j stored in the required work area of the RAM.

  Next, in step S30, the alignment information acquisition unit 24A determines whether j is equal to or less than m. If j is less than or equal to m, the actual vector group of the actual element coordinates that do not match the integrated vector 42 among the actual element coordinates belonging to the predetermined area 31 remains, and the process returns to step S22. On the other hand, if j is larger than m, the process proceeds to step S31.

  In step S31, when the element design coordinates are set as the reference coordinates, the alignment information acquisition unit 24A specifies the actual coordinates corresponding to the reference coordinates by specifying the actual element coordinates corresponding to the element design coordinates. Is identified. Further, when the reference coordinates are defined by a relative position with respect to the predetermined element design coordinates, the actual coordinates corresponding to the reference coordinates are specified by specifying the element actual coordinates corresponding to the predetermined element design coordinates.

  With the above procedure, it is possible to easily acquire information on real coordinates corresponding to the reference coordinates using the integrated vector 42 without using a registered image.

  The image processing apparatus 10A according to the second embodiment has the same effects as the image processing apparatus 10 according to the first embodiment.

  Further, the image processing apparatus 10A according to the present embodiment compares the integrated vector 42 and the real vector group. For this reason, compared with the image processing apparatus 10 according to the first embodiment, it is possible to reduce false detection when the configurations of a plurality of design vector groups are approximate.

  The number of objects to be matched with the real vector group is the number of design vector groups, that is, the number l of element design coordinates in the image processing apparatus 10 according to the first embodiment, whereas the image processing according to the present embodiment is performed. In apparatus 10A, there is only one integrated vector 42. Therefore, the real coordinates corresponding to the reference coordinates can be specified at a very high speed as compared with the image processing apparatus 10 according to the first embodiment.

  Further, the image processing apparatus 10A according to the present embodiment can detect the correspondence between the element design coordinates and the element actual coordinates by comparing the integrated vector 42 and the real vector group. For this reason, even if, for example, one object element to be formed on the object is missing, only one of the real vectors constituting the real vector group is lost, and the likelihood information 44 to be summed The result is not significantly affected. Therefore, the image processing apparatus 10 </ b> A according to the present embodiment is robust against the noise of the object image.

  Further, even when the element design coordinates are set as the reference coordinates and the target element is missing in the element actual coordinates that should correspond to the reference coordinates, the image according to the present embodiment The processing apparatus 10A can accurately detect the correspondence between other element design coordinates and element actual coordinates. For this reason, for example, the most reliable element actual coordinate (the likelihood of the total likelihood information 44 is the highest) is specified, and the actual coordinate of the reference coordinate is relative to the element design coordinate corresponding to the element actual coordinate. Can be specified. Therefore, even if the target element is missing in the element actual coordinates that should correspond to the reference coordinates, it corresponds to the reference coordinates based on the correspondence between the other element design coordinates and the element actual coordinates. Real coordinate information can be easily estimated and acquired.

  In addition, when using likelihood information 44 in which a negative likelihood is assigned to an element design coordinate that cannot correspond to an element real coordinate of a real vector group having the same real vector as the integrated element vector 43, the element design coordinate is more accurately used. And the correspondence between the actual coordinates of the elements can be detected.

(2-2-3. Other procedure of matching process using integrated vector)
FIG. 12 is a subroutine flowchart showing another procedure of the matching process using the integrated vector executed by the alignment information acquisition unit 24A in step S12 of FIG. Steps equivalent to those in FIG. 11 are denoted by the same reference numerals, and redundant description is omitted.

  This procedure differs from the procedure shown in FIG. 11 in that the rotation angle of the object in the object image is calculated in the matching process using the integrated vector.

  FIG. 13 is an explanatory diagram illustrating an example of a state in which a real vector group of an arbitrary element real coordinate REALj is compared with the integrated vector 42 when the target object in the target image has a predetermined rotation angle.

  Depending on the arrangement position of the object with respect to the optical axis of the imaging unit 1, the object may have a predetermined rotation angle in the object image as shown in FIG. In this case, it is convenient if there is information on the predetermined rotation angle in addition to the actual coordinates corresponding to the reference coordinates as the alignment information.

  FIG. 14 is an explanatory diagram showing an example of how to obtain an angle θk = θ1 between the real vector k = 1 and the integrated element vector 43 corresponding to the real vector. FIG. 15A is a diagram for explaining a state in which the integrated vector 42 is rotationally corrected by half of θ1 in the example shown in FIG. 14, and FIG. 15B is a rotational correction shown in FIG. It is explanatory drawing which shows an example of a mode which calculates | requires angle | corner (theta) k = (theta) 2 which real vector k = 2 and the integrated element vector 43 corresponding to this real vector use later integrated vector.

  When the integrated element vector corresponding to the real vector k is estimated in step S24, in step S41, the alignment information acquisition unit 24A acquires the likelihood information 44 of the integrated element vector 43 corresponding to the real vector k, and the real vector k. And an angle θk formed by the integrated element vector 43 is calculated (see FIG. 14).

  Next, in step S42, the alignment information acquisition unit 24A rotates the integrated vector 42 by half of θk (see FIG. 15A). At this time, information on the rotation angle (half of θk) may be stored in a required work area of the RAM and added for each rotation correction. As a result, the integrated element vector 43 used when returning from step S27 to step S24 and executing step S41 again is obtained after rotation correction by the rotation correction angle added in the required work area of the RAM (FIG. 15 (b)).

  In step S43, the alignment information acquisition unit 24A specifies real coordinates corresponding to the reference coordinates, and the integrated vector. In addition, when the reference coordinates are defined by relative positions with respect to the predetermined element design coordinates, the actual coordinates corresponding to the reference coordinates are specified by specifying the element actual coordinates corresponding to the predetermined element design coordinates. Identify the final rotation angle of the integrated vector. This final rotation angle can be regarded as the rotation angle of the object in the object image.

  According to the above procedure, it is possible to easily acquire the information of the real coordinates corresponding to the reference coordinates using the integrated vector 42 without using the registered image, and acquire the information of the rotation angle of the object in the object image. can do.

  According to the procedure shown in FIG. 12, it is possible to specify both the actual coordinates corresponding to the reference coordinates and the rotation angle of the object in the object image based only on the object image of one field of view. When two or more reference coordinates are used in order to obtain alignment information, it may be necessary to change the relative positional relationship between the imaging unit 1 and the object and perform two or more shootings and use an image with two or more fields of view. is there. On the other hand, according to the procedure shown in FIG. 12, alignment information can be acquired by only one imaging without changing the relative position between the imaging unit 1 and the object. For this reason, according to the image processing apparatus 10A according to the present embodiment, alignment information can be acquired easily and at a higher speed than in the case where an object image having two or more visual fields is required.

  Further, when shifting the half of the extracted vector angle by half as shown in step S42, the difference between the previously detected vector angle θk + 1 with respect to the previously detected vector angle θk + 1 is equal to or greater than a predetermined angle, or the previous vectors If an abnormality of the detected inter-vector angle θk + 1 is detected based on the inter-angle variance or average, it is possible to easily take measures such as interrupting the procedure or ignoring the actual vector. For this reason, it is possible to easily reduce the influence of element real coordinates and real vectors caused by noise such as dust.

  In the procedure shown in FIG. 12, every time the process returns from step S30 to step S22, that is, for each element actual coordinate j, the rotation angle is saved and the rotation of the integrated vector 42 is reset, so that all element actual coordinates j The average of the rotation angles may be the rotation angle of the object.

  Further, even when the rotation of the integrated vector 42 is not corrected as in the procedure shown in FIG. 11, for example, the angle θk formed by the real vector and the integrated element vector is detected and sequentially stored in step S24 or S25 of FIG. In addition, the average θk may be treated as the rotation angle of the object in the object image.

(3. Third embodiment)
(3-1. Configuration)
Next, a third embodiment of the image processing apparatus according to the present invention will be described.

  FIG. 16 is a block diagram illustrating a configuration example of an image processing apparatus 10B according to the third embodiment of the present invention.

  The image processing apparatus 10B shown in the third embodiment is different from the image processing apparatus 10 shown in the first embodiment in the element real coordinate acquisition process. Since other configurations and operations are not substantially different from those of the image processing apparatus 10 shown in FIG. 1 and the image processing apparatus 10A shown in FIG. 7, the same components are denoted by the same reference numerals and description thereof is omitted.

  Although FIG. 16 illustrates an example in which the image processing apparatus 10B includes the integrated vector generation unit 41, the integrated vector generation unit 41 may be omitted in the present embodiment.

  The alignment processing unit 20B of the control unit 15B of the image processing apparatus 10B includes a design vector generation unit 21, a real coordinate acquisition unit 22B, a real vector generation unit 23, an alignment information acquisition unit 24, and an integrated vector generation unit 41.

  The actual coordinate acquisition unit 22B acquires the element actual coordinates by extracting the object element based on the object image using the generalized Hough transform.

  17 and 18 are diagrams for explaining the generalized Hough transform. FIG. 17A is an explanatory view showing a triangle as an example of the extraction target, and FIG. 17B shows the feature points of the extraction target and the detection target graphic stored in advance in the storage medium according to the shape of the extraction target. It is explanatory drawing which shows an example of the vector between reference | standard points, (c) is explanatory drawing which shows an example of a mode that a voting process is performed with respect to a reference | standard point based on the feature point of an input image, (d) These are explanatory drawings showing examples of shapes of various extraction objects.

  Moreover, FIG. 18 is explanatory drawing which shows an example of the mode at the time of extracting the extraction object of circular shape by generalized Hough transformation.

  As shown in FIGS. 17D and 18, the generalized Hough transform (GHT), which is a generalized Hough transform for shape extraction, can detect an irregular figure that cannot be represented by a parameter.

  When performing the generalized Hough transform, the real coordinate acquisition unit 22B obtains information related to the shape of the object to be detected (vector between the feature point to be extracted and the reference point) from the design information in advance, via a network, or Obtained via the input unit 11 (see FIG. 17B). Then, feature points on the image are extracted, and an object is detected by repeating voting to a predetermined cell (reference point) for each feature point based on the position of the feature point and the normal direction of the edge. (See FIG. 17 (c)).

  In particular, when the object element is a bump, the generalized Hough transform method is suitable as a method for extracting the object element from the object image.

  Further, the real coordinate acquisition unit 22B generates a binary image from the object image using a predetermined intensity threshold th, and acquires the element actual coordinates by extracting the object element based on the binary image. Also good.

  Further, the actual coordinate acquisition unit 22B may perform a labeling process on the binary image and acquire the representative coordinates of each label after the labeling process as element actual coordinates. As the representative coordinates of the label, the center or the center of gravity of the label can be used.

  The predetermined intensity threshold th can be automatically set based on the number of object elements estimated to be included in the object image. At this time, the real coordinate acquisition unit 22B sets a predetermined range according to the number of object elements, generates a binary image by extracting pixels having an intensity higher than the intensity threshold th, and performs a labeling process, The auxiliary process may be performed so that the number of labels after the labeling process is within a predetermined range.

  In the auxiliary processing, when the number of labels is outside the predetermined range, the intensity threshold th is automatically changed so that the number falls within the predetermined range, and the binary image is generated again and the labeling processing is repeated. If the number is within a predetermined range, the representative coordinates of each label after the labeling process are acquired as element actual coordinates. Further, an upper limit may be set for the number of times the intensity threshold th is changed in order to prevent an adverse effect that the intensity threshold th is continuously changed.

  When labeling processing is performed, the element actual coordinates can be acquired if there is label information. For this reason, after performing the labeling process, original images such as the object image and the generalized Hough transform image may be deleted from the storage medium.

  Note that the image to be binarized may be the object image itself (luminance image) or an image after generalized Hough transform. In the present embodiment, an example in which the real coordinate acquisition unit 22B performs binarization processing and labeling processing on the image after the generalized Hough transform will be described.

(3-2. Operation)
Next, an example of the operation of the image processing apparatus 10B according to the third embodiment will be described.
(3-2-1. Main flow)
FIG. 19 is a flowchart showing a procedure of element actual coordinate acquisition processing executed by the actual coordinate acquisition unit 22B of the image processing apparatus 10B shown in FIG. 16 in step S3 of FIG. In the following description, an example in which the object element has a circular shape such as a bump is shown.

  In step S51, the real coordinate acquisition unit 22B performs generalized Hough transform on the object image to generate a generalized Hough transform image (GHT image).

  Next, in step S52, the real coordinate acquisition unit 22B performs binarization processing and labeling processing on the generalized Hough transform image.

  Next, in step S53, the actual coordinate acquisition unit 22B acquires the representative coordinate of the label as the element actual coordinate, and proceeds to step S4 in FIG.

  With the above procedure, element actual coordinates can be acquired based on the generalized Hough transform image.

(3-2-2. Automatic binarization and labeling)
FIG. 20 is a subroutine flowchart showing the procedure of the binarization process and the labeling process executed by the real coordinate acquisition unit 22B in step S52 of FIG.

  FIG. 21 is a diagram for explaining the binarization process and the labeling process in the procedure shown in FIG.

  In step S61, the real coordinate acquisition unit 22B obtains the number nA of object elements estimated to be included in the predetermined area of the object image.

  Next, in step S62, the real coordinate acquisition unit 22B acquires information on the average number of pixels nB occupied by one target element from parameters or the like. The number of pixels occupied by all object elements included in the predetermined area can be expressed as nA × nB.

  Steps S61 and S62 may be performed in advance before generating the object image.

  Next, in step S63, the real coordinate acquisition unit 22B acquires the generalized Hough transform image generated in step S51 as the processing target image. Note that the processing target image may be a target object image.

  Next, in step S64, the real coordinate acquisition unit 22B forms a histogram of the intensity distribution of the pixels of the acquired image (see FIG. 21).

  Next, in step S65, the real coordinate acquisition unit 22B obtains the intensity threshold th so that the number of effective pixels is nA × nB.

  Next, in step S66, the real coordinate acquisition unit 22B generates a binary image by extracting pixels having an intensity higher than the intensity threshold th and executes a labeling process.

  The threshold value th of the binarization process can be automatically set appropriately by the procedure of steps S61-66 described above. Moreover, the real coordinate acquisition part 22B can acquire the representative coordinate of each label after the labeling process obtained as a result as an element real coordinate.

  Further, the next auxiliary process (steps S67-70) may be performed so that the value of the intensity threshold th becomes a more appropriate value. The auxiliary process is a process for automatically setting the intensity threshold th so that the number of labels is more surely within a predetermined range, and whether or not the number of labels obtained as a result of step S66 is within a predetermined range. If it is outside the predetermined range, the strength threshold th is reset so that the number of labels is within the predetermined range, and the process returns to step S66.

  In addition, when performing an auxiliary process, the memory | storage part 14 may memorize | store the upper limit and lower limit of the number of labels beforehand.

  In step S67, the real coordinate acquisition unit 22B determines whether or not the number of labels obtained as a result of step S66 is smaller than the upper limit. If the number of labels is equal to or greater than the upper limit, the process proceeds to step S68, the intensity threshold th is multiplied by a predetermined constant α (where α> 1) so that the number of labels is within a predetermined range, and the process returns to step S66. On the other hand, if the number of labels is smaller than the upper limit, the process proceeds to step S69.

  Next, in step S69, the real coordinate acquisition unit 22B determines whether or not the number of labels obtained as a result of step S66 is greater than the lower limit. If the number of labels is equal to or less than the lower limit, the process proceeds to step S70, and the intensity threshold th is multiplied by a predetermined constant β (where β <1) so that the number of labels is within a predetermined range, and the process returns to step S66. On the other hand, if the number of labels is greater than the lower limit, it is determined that the number of labels is within a predetermined range, and the process returns to step S53 in FIG.

  Note that the auxiliary process (steps S67-70) may not be executed, and even if it is executed, a limit may be set on the number of loops (number of auxiliary processes).

By performing the auxiliary process, it is possible to automatically set the intensity threshold th so that the number of labels is more reliably within a predetermined range.
(3-2-3. Co-occurrence generalized Hough transform)

  The feature points extracted in the generalized Hough transform often include unintended feature points due to dust or the like. However, in the conventional generalized Hough transform, all the extracted feature points are voted equally. For this reason, when there is a lot of noise such as dust in the input image, it is difficult to accurately detect the object.

  On the other hand, in the generalized Hough transform, information on the shape of the object to be detected is used. Then, the feature points on the image are extracted, and the object is detected by repeating voting to a predetermined cell for each feature point based on the position of the feature point and the normal direction of the edge. Yes.

  If the shape of the object to be detected is used well and the co-occurrence of feature points according to the shape of the object to be detected can be used, it is considered that more accurate object detection can be performed. Therefore, the real coordinate acquisition unit 22B according to the present embodiment replaces the generalized Hough transform with the generalized Hough transform (adding the co-occurrence information) using the co-occurrence of feature points according to the shape of the detection target object. Generalized Hough transform, hereinafter referred to as co-occurrence generalized Hough transform (referred to as co-occurrence GHT)).

  FIG. 22A is an explanatory diagram illustrating an example of a state in which three predicted feature points to be co-occurred with respect to one feature point (x, y) are extracted when the object element has a circular shape. (B) is an explanatory view showing an example of the direction of the normal direction of the edge of the feature point to be co-extracted extracted in the case shown in (a).

  FIG. 23 is a diagram for explaining that the co-occurrence GHT can be applied even if the object element has an arbitrary shape.

  Since the shape of the object to be detected is known, when one feature point is extracted, if this feature point is a feature point belonging to the object element, an expectation to co-occur with the extracted feature point It is possible to predefine the position of feature points (predicted co-occurrence position) and edge normal direction (predicted co-occurrence edge normal direction) (co-occurrence coordinate distances Dx1, Dy1, edge normal direction in FIG. Direction Vx1, Vy1).

  Therefore, when an edge actually exists at the predicted co-occurrence position of the predicted feature point and the normal direction of this edge coincides with the predicted co-occurrence edge normal direction, the extracted feature point belongs to the object element. The possibility of a feature point is considered extremely high. For this reason, the voting frequency of the extracted feature points to the reference point of the figure to be detected is increased.

  As described above, according to the co-occurrence generalized Hough transform, it is possible to greatly reduce the influence of the feature points due to dust, and to detect the detection target more accurately.

  FIG. 22 shows an example in which the predicted co-occurrence position is shifted by 90 degrees when the object element has a circular shape. (Corner) position only, or more positions such as positions shifted by 45 degrees may be defined.

  FIG. 24 is a flowchart showing another procedure of element actual coordinate acquisition processing executed by the actual coordinate acquisition unit 22B of the image processing apparatus 10B shown in FIG. 16 in step S3 of FIG. Steps equivalent to those in FIG. 19 are denoted by the same reference numerals, and redundant description is omitted.

  When executing the co-occurrence generalized Hough transform, the real coordinate acquisition unit 22B acquires in advance information on the co-occurrence of feature points according to the shape of the object element (see Dx1 and the like in FIG. 23).

  In step S80, the real coordinate acquisition unit 22B performs co-occurrence generalized Hough transform on the object image to generate a co-occurrence generalized Hough transform image (co-occurrence GHT image).

  With the above procedure, the element actual coordinates can be acquired based on the co-occurrence generalized Hough transform image. The co-occurrence generalized Hough transform image can detect the detection target object from the image more accurately than the generalized Hough transform image. For this reason, according to the procedure shown in FIG. 22, compared with the procedure shown in FIG. 19, the influence of erroneous detection of dust or the like can be reduced, and more accurate element actual coordinates can be acquired.

  Next, the co-occurrence generalized Hough transform image generation process will be described in more detail.

  FIG. 25 is a subroutine flowchart showing the procedure of co-occurrence GHT image generation processing executed by the real coordinate acquisition unit 22B in step S80 of FIG.

  In step S81, the real coordinate acquisition unit 22B extracts feature points from the object image. Next, in step S <b> 82, the real coordinate acquisition unit 22 </ b> B acquires information on expected feature points that should co-occur with the extracted feature points. The information on the predicted feature point includes information on the predicted co-occurrence position (for example, a relative vector with respect to the extracted feature point) and information on the predicted co-occurrence edge normal direction.

  Next, in step S83, the actual coordinate acquisition unit 22B determines whether or not an edge actually exists at the predicted co-occurrence position of the predicted feature point and whether the normal direction of this edge matches the predicted co-occurrence edge normal direction. Determine whether. If there is an edge and the normal direction of this edge coincides with the predicted co-occurrence edge normal direction, the process proceeds to step S84, and the voting frequency of the feature point extracted in step S81 to the reference point of the figure to be detected is set. Increase and go to step S86. On the other hand, if the edge does not exist or the edge normal direction does not match the predicted co-occurrence edge normal direction even if the edge exists, the process proceeds to step S85, and the feature point extracted in step S81 is displayed. The voting frequency for the reference point of the figure to be detected is reduced, and the process proceeds to step S86.

  In step S86, the real coordinate acquisition unit 22B generates a co-occurrence generalized Hough transform image based on the vote count, and the process proceeds to step S52 in FIG.

  By the above procedure, the object element can be more accurately extracted from the object image by the generalized Hough transform (co-occurrence generalized Hough transform) using information on the co-occurrence.

  The image processing apparatus 10B according to the third embodiment has the same effects as the image processing apparatus 10 according to the first embodiment. Further, when the image processing apparatus 10B includes the integrated vector generation unit 41, the same effects as those of the image processing apparatus 10A according to the second embodiment can be obtained.

  Further, the image processing apparatus 10B according to the present embodiment can extract the object element from the object image by the generalized Hough transform. In the generalized Hough transform, only the contour information of the object element is used. For this reason, compared with the case where the normalized correlation method is used, even if there is an abnormality in the three-dimensional shape of the object element, the object element can be extracted without any problem.

  In addition, the image processing apparatus 10B according to the present embodiment can automatically set the threshold th appropriately when generating a binary image, which is very convenient for the user. Further, when the auxiliary process is performed when the threshold value th is automatically set, the threshold value th can be set so that the number of labels is surely within a predetermined range.

  Further, the image processing apparatus 10B according to the present embodiment can perform co-occurrence generalized Hough transform. When performing the co-occurrence generalized Hough transform, it is possible to reduce the influence of erroneous detection of dust and the like compared to the case of performing the generalized Hough transform, and to obtain more accurate element actual coordinates.

  In addition, when performing generalized Hough transform or co-occurrence generalized Hough transform, if the input image (object image) is compressed, the calculation speed can be improved. Further, the input image may be smoothed, and in this case, the influence of dust in the image can be reduced.

(4. Fourth embodiment)
Next, a fourth embodiment of the image processing apparatus according to the present invention will be described.

  FIG. 26 is a block diagram illustrating a configuration example of the three-dimensional shape measurement apparatus 100 including the image processing apparatus 10C according to the fourth embodiment of the present invention.

  The image processing apparatus 10C shown in the fourth embodiment is different from the image processing apparatus 10 shown in the first embodiment in that it has a function of measuring the three-dimensional shape of an object. Since other configurations and operations are not substantially different from those of the image processing apparatus 10 illustrated in FIG. 1, the same components are denoted by the same reference numerals and description thereof is omitted. In the following description, an example in which the optical axis direction is the Z axis and the directions perpendicular to the optical axis direction are the X axis and the Y axis will be described.

  The three-dimensional shape measuring apparatus 100 includes an illumination optical system 111 having a light source 111a, an aperture plate 112, an objective lens 113, a focal position changing unit 114, and an object element 115 arranged so that a main surface thereof is perpendicular to the optical axis direction. A mounting table 117 for mounting the object 116, a mounting table driving unit 118 for moving the mounting table 117 in each of the XYZ directions, a mounting table 117, and a support base that supports the mounting table driving unit 118. It has the stand 119, the imaging part 1 into which the reflected light from the target object 116 is incident, and the image processing apparatus 10C. The loading table driving unit 118 includes a loading table Z displacement unit 121 and a loading table XY displacement unit 122.

  For example, a halogen lamp or a laser can be used as the light source 111a. The light emitted from the light source 111a becomes a planar illumination light beam through the illumination lens 123. This light illuminates the aperture plate 112 via the polarization beam splitter 124.

  The aperture plate 112 has a plurality of apertures 125 each functioning as a confocal aperture. The aperture plate 112 may be formed with a plurality of apertures 125, and the aperture plate 112 may be one in which the apertures 125 are two-dimensionally arranged to form an aperture array. It may be used. When using a Nipkow disc, the Nipkow disc is rotationally driven by a drive mechanism (not shown).

  The light that has passed through each aperture 125 irradiates the objective lens 113 via the focal position changing unit 114, and is projected onto the object 116 by the objective lens 113 while being a spot conjugate with the aperture 125 (object-side condensing point). ). Each object-side condensing point is located on a surface perpendicular to the optical axis direction at a predetermined position in the Z-axis direction (hereinafter referred to as an object-side condensing surface). Note that the objective lens 113 may be constituted by, for example, a plurality of lenses and a diaphragm to form a double-sided telecentric optical system.

  The mounting table Z displacement unit 121 is configured by a general driving device such as a stepping motor, a servo motor, or a piezo motor, and displaces the mounting table 117 in the optical axis direction. The amount, direction, and timing of this displacement are controlled via the Z-axis driver 130 by the image processing apparatus 10C. The mounting table Z displacement unit 121 displaces the mounting table 117 roughly in the direction of the optical axis before starting measurement, for example.

  FIG. 27 is a perspective view illustrating a configuration example of the focal position changing unit 114. FIG. 27 shows an example in which the optical path changing piece 131 is provided on the surface on the object side.

  When a parallel plate-shaped optical path changing piece 131 made of a transparent body is disposed in the optical path of the objective lens 113, the position of the object-side condensing surface of the objective lens 113 moves in the Z direction. This focal movement width can be controlled by the refractive index and thickness of the optical path changing piece 131.

  Therefore, as shown in FIG. 27, optical path changing pieces 131 having different focal movement widths are arranged at equal intervals along the rotation direction with respect to the rotating plate 132. By continuously rotating the rotating plate 132 at a predetermined speed by a driving unit 133 such as a motor, the objective lens 113 is discretely (stepwise) each time the optical path changing piece 131 intersects the optical axis of the objective lens 113. The position of the object side condensing surface can be moved in the Z direction (optical axis direction). A part of the optical path changing piece 131 disposed on the rotating plate 132 may have the same refractive index and thickness as the other optical path changing pieces 131 and the same focal point movement width.

  The rotation state of the rotating plate 132 is detected by the timing sensor 134. The output of the timing sensor 134 is given to the image processing apparatus 10C. The image processing apparatus 10C holds information in which each optical path changing piece 131 and the Z-axis coordinate of the object-side condensing point are associated in advance. The image processing apparatus 10 </ b> C repeats exposing the imaging unit 1 at the timing at which each optical path changing piece 131 and the optical axis intersect based on the output of the timing sensor 134, whereby the positions of a plurality of discrete object-side condensing points are detected. With each of these, it is possible to perform imaging easily and at high speed.

  The drive unit 133 may be configured to be controllable by the image processing apparatus 10C. In this case, the image processing apparatus 10 </ b> C can control the rotation speed of the rotating plate 132.

  In addition, although the following description shows an example in which the shape of the optical path changing piece 131 is circular, these shapes may be elliptical, polygonal, or the like.

  Of the light reflected by the object 116, the light reflected particularly at the object side condensing point is condensed by the objective lens 113 to a point optically conjugate with the object side condensing point (hereinafter referred to as an image side condensing point). Is done. The opening 125 as a point light source and the object-side condensing point correspond one-to-one. In the present embodiment, an example in which the image side condensing point coincides with the opening 125 as a point light source will be described. In this case, the light that has passed through the opening 125 is condensed at the object-side condensing point, reflected by the object-side condensing point, and incident on the opening 125 again.

  The light that has entered the opening 125 again is deflected by the polarization beam splitter 124, enters the imaging optical system 138, and enters the imaging device that constitutes the imaging unit 1. Here, the imaging optical system 138 is configured to connect the image of the opening 125 to the photoelectric conversion surface of the imaging element. The aperture 125 (image-side condensing point) and the image sensor disposed at a position corresponding to the aperture 125 are conjugated with each other by the polarization beam splitter 124 and the imaging optical system 138.

  The imaging unit 1 is configured by a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and outputs a signal corresponding to the intensity of irradiated light to the image processing apparatus 10C. The imaging unit 1 is controlled in light detection timing by the image processing apparatus 10C.

  The image processing apparatus 10C can use the signal received from the imaging unit 1 as image data (hereinafter referred to as confocal image data) for generating an image (hereinafter referred to as confocal image). Further, the image processing apparatus 10C may generate a confocal image based on the confocal image data, for example, for each exposure. Note that if the image processing apparatus 10C can obtain the output signal of the imaging unit 1 for each exposure, the image processing apparatus 10C can generate an object image and generate alignment information or perform a three-dimensional shape measurement based on the output signal. Is possible.

  The mounting table XY displacement unit 122 of the mounting table driving unit 118 displaces the mounting table 117 in a direction perpendicular to the optical axis direction. For example, the stage XY displacement unit 122 is used to move the measurement target region in the XY plane between measurements.

  The mounting table XY displacement unit 122 includes an X-axis displacement mechanism 141 and a Y-axis displacement mechanism 142 that position the mounting table 117 in the X-axis direction and the Y-axis direction. The X-axis displacement mechanism 141 and the Y-axis displacement mechanism 142 are configured by, for example, servo motors, and the amount, direction, and timing of the displacement are controlled by the image processing apparatus 10C via the XY axis driver 145.

  FIG. 28 is a block diagram showing a configuration example of an image processing apparatus 10C according to the fourth embodiment of the present invention.

  The control unit 15C of the image processing apparatus 10C functions as at least the alignment processing unit 20 and the three-dimensional shape measurement unit 200 by a program. The alignment processing unit 20 and the three-dimensional shape measuring unit 200 use a required work area in the RAM as a temporary storage location for data. In addition, this function realization part may be comprised by hardware logics, such as a circuit, without using CPU.

  Although FIG. 28 shows an example in which the control unit 15C includes the alignment processing unit 20, the alignment processing unit 20A according to the second embodiment may be used instead of the alignment processing unit 20 or the third embodiment. You may use the alignment process part 20B which concerns on a form. Further, when the alignment processing unit 20B is used, the integrated vector generation unit 41 may be omitted.

  The three-dimensional shape measurement unit 200 executes at least arithmetic processing necessary for three-dimensional shape measurement and general control of the operation of the three-dimensional shape measurement apparatus 100.

  Specifically, the three-dimensional shape measuring unit 200 has the maximum intensity of reflected light incident on the imaging unit 1 based on the signal of the imaging element for each position of the object-side condensing point that is discretely changed in the optical axis direction. A distance image of the target object is generated by estimating the position of the target object in the optical axis direction. In addition, the solid shape measuring unit 200 uses the actual coordinates corresponding to the reference coordinates acquired by the alignment processing unit 20 to calculate the element actual coordinates of the object element included in the distance image and the element design coordinates of the design information. By associating (aligning), the three-dimensional shape of the object is measured based on the distance image.

  According to the image processing apparatus 10C according to the present embodiment, by using the alignment information, the element actual coordinates of the object element included in the distance image and the element design coordinates of the design information are easily and accurately associated with each other. Can do. For this reason, the three-dimensional shape of the object element actually disposed on the object can be measured and evaluated quickly and accurately.

  The alignment processing unit 20 may use the distance image generated by the three-dimensional shape measuring unit 200 as an input image.

  As mentioned above, although some embodiment of this invention was described, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

  Further, in the embodiment of the present invention, each step of the flowchart shows an example of processing that is performed in time series in the order described. The process to be executed is also included.

DESCRIPTION OF SYMBOLS 1 Imaging part 10, 10A, 10B, 10C Image processing apparatus 20, 20A, 20B Alignment process part 21 Design vector generation part 22, 22B Real coordinate acquisition part 23 Real vector generation part 24, 24A Alignment information acquisition part 41 Integrated vector generation part 42 Integrated Vector 43 Integrated Element Vector 44 Likelihood Information 100 Solid Shape Measuring Device 111a Light Source 112 Aperture Plate 113 Objective Lens 114 Focus Position Changing Unit 115 Object Element 116 Object 125 Opening 200 Solid Shape Measuring Unit

Claims (9)

Design coordinates indicating the target placement position of each object element in the design coordinate system for defining the position on the design information from the design information for creating the object on which a plurality of object elements are arranged A design vector generation unit that acquires a certain element design coordinate and generates a design vector group including a relative vector between the element design coordinate of the target element and a plurality of other element design coordinates for each target element;
The actual arrangement position of each object element in the real coordinate system for defining the position on the object image based on the object image obtained by imaging the object manufactured based on the design information. An actual coordinate acquisition unit that acquires element actual coordinates, which are the actual coordinates shown;
A real vector generation unit that generates a real vector group including a relative vector between its own element real coordinates and a plurality of other element real coordinates for each object element in the object image;
By comparing the design vector group and the real vector group, an alignment information acquisition unit that identifies real coordinates corresponding to design coordinates of a reference point included in the design information;
Comprising an alignment portion having,
The alignment unit further includes an integrated vector generation unit that generates an integrated vector by integrating the design vector group,
The alignment information acquisition unit
Identifying the real coordinates corresponding to the design coordinates of the reference point by comparing the integrated vector and the real vector group;
Image processing device. The integrated vector generation unit
Based on the appearance frequency of the integrated element vectors constituting the integrated vector, the element design coordinates that are more likely to correspond to the element real coordinates of the real vector group having the same relative vector as the integrated element vector have a higher likelihood. Likely, likelihood information that associates each of the element design coordinates and the likelihood is generated for each integrated element vector,
The image processing apparatus according to claim 1 . The integrated vector generation unit
For the element design coordinates that the element real coordinates of the real vector group having the same relative vector as the integrated element vector cannot correspond, a negative likelihood is assigned to generate the likelihood information.
The image processing apparatus according to claim 2 . The alignment information acquisition unit
Obtaining the likelihood information of the integrated element vector closest to each of the real vectors constituting the real vector group of the element real coordinates, and element design coordinates corresponding to the element real coordinates from the obtained plurality of likelihood information Identifying the actual coordinates corresponding to the design coordinates of the reference point by repeating identifying
The image processing apparatus according to claim 2 or 3. The alignment information acquisition unit
Identifying a rotation angle of the object in the object image by comparing the integrated vector and the real vector group;
The image processing apparatus according to any one of claims 1 to 4. Design coordinates indicating the target placement position of each object element in the design coordinate system for defining the position on the design information from the design information for creating the object on which a plurality of object elements are arranged A design vector generation unit that acquires a certain element design coordinate and generates a design vector group including a relative vector between the element design coordinate of the target element and a plurality of other element design coordinates for each target element;
The actual arrangement position of each object element in the real coordinate system for defining the position on the object image based on the object image obtained by imaging the object manufactured based on the design information. An actual coordinate acquisition unit that acquires element actual coordinates, which are the actual coordinates shown;
A real vector generation unit that generates a real vector group including a relative vector between its own element real coordinates and a plurality of other element real coordinates for each object element in the object image;
By comparing the design vector group and the real vector group, an alignment information acquisition unit that identifies real coordinates corresponding to design coordinates of a reference point included in the design information;
Comprising an alignment portion having,
The real coordinate acquisition unit
Information on the co-occurrence of feature points according to the shape of the object element is acquired in advance, and the object element is obtained based on the object image using a generalized Hough transform using the information on the co-occurrence. By obtaining the element actual coordinates by extracting,
The real coordinate acquisition unit further includes
A feature point is extracted from the object image, and when the extracted feature point is a feature point belonging to the object element, a predicted co-occurrence position and a prediction of an expected feature point to be co-occurred with the extracted feature point When information on the co-occurrence edge normal direction is obtained, and an edge actually exists at the predicted co-occurrence position of the predicted feature point, and the normal direction of this edge coincides with the expected co-occurrence edge normal direction , Increase the vote count to the reference point of the extracted feature points,
Image processing device. The real coordinate acquisition unit
An intensity threshold value is automatically set based on the number of object elements estimated to be included in the object image, and a binary image is generated by extracting pixels having an intensity higher than the intensity threshold value to perform a labeling process. To give the representative coordinates of each label after the labeling process as the element actual coordinates,
The image processing apparatus according to any one of claims 1 to 6. The real coordinate acquisition unit
The intensity threshold is set based on the estimated number of the object elements, and a predetermined range is set according to the estimated number of the object elements, and pixels having an intensity higher than the intensity threshold are extracted. To generate a binary image and perform a labeling process, and when the number of labels after the labeling process is outside the predetermined range, the intensity threshold value is automatically changed to be within the predetermined range. While repeatedly generating a binary image and applying a labeling process, the representative coordinates of each label after the labeling process are acquired as the element actual coordinates when the number of labels is within the predetermined range.
The image processing apparatus according to claim 7 . A three-dimensional shape measuring apparatus using a confocal optical system,
A light source;
An aperture plate in which a plurality of confocal apertures are formed;
The respective lights that have passed through the plurality of confocal openings are condensed on an object-side condensing point, and the reflected light of the collected light from the object is again condensed on the corresponding confocal openings. An objective lens;
A focal position changing unit that discretely changes the focal point of the objective lens in the optical axis direction;
An imaging unit having a plurality of imaging elements to which the reflected light from the object condensed again on the confocal opening is input;
Each of the imaging elements based on the alignment unit according to any one of claims 1 to 8 and a signal of the imaging element for each position of the object-side condensing point that is discretely changed in an optical axis direction. The distance image of the object is generated by estimating the position of the object in the optical axis direction where the intensity of the reflected light that is incident on the imaging element is maximized, and the reference acquired by the alignment unit Solid shape measurement for measuring the solid shape of the object based on the distance image by associating the element actual coordinates and the element design coordinates included in the distance image using the real coordinates corresponding to the design coordinates of the points. An image processing apparatus comprising:
A three-dimensional shape measuring apparatus.

Images