JP7766291B2 - Image analysis method and image analysis device - Google Patents

Description

Article 30, Paragraph 2 of the Patent Act applies. Date held: July 20, 2020. Name and location of the meeting: International conference "42nd Annual International Conferences of the IEEE Engineering in Medicine and Biology Society in conjunction with the 43rd Annual Conference of the Canadian Medical and Biological Engineering Society" (held online).

This disclosure relates to an image analysis method and an image analysis device, and in particular to an image analysis method and an image analysis device that analyzes the correspondence between a reference pattern image and a captured image generated from a subject onto which pattern light is projected.

Technologies exist for expressing a sense of depth in two-dimensional images (i.e., constructing three-dimensional images) that are generated by photographing an object (subject) using an imaging device (camera). For example, stereo imaging is a typical technique for expressing three-dimensional images.

Stereo imaging methods are generally classified into passive stereo and active stereo. In active stereo, a light projector installed separately from the imaging device projects light onto the subject, which is then captured by the imaging device. In active stereo, the light from the projector is reflected from the subject and reaches the imaging device, meaning that the optical path of the light from the projector is shifted according to a distance corresponding to the depth of the subject. Therefore, active stereo uses this optical path shift to calculate the distance between the imaging device and the subject based on triangulation, and represents a three-dimensional image (depth) based on this distance.

JP 2009-300277 A

In the active stereo method described above, in order to calculate the distance corresponding to the deviation in the optical path, it is necessary to associate the original position of the light to be projected onto the subject on a two-dimensional plane of the subject with the actual position of the projected light. One type of active stereo method is the pattern light projection method, which projects light that forms a specific pattern (such as a grid) from a projector. Compared to methods that project spot light, for example, the pattern light projection method can reduce the number of times the subject needs to be photographed.

However, with the pattern light projection method, it is necessary to associate the original position of the pattern light with the actual projected position for each of the multiple gratings, making this association difficult. This is because the shape of the pattern light reflected from the subject is distorted depending on the depth of the subject, making it difficult to determine which grating in the pattern light to be projected corresponds to which grating in the actually projected pattern light.

There is a technology that solves the above problem by considering the epipolar geometry between the projection device and the imaging device. To consider epipolar geometry, the positional relationship between the projection device and the imaging device must be known. For example, when an imaging device is incorporated into an endoscope and an image of an object inside the human body is captured, the endoscope enters the human body, making it impossible to know the positional relationship between the projection device and the imaging device. In such cases, it is not possible to consider epipolar geometry, making the above-mentioned correspondence difficult.

Patent Document 1 discloses a technology that achieves three-dimensional reconstruction by using a simple grid pattern made up of a large number of straight lines, where only the vertical and horizontal directions can be distinguished, and using the intersections of these lines as feature points. While the technology disclosed in Patent Document 1 can easily achieve three-dimensional reconstruction by using a simple pattern, it does not solve the problems mentioned above.

In one embodiment, a computer-implemented method analyzes the correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, the reference image including a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image represents that at least one of the first set of elements and the second set of elements includes elements that extend intermittently. The method includes the steps of: identifying, in the captured image, respective nodes composed of the first set of elements and the second set of elements; classifying each of the nodes based on its relative position with respect to adjacent nodes and assigning a classification value to the node; generating a lattice graph from the classified nodes; and matching the nodes in the captured image with nodes in the reference image by inputting the classification values and the lattice graph into a graph convolutional network (GCN), the GCN being configured to learn the classification values for the classified nodes from the reference image and the lattice graph generated from the reference image.

Another embodiment of the method is a computer-implemented method for analyzing the correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, and the reference image represents each of the second set of elements being arranged at random intervals in the same direction. The method includes the steps of: identifying, in the captured image, each node composed of the first set of elements and the second set of elements; classifying each of the nodes based on its relative angle with respect to adjacent nodes and assigning a classification value to each of the nodes; generating a lattice graph from the classified nodes; and matching the nodes in the captured image with nodes in the reference image by inputting the classification values and the lattice graph into a graph convolutional network (GCN), wherein the GCN is configured to learn the classification values for the classified nodes from the reference image and the lattice graph generated from the reference image.

The image analysis method and image analysis device according to the embodiment make it easy to associate an original pattern with a pattern contained in an image of a photographed subject.

FIG. 1 is a diagram illustrating an example of the configuration of an image analysis system. FIG. 1 is a block diagram illustrating an example of the configuration of a computer device. FIG. 2 is a diagram illustrating an example of a reference image according to the first embodiment. FIG. 10 is a diagram showing another example of the reference image according to the first embodiment. 4 is a flowchart illustrating an example of processing executed by the image analysis system according to the first embodiment. FIG. 10 is a diagram illustrating an example of classifying nodes in an image. FIG. 10 is a diagram showing a region with a marker at the center. FIG. 2 is a block diagram illustrating an example of the configuration of a GCN. FIG. 10 is a diagram illustrating an operation that takes into account feature vectors, relationships with adjacent nodes, and weight values. 10 is a flowchart illustrating an example of processing executed by an image analysis system according to a second embodiment. 10 shows an example of a process for calculating the inner product of feature vectors of a reference image and a captured image according to the second embodiment. FIG. 13 is a diagram illustrating an example of a reference image according to the third embodiment. 10A and 10B are diagrams illustrating an example of processing for determining the phase of a grid (the relative positional relationship between a repeating grid and each pixel expressed as a rotation angle greater than or equal to 0 and less than 2π) from a captured image according to the third embodiment. FIG. 11 is a diagram illustrating an example of a connection based on a proximity relationship according to a third embodiment. FIG. 13 is a diagram illustrating an example of classifying nodes in an image according to the third embodiment.

An image analysis method and image analysis device according to one embodiment will be described in detail below with reference to the accompanying drawings. In this embodiment, the image analysis method and image analysis device are implemented in an image analysis system. In this embodiment, the object onto which light from a projector is projected and photographed by a camera is referred to as the "photographed object." The photographed object includes any object with depth, such as a human, animal, object, or part of any of these.

The following describes an image analysis system according to an embodiment. The image analysis system is applied to an example of capturing images of an object inside the human body. Based on the active stereo method, the image analysis system analyzes the correspondence between a pattern image (reference image) corresponding to the pattern light projected onto the object and a captured image generated by capturing an image of the object onto which the pattern light is projected.

First Embodiment
First, an example of the configuration of an image analysis system 100 will be described with reference to Fig. 1. In this embodiment, the image analysis system 100 includes a computer device 1 and an endoscope 2. The computer device 1 is mutually coupled to the endoscope 2 via a bus, a network, or the like.

Computer device 1 is any information processing device that includes at least a calculation function. Computer device 1 generates an image based on an imaging signal received from an imaging device (described below), and analyzes the correspondence between a reference image corresponding to the pattern light projected by the light projecting device and a captured image generated by capturing an image of a subject onto which the pattern light is projected.

The endoscope 2 includes an imaging device 21 (camera) and a light projecting device 22 built into the tip (head). Figure 1 shows an enlarged view of the imaging device 21, light projecting device 22, and endoscope head 23 within an ellipse O2 that corresponds to the ellipse O1 surrounding the tip of the endoscope 2. The endoscope head 23 is inserted into the human body, the light projecting device 22 projects pattern light PL onto the subject, and the imaging device 21 captures an image of the subject.

The imaging device 21 includes one or more CCD image sensors or CMOS image sensors, captures an image of the subject, and transmits an image signal to the computing device 1. It is desirable for the imaging device 21 to have an array of more image sensors in order to generate high-pixel images.

The light-projecting device 22 includes a diffractive optical element (DOE) 22a, a lens 22b, and an optical fiber 22c. Figure 1 shows an enlarged view of the diffractive optical element 22a, lens 22b, and optical fiber 22c within a rectangle R2 corresponding to a rectangle R1 surrounding the light-projecting device 22.

The diffractive optical element 22a diffracts the projected pattern light PL. The lens 22b is implemented, for example, by a gradient index (GRIN) lens. By implementing a gradient index lens, the refractive index of light can be changed in the radial direction, making it possible to reduce the size of the light projector 22. Light from a light source (not shown) propagates through the optical fiber 22c and emits the pattern light PL through the lens 22b and the diffractive optical element 22a. Details of the pattern light PL will be described later.

In this embodiment, the imaging device 21 and the light projector 22 are incorporated into the endoscope 2 to capture images of a subject inside the human body, but this embodiment is not limited to such a configuration. For example, to capture an arbitrary subject, an imaging device that captures the subject and a light projector that projects patterned light onto the subject may exist independently. In other words, this embodiment employs an active stereo system configuration that includes at least an imaging device that captures images of the subject and a light projector that projects patterned light onto the subject.

Next, detailed components of the computer device 1 will be described with reference to Figure 2. The computer device 1 includes a control device 11, memory 12, storage device 13, communication device 14, input device 15, and output device 16. The memory 12, storage device 13, communication device 14, input device 15, and output device 16 are each coupled to the control device 11 via an internal bus and are controlled by the control device 11.

The control device 11, also referred to as a processor, includes a central processing unit (CPU) and a graphics processing unit (GPU). The control device 11 generates an image based on the imaging signal received from the imaging device 21, and performs calculations such as analyzing the correspondence between the generated image and a reference image corresponding to the pattern light PL projected by the light projector 22 based on the pattern within the image.

Memory 12 is a volatile data storage device that stores computer-executable instructions processed by control device 11, as well as data after calculations performed by those instructions. Memory 12 may be implemented as RAM (random access memory) (e.g., SRAM (static RAM) and DRAM (dynamic RAM)).

Storage device 13 is a non-volatile data storage device that stores programs and other data including the computer-executable instructions described above. Storage device 13 may be implemented using non-volatile semiconductor memory such as ROM (read-only memory), a magnetic storage device (such as a hard disk drive), or an optical disk. Note that data such as programs may also be stored in a NAS (Network Attached Storage) and/or a SAN (Storage Area Network) in addition to or instead of storage device 13.

The communication device 14 communicates with external devices connected to the computer device 1, such as receiving an image capture signal from the endoscope 2 (imaging device 21) and transmitting a signal corresponding to a reference image to the endoscope 2 (light projector device 22).

The input device 15 accepts input from the user and transmits the accepted input to the control device 11. The input device 15 may be implemented, for example, as a mouse, touchpad, keyboard, or trackball.

The output device 16 outputs the results of calculations performed by the control device 11 (for example, displays the reference image generated by the control device 11). The output device 16 is implemented, for example, by a display (liquid crystal display, CRT, etc.).

Note that, although this embodiment employs a configuration in which the input device 15 and output device 16 are incorporated into the computer device 1, this configuration is not limiting. Either or both of the input device 15 and output device 16 may be configured as devices independent of the computer device 1.

Next, referring to Figure 3, the reference image RI that constitutes the pattern light PL emitted by the light projecting device 22 will be described. As described above, the image analysis system 100 projects the pattern light PL onto the subject. The reference image RI is an image that represents a predetermined pattern and is stored in the computer device 1 or the endoscope 2. The light projecting device 22 emits the pattern light PL based on the reference image RI.

The reference image RI includes a set of elements (a set of multiple first elements) SE1 extending vertically in a two-dimensional plane, and a set of elements (a set of multiple second elements) SE2 extending horizontally in the two-dimensional plane. In the example shown in FIG. 3 , the set of first elements SE1 is a set of straight lines extending vertically in the two-dimensional plane, each of which is a first element E1a to E1n (n is any integer equal to or greater than 2). The set of second elements SE2 is a set of straight lines extending horizontally in the two-dimensional plane, each of which is a second element E2a to E2n (n is any integer equal to or greater than 2). Each first element in the set of first elements SE1 intersects with each second element in the set of second elements SE2. Note that the value of n mentioned above and below refers to any integer equal to or greater than 2, and not all of the n values mentioned are the same value.

As shown in Figure 3, each straight line in the first set of elements SE1 extends continuously vertically in a two-dimensional plane within the reference image RI. Meanwhile, a portion of each straight line in the second set of elements SE2 extends intermittently horizontally in a two-dimensional plane within the reference image RI. Figure 3 shows an enlarged view of a portion of the reference image RI surrounded by circle C1, within circle C2, which corresponds to circle C1 that surrounds a portion of the reference image RI.

In the area within circle C2, first elements E1a to E1d each extend continuously in the vertical direction. Second elements E2a, E2d, and E2g extend intermittently in the horizontal direction. Second elements E2b, E2e, and E2h extend continuously in the horizontal direction. Second elements E2c, E2f, and E2i extend intermittently in the horizontal direction.

The second element E2a extends laterally to the right from the intersection I1 with the first element E1a and terminates at the intersection I2 with the first element E1b. The second element E2d extends laterally to the right from the intersection I3 with the first element E1b and terminates at the intersection I4 with the first element E1c. The second element E2g extends laterally to the right from the intersection I5 with the first element E1c and terminates at the intersection I6 with the first element E1d. The second element E2d extends from the intersection I3 at a higher position than the second element E2a, and the second element E2g extends from the intersection I5 at a lower position than the second element E2d.

The second element E2b extends laterally to the right from the intersection I7 with the first element E1a and connects to the second element E2e at the intersection I8 with the first element E1b. The second element E2e extends laterally to the right from the intersection I8 and connects to the second element E2h at the intersection I9 with the first element E1c. The second element E2h extends laterally to the right from the intersection I9 and connects to the next second element (unsigned) at the intersection I10 with the first element E1d.

The second element E2c extends laterally to the right from the intersection I11 with the first element E1a and terminates at the intersection I12 with the first element E1b. The second element E2f extends laterally to the right from the intersection I13 with the first element E1b and terminates at the intersection I14 with the first element E1c. The second element E2i extends laterally to the right from the intersection I15 with the first element E1c and terminates at the intersection I16 with the first element E1d. The second element E2f extends from the intersection I13 at a lower position than the second element E2c, and the second element E2i extends from the intersection I15 at a higher position than the second element E2f.

In addition, n markers of any shape are placed on the reference image RI at positions on predetermined intersections. In this embodiment, nine markers M1 to M9 are placed. The markers M1 to M9 may be placed at regular intervals on the two-dimensional plane of the reference image RI, or may be placed at random intervals. The same applies to the reference image shown in Figure 4, which will be described later. Each of the markers M1 to M9 is used to improve the accuracy of matching, and details will be given later.

Note that, in this embodiment, a reference image is employed that includes a set of elements that extend continuously vertically in a two-dimensional plane and a set of elements that extend intermittently horizontally in the two-dimensional plane; however, the reference image is not limited to such a configuration. For example, as shown in FIG. 4, the reference image RI may include a set of elements that extend intermittently vertically in a two-dimensional plane and a set of elements that extend continuously horizontally in the two-dimensional plane.

As shown in Figure 4, a portion of each straight line in the first set of elements SE1 extends intermittently vertically in a two-dimensional plane within the reference image RI. On the other hand, each straight line in the second set of elements SE2 extends continuously horizontally in a two-dimensional plane within the reference image RI. Figure 4 shows an enlarged view of a portion of the reference image RI surrounded by circle C1, within circle C2, which corresponds to circle C1 that surrounds a portion of the reference image RI.

In the area within circle C2, second elements E2a to E2d each extend continuously in the horizontal direction. First elements E1a, E1d, and E1g extend intermittently in the vertical direction. First elements E1b, E1e, and E1h extend continuously in the vertical direction. First elements E1c, E1f, and E1i extend intermittently in the vertical direction.

The first element E1a extends downward from the intersection I1 with the second element E1a and terminates at the intersection I2 with the second element E2b. The first element E1d extends downward from the intersection I3 with the second element E2b and terminates at the intersection I4 with the second element E2c. The first element E1g extends downward from the intersection I5 with the second element E2c and terminates at the intersection I6 with the second element E2d. The first element E1d extends from the intersection I3 to a position to the left of the first element E1a, and the first element E1g extends from the intersection I5 to a position to the right of the first element E1d.

The first element E1b extends downward from the intersection I7 with the second element E2a and connects to the first element E1e at the intersection I8 with the second element E2b. The first element E1e extends downward from the intersection I8 and connects to the first element E1h at the intersection I9 with the second element E2c. The first element E1h extends downward from the intersection I9 and connects to the next first element (unsigned) at the intersection I10 with the second element E2d.

The first element E1c extends downward from the intersection I11 with the second element E2a and terminates at the intersection I12 with the second element E2b. The first element E1f extends downward from the intersection I13 with the second element E2b and terminates at the intersection I14 with the second element E2c. The first element E1i extends downward from the intersection I15 with the second element E2c and terminates at the intersection I16 with the second element E2d. The first element E1f extends from the intersection I13 to the right of the first element E1c, and the first element E1i extends from the intersection I15 to the left of the first element E1f.

The patterns shown in Figures 3 and 4 can both be said to include a set of straight line elements extending vertically in a two-dimensional plane, and a set of straight line elements extending horizontally in a two-dimensional plane, each of which intersects with a set of straight line elements extending vertically. Furthermore, the above patterns can also be said to include elements in which at least one of the set of straight line elements extending vertically and the set of straight line elements extending horizontally extends intermittently.

An element extending intermittently in a two-dimensional plane means that the element intersects with the same element of the other intersecting elements at multiple positions, and the positions of the intersections are different in the two-dimensional plane. In the pattern shown in Figure 3, a second element extending intermittently in the horizontal direction in a two-dimensional plane means that a second element (e.g., second elements E2a and E2d) intersects with the same first element (e.g., first element E1b) at multiple positions (e.g., intersections I2 and I3), and the positions of the intersections are different in the two-dimensional plane. In the pattern shown in Figure 4, a first element extending intermittently in the vertical direction in a two-dimensional plane means that a first element (e.g., first elements E1a and E1d) intersects with the same second element (e.g., second element E2b) at multiple positions (e.g., intersections I2 and I3), and the positions of the intersections are different in the two-dimensional plane.

In this embodiment, the reference image RI includes a set of straight lines extending vertically in a two-dimensional plane and a set of straight lines extending horizontally in a two-dimensional plane, but is not limited to such a configuration. For example, the reference image RI may include a set of elements having an arbitrary shape, such as a set of curved lines extending vertically in a two-dimensional plane, and a set of elements having an arbitrary shape, such as a set of curved lines extending horizontally in a two-dimensional plane.

Next, an example of processing performed by the image analysis system 100 will be described with reference to the flowchart shown in Figure 5. As described above, in this embodiment, the correspondence between a reference image corresponding to a pattern light projected onto the subject and a captured image generated by capturing an image of the subject onto which the pattern light is projected is analyzed. Specifically, the correspondence between the reference image shown in Figure 3 or Figure 4 and a captured image generated by capturing an image of the subject onto which pattern light corresponding to the reference image is projected is analyzed. The correspondence is analyzed using a neural network.

When patterned light is projected onto a subject, the shape of the patterned light reflected from the subject is distorted depending on the depth, and the shape of the pattern recognized in the generated captured image is also distorted accordingly. With conventional image processing, it is not always possible to accurately recognize a pattern from a captured image that contains such a distorted pattern. In this embodiment, a neural network learns the nodes that make up the pattern in the reference image, and, based on information about the learned nodes, extracts the nodes that make up the pattern in the captured image by associating them with the nodes in the reference image.

In this embodiment, it is assumed that one of the reference images RI shown in Figure 3 is generated in advance and stored in the storage device 13 of the computer device 1. The reference image RI includes the markers M1 to M9 described above.

First, the light projecting device 22 of the endoscope 2 projects pattern light PL corresponding to the reference image RI onto the subject (step S501). The reference image RI is transmitted to the light projecting device 22 from the communication device 14 of the computer device 1 via the optical fiber 22c. The pattern light PL is diffracted by the diffractive optical element 22a and reaches the entire subject, which has depth.

Next, the imaging device 21 of the endoscope 2 captures an image of the subject onto which the pattern light PL is projected (step S502). After capturing an image of the subject, an imaging signal is transmitted from the imaging device 21 to the communication device 14 of the computing device 1, and the control device 11 generates a captured image DI based on the imaging signal. The generated captured image is stored in the memory 12 or the storage device 13.

Next, the control device 11 of the computer device 1 identifies the lines (columns) that make up all first elements in the set of first elements that make up the pattern in the captured image DI. Similarly, it identifies the lines (rows) that make up all second elements in the set of second elements (step S503). That is, the control device 11 identifies all columns and all rows in the captured image DI. The identified columns and rows are assigned identification numbers (first element identifiers to columns, second element identifiers to rows). As described above, some of the second elements extend intermittently in a two-dimensional plane, but for the intermittently extending elements, linear lines are recognized.

Next, the control device 11 identifies all intersections of the columns and rows identified in step S503 as nodes (step S504). Next, the control device 11 classifies each node based on its relative position to adjacent nodes (step S505). The identified and classified nodes are labeled based on their relative positions (labels are added to the nodes in the captured image DI (classification values are assigned)).

The processing of steps S503 to S505 is performed based on the learning data learned through deep learning. In this embodiment, the control device 11 implements a U-Net for learning. The U-Net is a fully convolutional network (FCN) that uses deep learning and estimates segmentation within an image. Segmentation refers to the process of dividing an image into multiple regions. The U-Net classifies what each pixel represents based on the learning data through convolution operations, activation operations (ReLU), max pooling operations, and other operations.

Figure 6 visually illustrates an example of the process of training a U-Net based on a captured image DI. While Figure 6 presents an example of training a U-Net based on a single image to illustrate an example of training in a U-Net, in reality, the system is trained to identify and classify nodes based on any reference image and any captured image generated by capturing an image of a target onto which patterned light corresponding to the reference image is projected. Note that the example shown in Figure 6 does not take into account the markers M1 to M9 described above. Also, Figure 6 shows only some of the patterns in the captured image DI.

First, as shown in FIG. 6(a), the control device 11 recognizes the captured image DI as learning data. The captured image DI includes a first set of elements SE1 extending vertically in a two-dimensional plane, and a second set of elements SE2 extending horizontally in the two-dimensional plane. Note that in the captured image DI, the first set of elements SE1 is distorted due to a deviation in the path of the pattern light caused by the depth of the subject.

Next, the user adds vertical marks VM to each of the sets of first elements SE1 to label each first element for identification. As shown in FIG. 6(b), the vertical marks VM are added by drawing a curve that traces each first element. The vertical marks VM are recognized as learning data by the control device 11.

Next, the control device 11 recognizes, for example, a predetermined area of the vertical mark VM and adds a mark. In this embodiment, as shown in FIG. 6(c), an area spanning a predetermined number of pixels on the right side of the vertical mark VM in the two-dimensional plane is recognized as vertical mark area VR1, and an area spanning a predetermined number of pixels on the left side of the vertical mark VM in the two-dimensional plane is recognized as vertical mark area VR2. Vertical mark area VR1 and vertical mark area VR2 are recognized as learning data by the control device 11.

Similarly, to add labels to identify each second element, the user adds horizontal marks to depict each set of second elements SE2. The horizontal marks are added by drawing a curve that traces each second element. The horizontal marks are recognized as learning data by the control device 11.

Next, the control device 11 recognizes, for example, a predetermined area of the horizontal mark and adds a mark. In this embodiment, as shown in FIG. 6(d), the area covering a predetermined number of pixels above the horizontal mark in the two-dimensional plane is recognized as horizontal mark area HR1, and the area covering a predetermined number of pixels below the horizontal mark in the two-dimensional plane is recognized as horizontal mark area HR2. Horizontal mark area HR1 and horizontal mark area HR2 are recognized as learning data by the control device 11.

As described above, the second element set SE2 includes elements that extend intermittently in a two-dimensional plane. The intermittently extending elements intersect with the same first element at two different points. In the example of FIG. 6(a), intersection I1 is located lower in the two-dimensional plane than intersection I2, and intersection I3 is located lower in the two-dimensional plane than intersection I4. Horizontal mark areas HR1 and HR2 are drawn as linear lines that cover these intersections. As shown by the area within the dashed line frame in FIG. 6(d), horizontal mark areas HR1 and HR2 extend upward and to the right in the two-dimensional plane corresponding to intersections I1 and I2, and extend downward and to the right in the two-dimensional plane corresponding to intersections I3 and I4.

Next, the control device 11 assigns an identification number (first element identifier) to the boundary between the vertical mark area VR1 and the vertical mark area VR2 as a first element. The first element to which the first element identifier is assigned is recognized as learning data. Similarly, the control device 11 assigns an identification number (second element identifier) to the boundary between the horizontal mark area HR1 and the horizontal mark area HR2 as a second element. The second element to which the second element identifier is assigned is recognized as learning data. Note that the first element identifier may have any symbol, shape, color, etc. to identify each first element. The same applies to the second element identifier.

In addition, U-Net may learn to recognize rows and columns by, for example, displaying a reference image on the output device 16 and having the user input vertical and horizontal marks on the displayed reference image via the input device 15.

Next, the control device 11 recognizes the intersection of the first element and the second element as a node. The control device 11 then classifies all nodes based on their relative position with respect to adjacent nodes and adds a label (assigns a classification value). In this embodiment, based on the relative positions of two adjacent nodes, the two nodes are classified into three classes: the two nodes are at the same or approximately the same height in a two-dimensional plane, the node located to the left of the two nodes is at a higher position in the two-dimensional plane, or the node located to the right of the two nodes is at a higher position in the two-dimensional plane.

In practice, it may be impossible to recognize the relative position of a node relative to its adjacent nodes, so such nodes may be classified as unknown. In this case, the nodes are classified into four classes. In this embodiment, the nodes are classified into four classes. Figure 6(e) shows the labeled nodes.

Two nodes being at the same or nearly the same height in a two-dimensional plane means, for example, that the difference in height between the two nodes in the two-dimensional plane is within a predetermined threshold range (for example, m pixels (m is an arbitrary number)). Nodes classified in this way are labeled red (R), and in Figure 6(e), the open circle represents the R label.

Nodes labeled with an R correspond to the intersections of continuously extending second and first elements in the pattern. In the example shown in Figure 6(a), the second elements that make up intersections I7 and I8 extend continuously, so nodes N4 and N5 are identified and labeled with an R in correspondence with intersections I7 and I8. Similarly, nodes N6 to N11, N14, and N16 to N20 also correspond to the intersections of continuously extending second and first elements, and are labeled with an R.

When the right-side node of two nodes is positioned higher on a two-dimensional plane, it means, for example, that the right-side node is positioned higher on a two-dimensional plane than the left-side node, and the difference in height between the two nodes on the two-dimensional plane exceeds a predetermined threshold. Nodes classified in this way are labeled green (G), and in Figure 6(e), a black circle represents the G label.

Nodes labeled with the G label correspond to two intersections between intermittently extending second elements and first elements in the pattern (the intersection with the second element on the right is higher than the intersection with the second element on the left). In the example shown in Figure 6(a), the second elements constituting intersections I1 and I2 extend intermittently, and intersection I2 with the second element on the right is higher than intersection I1 with the second element on the left. Therefore, node N1 is identified corresponding to intersections I1 and I2 and labeled with the G label. Similarly, the second elements constituting intersections I5 and I6 extend intermittently, and intersection I6 with the second element on the right is higher than intersection I5 with the second element on the left. Therefore, node N3 is identified corresponding to intersections I5 and I6 and labeled with the G label. Similarly, a G label is added to node N13, corresponding to the intersection between the intermittently extending second element and the first element.

When we say that the left-most node of two nodes is higher in a two-dimensional plane, it means, for example, that the left-most node is higher in a two-dimensional plane than the right-most node, and the difference in height between the two nodes in the two-dimensional plane exceeds a predetermined threshold. Nodes classified in this way are labeled blue (B), and in Figure 6(e), the shaded circle represents the B label.

Nodes labeled with the B label correspond to two intersections between discontinuously extending second elements and first elements in the pattern (the intersection with the second element on the left is at a higher position than the intersection with the second element on the right). In the example shown in Figure 6(a), the second elements constituting intersections I3 and I4 extend discontinuously, and intersection I3 with the second element on the left is at a higher position than intersection I4 with the second element on the right. Therefore, node N2 is identified in correspondence with intersections I3 and I4, and a B label is added. Similarly, nodes N12 and N15 correspond to intersections between discontinuously extending second elements and first elements, and are also labeled with the B label.

In this embodiment, an example has been shown in which nodes are classified into three classes based on their relative positions, but the number of classes to be classified is not limited to three. For example, if both the first element and the second element in the reference image RI include elements that extend intermittently, adjacent nodes may be classified into classes in which the upper node is located to the right (or left) of the lower node in a two-dimensional plane. Note that the use of colors to represent labels as described above is merely an example, and any symbol indicating relative position may be added as a label.

In this way, the captured image DI is input to U-Net, columns (first elements) and rows (second elements) are identified from the captured image DI, and nodes are identified and classified. Following the procedure described above, U-Net is trained to recognize columns and rows and identify and classify nodes based on any reference image and a captured image generated by capturing an object onto which a pattern of light corresponding to the reference image is projected.

For each node, U-Net outputs a first element identifier and a second element identifier (a node is identified by a pair of a first element identifier and a second element identifier), as well as a label. The first element identifier and the second element identifier used to identify the node are represented as a two-dimensional feature vector. The labels of the three classes (RGB) and the unknown class are represented as a four-dimensional feature vector F.

As described above, once the nodes have been identified and classified, the control device 11 assigns edges connecting the nodes based on the node classification (i.e., the positional relationship of the nodes) and their adjacency with adjacent nodes. In the processing of steps S503 to S505, a feature vector F containing a six-dimensional vector (two dimensions (first element identifier and second element identifier) + four dimensions (three classes + unknown class)) is extracted from the captured image DI. In addition, a lattice graph G is generated in which labels are added to the nodes.

Optionally, in the processing of steps S503 to S505, in order to improve the accuracy of the matching, the nodes may be classified and labels may be added to the nodes (assigned classification values) based on the predetermined areas in which the markers M1 to M9 placed in the reference image RI (markers M1 to M9 are also placed in the captured image DI) are located.

Figure 7 shows regions R1 to R9, which are predetermined regions centered on markers M1 to M9, respectively, within the reference image RI shown in Figure 3. For example, region R1 includes four grids made up of the first and second elements that make up the node where marker M1 is located, and the first and second elements adjacent to those. Regions R2 to R9 also include four grids. In other words, regions R1 to R9 each include the nodes where markers M1 to M9 are located, and the nodes adjacent to those nodes.

Regions R1 to R9 each have relationships such as identical, different, left-right symmetric, and vertically symmetric in the shape of a lattice made up of first and second elements. In other words, regions R1 to R9 have relationships such as identical, different, left-right symmetric, and vertically symmetric between the center node and adjacent nodes within the regions. Therefore, nodes within regions R1 to R9 are classified based on the relationship between the center node and adjacent nodes.

As shown in Figure 7, regions R1 to R5 have mutually different shapes. Region R6 has a shape that is vertically and horizontally symmetrical with region R4 in a two-dimensional plane. Region R7 has a shape that is horizontally symmetrical with region R2 in a two-dimensional plane. Region R8 has a shape that is horizontally symmetrical with region R3 in a two-dimensional plane. Region R9 has a shape that is vertically and horizontally symmetrical with region R1 in a two-dimensional plane. These classifications are also recognized by U-Net as learning data.

In steps S503 to S505, the captured image DI is input to U-Net, and markers M1 to M9 in the captured image DI are classified into five classes based on the learning data generated by the learning described above. In the example of Figure 7, for example, regions R1 and R9 are classified into class 1, regions R2 and R7 into class 2, regions R3 and R8 into class 3, regions R4 and R6 into class 4, and region 5 into class 5.

In practice, it may be impossible to identify the area surrounding a marker or its shape, so such markers may be classified as unknown. In this case, markers are classified into six classes. In this embodiment, markers are classified into six classes.

These classes can identify the relationship between the node where each marker is located and the nodes adjacent to that node. For example, in region R1, the node below the node where marker M1 is located (the node formed by intersections I1 and I2 (nodes not shown in Figure 7)) is labeled G (the node on the right is classified as being higher in a two-dimensional plane than the node on the left). Therefore, learning these positional relationships for each classified class makes it easier to analyze the correspondence between a reference image and an image generated by photographing an object onto which a patterned light corresponding to the patterned light projected onto the object is projected.

Through the optional processing described above, in steps S503 to S505, a feature vector F containing a 12-dimensional (2-dimensional + 4-dimensional + 6-dimensional (5 classes + unknown class)) vector is extracted from the captured image DI.

In this embodiment, first element identifiers and second element identifiers are assigned in U-Net, but this method is not limited to this. Numbers identifying each first element and numbers identifying each second element may be assigned in advance, and these numbers (arbitrary symbols identifying the elements) may be embedded in the reference image RI.

Furthermore, in this embodiment, a set of information indicating corresponding nodes and a set of information indicating corresponding relative positions are extracted from a U-Net that has learned the correspondence between each node in an image and its corresponding relative position, but the neural network used is not limited to a U-Net. For example, a known neural network other than a U-Net that has learned the correspondence between each node in an image and its corresponding relative position may also be used.

Alternatively, two adjacent nodes in an image may be recognized and their relative positions determined (e.g., by image processing such as Hough transform, projective transformation, and/or binarization) without using a neural network. That is, steps S503 through S505 all identify respective columns and rows in an image and classify nodes, regardless of whether a neural network is used.

Returning to the explanation of Figure 7, the control device 11 associates each node in the grid graph G with a node in the grid graph generated from the corresponding reference image RI based on the feature vector F and the grid graph G (step S506).

The above-mentioned U-Net generates graph G based on the results of learning the reference image RI, etc. However, U-Net is not always able to generate an accurate graph, and an incorrect graph will not allow the above-mentioned correspondence to be performed accurately.

In this embodiment, the control device 11 implements a graph convolutional network (GCN) for learning, and the processing of step S506 is executed based on learning data learned through deep learning using the GCN. A GCN is a neural network that applies deep learning to graph data, and performs convolution operations on the graph data. In convolution operations in a GCN, likelihood is calculated for each node according to a weight value for each relationship with an adjacent node. The feature vector F described above represents the properties (relative position) of each node itself within the captured image DI, and the lattice graph G represents the adjacent relationships between nodes.

The GCN is trained based on the feature vectors F and lattice graph G, as described above, extracted and generated from all reference images and all captured images. Therefore, based on the lattice graph G generated from the reference image RI described above, the GCN is trained to extract nodes that correspond to nodes in the reference image RI. Therefore, the output from the GCN is information representing the nodes in the input lattice graph G.

Figure 8 shows the configuration of the GCN 800 implemented in this embodiment. The GCN 800 performs a predetermined number of calculations on the input feature vector F and lattice graph G, and therefore employs a hierarchical structure according to the number of calculations. The GCN 800 includes a GCN layer 801, a fully connected layer 802, and an output layer 803.

In the GCN layer 801, based on the feature vector F and the lattice graph G, the graph convolution calculation unit 801a performs a graph convolution calculation, the normalization calculation unit 801b performs a normalization calculation (Batch Normalization), and the activation calculation unit 801c performs an activation calculation (ReLU). In this process, the above calculations are performed for each node on the 12-dimensional feature vector F. Therefore, a matrix H of D x N (N is the number of nodes, and D is the number of dimensions (12)) is generated.

The GCN performs calculations for each node based on the relationship with adjacent nodes. In this embodiment, each node in the grid graph G has adjacent nodes in four directions (upward, downward, rightward, and leftward) (having an adjacent relationship with adjacent nodes in four directions (hereinafter, the number of directions is represented by l)). Therefore, matrices H ⁽¹⁾ to H ^(l) are generated for each direction. These matrices are accumulated in the layer data matrix, and finally, matrix H ^(l+1) is generated. Matrix H ^(l+1) can be expressed by equation (1).

is the adjacency matrix of the grid graph G with self-joins added, and I is the identity matrix.

teeth,

is the order matrix of the layer, W ^(l) is the weight matrix of this layer, and σ is the activation function (ReLU). The adjacency matrix A is {A ₀ (upward), A ₁ (downward), A ₂ (rightward), A ₃ (leftward)}.

The calculation of the matrix H ^(l+1) is performed according to equation (2).

is a weight matrix according to the direction d∈{0, 1, 2, 3}.

After executing equation (2), normalization and activation operations are performed. This process is repeated five times. In this way, the GCN layer 801 outputs the matrix H ^(l+1) .

In the fully connected layer 802, a linear transformation operation unit 802a performs a linear transformation operation based on the matrix H ^(l+1) , and an activation operation unit 802b performs an activation operation (ReLU). In this process, the above operation is performed for each of D dimensions (12 dimensions in this embodiment) and N nodes. Thus, a feature embedding matrix (having a size of N×D) for the feature vector F is generated. In this way, the fully connected layer 802 outputs the feature embedding matrix for the feature vector F.

The above-described calculations allow for convolution calculations that take into account each node's own feature vector (i.e., a value indicating its relative position with respect to adjacent nodes), its relationship with adjacent nodes, and weighting, thereby improving the accuracy of the correspondence with each node in the original reference image RI. Figure 9 visually illustrates calculations that take into account a node's feature vector, its relationship with adjacent nodes, and its weighting.

In the output layer 803, the Softmax calculation unit 803a performs Softmax calculations for each node on the feature embedding matrix and outputs a log-likelihood vector for each node.

The GCN is trained based on the lattice graph G output from the U-Net. Specifically, the nodes in the lattice graph G that are identified and labeled by the U-Net are recognized as training data. In addition, a first element identifier is assigned to identify each first element in the lattice graph, and a second element identifier is assigned to identify each second element, and the first element identifier and second element identifier are used as training data.

When the GCN outputs a log-likelihood vector for each node, the corresponding node is determined based on the log-likelihood vector. In this determination, nodes determined to have a certain probability from the training data are extracted as nodes corresponding to nodes in the reference image RI. The GCN outputs a set of first element identifiers and a set of second element identifiers that reflect the nodes after the above-mentioned calculations have been performed. The first set of element identifiers and the second set of element identifiers make it possible to recognize columns and rows in the image, and therefore identify nodes that are their intersections.

In this embodiment, the GCN outputs a set of first element identifiers and a set of second element identifiers, taking into account the load of outputting all nodes. However, it is also possible to assign identifiers to all nodes (node identifiers) and output the node identifiers.

Note that nodes corresponding to nodes in the reference image are extracted from a GCN trained based on feature vectors for nodes in the pattern and a lattice graph formed by the nodes, but the neural network used is not limited to a GCN. For example, a known neural network other than a GCN trained based on feature vectors and a lattice graph formed by nodes may also be used. There may be a function to aggregate feature vectors associated with nodes in the graph based on graph connections, and other methods and other neural networks are also possible. Even when a network other than a GCN is used, calculations are performed for each node in the lattice graph, taking into account the relationship with adjacent nodes and weight values.

Alternatively, without using a neural network, nodes corresponding to nodes in the reference image may be extracted from a lattice graph formed by nodes based on the relative positions of nodes in the captured image to adjacent nodes (classification based on relative positions) and relationships with adjacent nodes (for example, by image processing such as spatial filtering and convolution operations). In other words, regardless of whether a neural network is used, the processing of step S506 extracts nodes corresponding to nodes in the reference image based on the input lattice graph and feature vectors (classification based on the relative positions of nodes in the lattice graph to adjacent nodes), taking into account relationships with adjacent nodes and weight values.

In this way, the GCN is used to identify each node in the captured image DI that corresponds to each node in the reference image RI. In this way, more nodes in the captured image DI can be associated with nodes in the reference image RI. Based on this association, the distance between the imaging device and the subject can be calculated using triangulation, and a three-dimensional image can be constructed based on this distance. Furthermore, based on the association, external parameters (used in epipolar geometry) indicating positional information between the imaging device 21 and the light projector 22 can be calculated.

As mentioned above, the shape of the reflected pattern light distorts depending on the depth of the subject, making it difficult to match all nodes in an image with the original nodes using conventional image processing. In this embodiment, even if the shape of the pattern is distorted by elements that extend intermittently vertically and/or horizontally on a two-dimensional plane, the relationship between adjacent nodes is maintained in the captured image DI. Based on this relationship, the accuracy of matching nodes in the original pattern using GCN can be improved. For example, in the captured image DI captured by projecting the reference image RI shown in Figure 3, the horizontal height relationship between two nodes on a two-dimensional plane is maintained even if the shape of the pattern is distorted.

In addition, because the reference image RI is constructed by including elements that extend intermittently in at least one of the first and second elements, it is possible to reduce the amount of information required to match nodes. Furthermore, because the adjacency relationship between nodes is determined based on classification based on marks M1 to M9 placed in the reference image RI, the accuracy of matching can be further improved.

In addition to the above processing, it is also possible to determine the relative position between pixels in the captured image DI and adjacent nodes, and the relative position between pixels in the reference image RI and adjacent nodes, and perform pixel-by-pixel correspondence based on the relative positions of both. This processing recognizes the relative position between adjacent nodes for each pixel in both the captured image DI and the reference image RI, which increases the processing load but can further improve the accuracy of correspondence. Such pixel-by-pixel correspondences are also learned by the GCN.

The correspondence between nodes based on the output from the GCN and the relative position of the node to its neighbors described above may be performed by a GCN or other neural network, or by image processing that does not use a neural network (e.g., spatial filtering and convolution operations).

Second Embodiment
Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that a reference image RI is input to the U-Net and the GCN together with the captured image DI, and the outputs from both GCNs are compared.

An example of processing executed by image analysis system 100 according to the second embodiment will be described with reference to Fig. 10. Steps S1001 to S1005 shown in Fig. 10 are similar to steps S501 to S505 shown in Fig. 5, and therefore description thereof will be omitted. Note that from step S1005, a feature vector _Fd and a grid graph _Gd are output from the captured image DI.

In step S1006, the control device 11 inputs the reference image RI stored in the storage device 13 to the U-Net and identifies the lines (columns) that make up the first element and the lines (rows) that make up the second element in the reference image RI. The method for identifying columns and rows is the same as the method described for step S503 shown in Figure 5.

Next, the control device 11 identifies all intersections between the columns and rows identified in step S1006 as nodes (step S1007). The method for identifying nodes is the same as the method described for step S504 in Figure 5.

Next, the control device 11 classifies each node based on its relative position with respect to adjacent nodes (step S1008). The method for classifying the nodes is the same as that described for step S505 in Fig. 5. From step S1008, a feature vector _Fp and a grid graph _Gp are output from the reference image RI.

Next, the control device 11 inputs the output of step S1005 (feature vector _Fd and lattice graph _Gd ) to the GCN and outputs the feature embedding matrix _Fd calculated from the captured image DI (step S1009). The method for calculating the feature embedding matrix _Fd is the same as the method described for step S506 in FIG. 5.

Next, the control device 11 inputs the output of step S1008 (feature vector _Fp and lattice graph _Gp ) to the GCN and outputs the feature embedding matrix _Fd calculated from the reference image RI (step S1010). The method for calculating the feature embedding matrix _Fp is the same as the method described for step S506 in FIG. 5.

Next, the control device 11 determines the similarity between the reference image RI and the captured image DI by calculating the inner product of the feature vectors F _d and F _d for each node for the feature embedding matrix F _d and the feature embedding matrix F _p (step S1011). The inner product of the feature vectors F _d and F _d is given by

Represented by:

The control device 11 derives the Softmax value of the dot product using the Softmax function. The CGN is trained by performing the above-described operations based on the lattice graphs G _p and G _d and using a cost function of the cross entropy between the dot product of two feature vectors and the first element identifier and the second element identifier. From the above-described training, the Softmax value for each node is derived using the Softmax function of the dot product, and a correspondence can be established between two images for each node by evaluating a log-likelihood vector for each node.

The processing according to the second embodiment is basically the same as that according to the first embodiment, but also performs calculations on the reference image via GCN, calculates the inner product of the feature vectors of both the reference image RI and the captured image DI, and evaluates the log-likelihood vector for each node. Figure 11 visually represents this processing.

The second embodiment has been described above. The second embodiment also makes it possible to improve the accuracy of matching nodes in a captured image with nodes in a reference image.

Third Embodiment
Next, a third embodiment will be described. The third embodiment differs from the first and second embodiments in that the reference image RI used is different.

The reference image RI according to the third embodiment will be described with reference to Figure 12. As shown in Figure 12, the reference image RI includes a collection of rectangular elements extending continuously in the vertical direction on a two-dimensional plane and a collection of arbitrary symbol elements (in this embodiment, crosses) extending intermittently in the vertical direction on a two-dimensional plane.

As shown in FIG. 12, the first elements (shaded rectangles shown in FIG. 12) E1a to E1n (n is any integer) in the set of first elements SE1 extend continuously vertically in a two-dimensional plane within the reference image RI. Meanwhile, the second elements (cross symbols shown in FIG. 12) E2a to E2n (n is any integer) in the set of second elements SE2 extend intermittently vertically in a two-dimensional plane within the reference image RI. The second elements E2a to E2n are each arranged along the first elements E1a to E1n. FIG. 12 shows an enlarged view of a portion of the reference image RI enclosed by circle C1, within circle C2 corresponding to circle C1 enclosing a portion of the reference image RI.

In the area within circle C2, first elements E1a to E1c each extend continuously in the vertical direction. Second elements E2a to E2n are arranged at random intervals in the vertical direction. For example, the distance between second element E2b and second element E2e is different from the distance between second element E2e and second element E2h. Similarly, the distance between second element E2c and second element E2f is different from the distance between second element E2f and second element E2i.

Because second elements E2a to E2n are arranged at random intervals vertically in a two-dimensional plane, the angle at which a first second element is positioned relative to an adjacent second element in the two-dimensional plane is different from the angle at which a second second element is positioned relative to an adjacent second element in the two-dimensional plane. For example, in the area within circle C2, the angle at which second element E2b is positioned relative to adjacent second element E2c is different from the angle at which second element E2c is positioned relative to adjacent second element E2d. Similarly, the angle at which second element E2b is positioned relative to adjacent second element E2c is different from the angle at which second element E2h is positioned relative to adjacent second element E2i.

Note that, in this embodiment, a reference image is employed that includes a set of rectangular elements that extend continuously vertically in a two-dimensional plane and a set of symbol elements that extend intermittently vertically in the two-dimensional plane; however, the reference image is not limited to such a configuration. For example, although not shown, the reference image may include a set of rectangular elements that extend continuously horizontally in a two-dimensional plane and a set of symbol elements that extend intermittently horizontally in the two-dimensional plane. In this case, the set of symbol elements is arranged at random intervals in the horizontal direction.

Although not shown, n markers may also be arranged in the reference image RI shown in Figure 12, similar to the reference image RI shown in Figure 3.

For the reference image RI used in the third embodiment, a corresponding pattern light PL is projected onto the subject to be photographed, generating a photographed image DI. The method for processing the photographed image DI according to the third embodiment is similar to either of the methods described in the first and second embodiments, but differs from the processing according to the first and second embodiments in that only columns within the image are identified and nodes are classified based on the relative angles between adjacent nodes.

As described above, in the reference image RI used in the third embodiment, both the first and second elements extend vertically in a two-dimensional plane, so in the captured image DI, only columns within the image are identified. Columns are identified, for example, by tracing the edge of the first element or the second element (symbol) and drawing a linear line that extends vertically in the two-dimensional plane. This processing is similar to the method described with reference to Figures 6(b) and 6(c).

When identifying nodes, for example, as shown in the rectangular area labeled NR1 in Figure 12, a certain area between a first element and a second element is identified as a node. In other words, it is divided into certain areas based on the classified labels (relative angles (feature vectors) to adjacent nodes) described below. The node area NR1 in Figure 12 is divided as a certain area on the first element E1b adjacent to the second element E2f, and area NR1 is identified as a node.

Similarly, node region NR2 is divided as a fixed region on the first element E1b adjacent to the second element E2i, and region NR2 is identified as a node. Node region NR3 is divided as a fixed region on the first element E1c adjacent to the second element E2g, and region NR3 is identified as a node. The divided regions form multiple lattices, each containing a node. Such node regions may also be identified by the U-Net learning described above. The node regions shown in Figure 12 are merely examples, and a fixed region between the first element and the second element may be identified as a node according to predetermined rules.

In the captured image DI generated from the subject onto which pattern light PL corresponding to the reference image RI used in the third embodiment is projected, a certain area between the first element and the second element is identified as a node, as described above. In the third embodiment, there is no need to identify rows when identifying nodes, which simplifies calculation processing using U-Net, etc.

In the third embodiment, nodes are not identified by the intersection of columns and rows, as in the processing in the first embodiment. Measurement accuracy can be improved by projecting the pattern shown in Figure 12 and capturing an image, while calculating the correspondence for each pixel from the captured image. To achieve this, the U-Net can be trained to extract the grid phase from the captured image, that is, the relative position of each pixel based on the grid. Figure 13 shows an example of estimating the grid phase (the relative positional relationship between the repeating grid and each pixel, expressed as a rotation angle between 0 and 2π) from a captured image.

Figure 13 shows an example of grid information for a captured image, where the relative positional relationship between the grid and each pixel is expressed as a rotation angle. This rotation angle is linked to the repetition of the grid, rotating once for each grid. In other words, it increases from 0 to 2π and then returns to 0. Figure 13(a) shows an image projected with the pattern in Figure 12. Figure 13(b) shows an image in which, as described above, a cosine signal is estimated using U-Net so that the rotation angle is 0 at the grid division positions. Figures 13(b) and 13(c) show images in which a cosine signal is estimated using U-Net from the captured image shown in Figure 13(a) so that the rotation angle is 4π/5 at the grid division positions. Figure 13(d) shows the results of calculating grid phase information for each pixel from the phase estimation results including Figures 13(b) and 13(c). Figure 13(e) shows the results of calculating vertical grid phase information for each pixel.

The estimation of phase information (rotation angle) of the captured image shown in Figure 13 is merely an example, and it is also possible to detect the phase information of the lattice directly using U-Net without estimating the cosine signal. However, learning to detect the cosine signal will result in higher accuracy in estimating the phase information than directly detecting the phase information using U-Net. Gabor filters, etc. can also be used. As mentioned above, the phase information of the lattice may be identified using a neural network other than U-Net or other image processing.

In this embodiment, graph nodes are identified by dividing the phase information for each pixel shown in Figure 13 into regions, using 0-degree phase sections as region separators (i.e., dividing the captured image into multiple regions based on the relative position of each pixel and the grid information on the captured image). The node adjacency relationships are also extracted from the region adjacency relationships. Once the nodes are identified, the control device 11 uses a U-Net or similar to classify each node based on the image features around the node and assigns a label (assigns a classification value). Once the nodes are identified and classified, the control device 11 assigns edges connecting the nodes based on the adjacency relationships with adjacent nodes.

Note that when extracting nodes through region division, information other than topological information may be used. For example, a certain range from the grid may be recognized as a node using U-Net, and region division may be performed using techniques such as Delaunay division with that node at the center. Region division based on topological information may also be combined with region division centered on a node. Furthermore, connections between nodes may be based on proximity relationships in addition to adjacency relationships. In this case, the output graph will be a general graph rather than a grid graph. Figure 14 shows an example of connections based on proximity relationships.

In this embodiment, for each node, the relative positions of the two points associated with that node are classified into five classes: the two points are horizontal or nearly horizontal in a two-dimensional plane; the left-side point of the two points is at a positive angle in the two-dimensional plane relative to the right-side point, and that angle exceeds a predetermined angle; the left-side point of the two points is at a positive angle in the two-dimensional plane relative to the right-side point, and that angle is equal to or less than a predetermined angle; the left-side point of the two points is at a negative angle in the two-dimensional plane relative to the right-side point, and that angle is equal to or less than a predetermined angle; and the left-side point of the two points is at a negative angle in the two-dimensional plane relative to the right-side node, and that angle exceeds a predetermined angle.

A positive angle means that of two adjacent points, the point located to the left is lower on a two-dimensional plane than the point located to the right. A negative angle means that of two adjacent points, the point located to the left is higher on a two-dimensional plane than the point located to the right.

In practice, it may be impossible to recognize the relative positions of two points associated with a certain node, so such nodes may be classified as unknown. In this case, nodes are classified into six classes. In this embodiment, nodes are classified into six classes. Figure 15 shows labeled nodes.

When the left-hand point of two points is at a positive angle on a two-dimensional plane relative to the right-hand point, and that angle exceeds a predetermined angle, this is equivalent to, for example, the point on the left being lower on a two-dimensional plane than the point on the right, and the difference in height between the two points on the two-dimensional plane exceeding a predetermined threshold. Points classified in this way are labeled green (G), and in Figure 15, a black circle represents the G label. In the example shown in Figure 15, point N1 is lower on a two-dimensional plane than point N6, and the G label is added assuming that the angle of point N1 relative to point N6 exceeds the threshold.

When the left-hand point of two points is at a positive angle on a two-dimensional plane relative to the right-hand point, and that angle is less than or equal to a predetermined angle, this is equivalent to, for example, the left-hand point being lower on a two-dimensional plane than the right-hand point, and the difference in height between the two points on the two-dimensional plane being less than or equal to a predetermined threshold. Points classified in this way are labeled blue (B), and in Figure 15, the shaded circle represents the B label. In the example shown in Figure 15, point N2 is lower on a two-dimensional plane than point N7, and the B label is assigned based on the assumption that the angle between point N2 and point N7 is less than or equal to the threshold.

Two points being horizontal or nearly horizontal on a two-dimensional plane means, for example, that the difference in height between the two points on the two-dimensional plane is within a predetermined threshold range. Points classified in this way are labeled yellow (Y), and in Figure 15, a shaded circle (lighter than the B label) represents the Y label. In the example shown in Figure 15, point N3 is located horizontally or nearly horizontally with point N8, so the Y label is assigned to it.

When the left-hand point of two points is at a negative angle on a two-dimensional plane relative to the right-hand point, and that angle is less than or equal to a predetermined angle, this is equivalent to, for example, the left-hand point being higher on a two-dimensional plane than the right-hand point, and the difference in height between the two points on the two-dimensional plane being less than or equal to a predetermined threshold. Points classified in this way are labeled purple (P), and in Figure 15, a shaded circle (lighter than the Y label) represents the P label. In the example shown in Figure 15, point N4 is higher on a two-dimensional plane than point N9, and the P label is assigned assuming that the angle of point N4 relative to point N9 is less than or equal to the threshold.

When the left-hand point of two points is at a negative angle on a two-dimensional plane relative to the right-hand point, and that angle exceeds a predetermined angle, this is equivalent to, for example, the point on the left being higher on a two-dimensional plane than the point on the right, and the difference in height between the two points on the two-dimensional plane exceeding a predetermined threshold. Points classified in this way are labeled red (R), and in Figure 15, a white circle represents the R label. In the example shown in Figure 15, point N5 is higher on a two-dimensional plane than point N10, and the R label is added assuming that the angle of point N5 relative to point N10 exceeds the threshold.

In the third embodiment, after determining the correspondence for each node as in the first embodiment, the correspondence information for each node is combined with the phase information determined by U-Net to determine the correspondence information for each pixel. Specifically, by treating the correspondence information for each node adjacent to a pixel as an integer value and the phase information as a decimal value between 0 and 1, and adding them together, it is possible to determine the correspondence information for each pixel with real number precision.

In this embodiment, an example has been shown in which nodes are classified into five classes based on the relative angle between horizontally adjacent points, but the number of classes is not limited to five. For example, if both the first element and the second element in the reference image RI include elements that extend intermittently, they may be classified into further classes based on the relative angles between vertically and horizontally adjacent points. Note that representing labels with the above-mentioned colors is merely an example, and any symbol indicating relative position may be added as a label.

For the reference image RI shown in the third embodiment, even if the corresponding pattern light PL is projected onto the subject and the shape of the pattern is distorted, the relationship with adjacent nodes is maintained in the captured image DI. Therefore, based on this relationship, the accuracy of matching with nodes in the original pattern using GCN can be improved.

In the third embodiment, correspondence between nodes may also be performed based on the output from the GCN and the relative angles between points associated with the nodes (classification values assigned based on the relative angles) (by using a GCN or other neural network, or by image processing that does not use a neural network (e.g., spatial filtering and convolution operations)).

Instead of the reference image RI shown in Figures 3, 4, and 12, the pattern may be configured with a pattern including a square grid. Such a pattern itself cannot represent the differences in relative position and relative angle described above, but the differences may be represented by placing any symbols representing the relative position and relative angle within the grid. In other words, the reference image RI represents the differences in the relative position and/or relative angle described above in either the first element or the second element.

Please note that the hardware components described in the above embodiments are merely exemplary, and other configurations are possible. Furthermore, the processing steps described in the above embodiments do not necessarily have to be performed in the order described, and may be performed in any order. Furthermore, additional steps may be added without departing from the basic concept of the present invention.

The image analysis method according to one embodiment of the present invention is implemented by a computer program executed by the control device 11 (processor) of the computer device 1, and the computer program may be stored on a non-transitory storage medium. Examples of non-transitory storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media such as internal hard disks and removable disk devices, magneto-optical media, and optical media such as CD-ROM disks and digital versatile disks (DVDs).

Claims (17)

A computer-implemented method for analyzing a correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image represents that at least one of the first set of elements and the second set of elements includes elements extending intermittently;
identifying, within the captured image, respective nodes formed from the first set of elements and the second set of elements;
assigning a classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes;
generating a graph from the classified nodes;
For each node in the graph, determining its adjacency with adjacent nodes;
matching nodes in the graph with nodes in the reference image based on the classification values and the adjacency relationships;
A method comprising:
The reference image includes n markers, and the method comprises:
identifying n regions each containing said n markers;
For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value;
matching nodes in the graph with nodes in the reference image further based on the second classification value;
The method further comprising : the step of associating nodes in the graph with nodes in the reference image is performed by using a neural network, the neural network being trained to output correspondences of nodes in the graph to nodes in the reference image based on the classification values and the adjacency relationships.
2. The method of claim 1. The neural network is a graph convolutional network (GCN).
3. The method of claim 2. If the second set of elements represents intermittently extending elements, the reference image is
a plurality of second elements in the set of second elements intersect with a single first element in the set of first elements at a plurality of locations;
a first height in a two-dimensional plane of an intersection between a first element of the plurality of second elements and the first element is different from a second height in a two-dimensional plane of an intersection between a second element of the plurality of second elements and the first element;
3. The method according to claim 1 or 2. the relative position is based on the first height being different from a second height represented by the reference image;
5. The method of claim 4. In each of the n regions,
For a lattice consisting of the central node and adjacent nodes in the region,
Based on whether the shapes of the lattices are the same, different, and symmetrical,
The second classification value is assigned .
2. The method of claim 1 . the steps of identifying nodes, assigning classification values, and generating a graph are performed by using a second neural network, the second neural network having trained the reference images to identify nodes from the reference images, assign classification values, and generate a graph.
7. The method according to any one of claims 1 to 6 . The second neural network is a U-Net.
8. The method of claim 7 . For each pixel in the captured image,
determining a relative position of the identified nodes with respect to nodes that are proximate to the pixel;
Corresponding pixels in the captured image to pixels in the reference image based on the determined relative positions and relative positions of the pixels in the reference image relative to nodes adjacent to the corresponding pixels;
9. The method of any one of claims 1 to 8 , further comprising: identifying respective nodes in the reference image that are comprised of the first set of elements and the second set of elements;
assigning a third classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes;
generating a second graph from the classified nodes;
and for each node in the second graph, determining an adjacency relationship with adjacent nodes;
The step of associating nodes in the graph with nodes in the reference image comprises:
generating a first feature vector for each node in the graph by inputting the graph into a neural network;
generating a second feature vector for each node in the second graph by inputting the second graph into the neural network;
calculating , for each node, an inner product value of the first feature vector and the second feature vector;
evaluating the dot product value;
10. The method according to any one of claims 1 to 9 , comprising: A computer-implemented method for analyzing a correspondence between a reference image including a pattern and a captured image generated from a subject onto which a pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, and the reference image represents each of the second set of elements being arranged at random intervals in the same direction;
identifying, within the captured image, respective nodes formed from the first set of elements and the second set of elements;
assigning a classification value to each of the nodes by classifying the node based on its relative angle to adjacent nodes;
generating a graph from the classified nodes;
For each node in the graph, determining its adjacency with adjacent nodes;
matching nodes in the graph with nodes in the reference image based on the classification values and the adjacency relationships;
A method comprising: The step of identifying the node is performed by using a neural network, and the neural network is trained on the captured image and the captured image rotated at a predetermined angle to identify the first element or the second element.
12. The method of claim 11 . the reference image represents that an angle of a first element of the set of second elements relative to an adjacent second element is different from an angle of a second element of the set of second elements relative to an adjacent second element;
13. The method according to claim 11 or 12 . the step of generating the graph includes a step of dividing the captured image into a plurality of regions including the nodes based on a relative position of each pixel in the captured image and grid information on the captured image;
14. The method according to any one of claims 11 to 13 . A computer device that analyzes a correspondence relationship between a reference image including a pattern and a captured image generated from a subject onto which pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically in a two-dimensional plane and a second set of elements extending horizontally in the two-dimensional plane, and the reference image indicates that at least one of the first set of elements and the second set of elements includes elements that extend intermittently;
Identifying nodes in the captured image that are each made up of the first set of elements and the second set of elements;
assigning a classification value to each of the nodes by classifying the node based on its relative position to adjacent nodes;
generating a graph from the classified nodes;
For each node in the graph, determining an adjacency relationship with adjacent nodes;
Corresponding nodes in the graph with nodes in the reference image based on the classification values and the adjacency relationships;
a control device configured to
The reference image includes n markers, and the control device
identifying n regions each containing said n markers;
For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value;
Corresponding nodes in the graph with nodes in the reference image further based on the second classification value.
1. A computing device comprising: A computer device that analyzes a correspondence relationship between a reference image including a pattern and a captured image generated from a subject onto which pattern light corresponding to the reference image is projected, wherein the reference image includes a first set of elements extending vertically or horizontally in a two-dimensional plane and a second set of elements extending intermittently in the same direction as the first set of elements, and the reference image represents each of the second set of elements being arranged at random intervals in the same direction;
Identifying nodes in the captured image that are each made up of the first set of elements and the second set of elements;
assigning a classification value to each of the nodes by classifying the node based on its relative angle to adjacent nodes;
generating a graph from the classified nodes;
For each node in the graph, determining an adjacency relationship with adjacent nodes;
Corresponding nodes in the graph with nodes in the reference image based on the classification values and the adjacency relationships;
a control device configured to
The reference image includes n markers, and the control device
identifying n regions each containing said n markers;
For each of the n regions, classifying the nodes based on a relationship between a central node and adjacent nodes within the region and assigning a second classification value;
Corresponding nodes in the graph with nodes in the reference image further based on the second classification value.
1. A computing device comprising: A computer program comprising computer-executable instructions, which when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 14 .
A computer program characterized by: