JP7614537B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

Texture mapping technology is known as a technique for placing surface information on a 3D model (see, for example, Patent Document 1). Texture mapping technology creates highly realistic depictions by placing an image on the surface of a 3D model. To perform texture mapping using an image, information on the position and orientation of the camera when capturing the image is required (see process i in step 1 in Figure 7 of Patent Document 1).

JP 2016-119086 A

Although the conventional methods described above enable realistic image confirmation, they do not allow for judgment on whether repairs are required. This is because judgment on whether repairs are required requires the skills of an expert. This makes it difficult to quantitatively evaluate the strength of the structure.

In addition, the above-mentioned conventional methods cannot convert inspection images containing deterioration information such as cracks into 3D images. As a result, it is not possible to perform facility management based on structural strength evaluations that include deterioration information.

In the conventional methods described above, the position and orientation of the camera in CAD space cannot be estimated from the image, so images containing information about the deterioration of the structure cannot be mapped onto a CAD model. A technology is needed that can create three-dimensional data containing information about the deterioration by mapping, and estimate the structural strength.

In consideration of these points, the object of the present disclosure is to provide an information processing device, an information processing method, and a program that can appropriately map a captured image of an object onto a CAD model.

In order to solve the above problems, the information processing device according to the present disclosure includes:
An information processing device including a control unit,
Obtaining a CAD model of an object;
converting the CAD model into a CAD model point cloud;
acquiring a plurality of images of the object;
applying a 3D modeling algorithm to the images to obtain a point cloud and information indicative of the position and orientation of a camera that captured the images;
performing an alignment between the CAD model point cloud and the acquired point cloud to obtain a transformation matrix from the acquired point cloud to the CAD model point cloud;
converting the information indicating the position and orientation of the camera into information indicating the position and orientation of the camera in a CAD space using the transformation matrix;
extracting degradation information of the object from the plurality of images;
applying the degradation information to the CAD model using information indicating a position and orientation of a camera in the CAD space to obtain a degraded CAD model;
Execute.

In order to solve the above problem, the information processing method according to the present disclosure includes:
An information processing method by an information processing device,
obtaining a CAD model of the object;
converting the CAD model into a CAD model point cloud;
acquiring a plurality of images of the object;
applying a 3D modeling algorithm to the images to obtain a point cloud and information indicative of the position and orientation of a camera that captured the images;
performing a registration between the CAD model point cloud and the acquired point cloud to obtain a transformation matrix from the acquired point cloud to the CAD model point cloud;
converting the information indicating the position and orientation of the camera into information indicating the position and orientation of the camera in a CAD space using the transformation matrix;
extracting degradation information of the object from the plurality of images;
applying the degradation information to the CAD model using information indicating a position and orientation of a camera in the CAD space to obtain a degraded CAD model;
Includes.

In order to solve the above problems, the program according to the present disclosure comprises:
The computer functions as the information processing device.

The information processing device, information processing method, and program disclosed herein enable a captured image of an object to be appropriately mapped onto a CAD model.

1 is a functional block diagram of an information processing apparatus according to an embodiment of the present invention; FIG. 2 is a schematic diagram of a flowchart of the operation of the information processing device of the present embodiment. 3 is a flow chart illustrating the schematic diagram shown in FIG. 2 . FIG. 1 is a diagram showing the contents of SfM processing. FIG. 13 is a diagram showing a method for calculating a value in the depth direction. FIG. 2 is a hardware block diagram of the information processing device.

Figure 1 is a functional block diagram of information processing device D of this embodiment. Information processing device D has a control unit 1, a memory unit 2, and an input/output unit 3. Each component of information processing device D is connected to each other so that they can communicate with each other, for example, via a dedicated line.

The processing performed by the information processing device D of the embodiment may be performed by multiple information processing devices D in a distributed arrangement.

Information processing device D is a computer such as a server belonging to a cloud computing system or other computing system.

The internal configuration of information processing device D is described in detail. Below, each function of information processing device D is described, but this is not intended to exclude other functions possessed by information processing device D.

The control unit 1 includes, for example, one or more general-purpose processors including a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). The control unit 1 may include one or more dedicated processors specialized for specific processing. Instead of including a processor, the control unit 1 may include one or more dedicated circuits. The dedicated circuits may be, for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 1 may include an ECU (Electronic Control Unit).

The memory unit 2 includes, but is not limited to, for example, a semiconductor memory, a magnetic memory, an optical memory, or a combination of at least two of these. The semiconductor memory is, for example, a RAM (Random Access Memory) or a ROM (Read Only Memory). The RAM is, for example, an SRAM or a DRAM. The ROM is, for example, an EEPROM. The memory unit 2 may function, for example, as a main memory device, an auxiliary memory device, or a cache memory. The memory unit 2 may store information resulting from analysis or processing by the control unit 1. The memory unit 2 may store various information related to the operation or control of the information processing device D. The memory unit 2 may store a system program, an application program, embedded software, etc. The memory unit 2 may be provided outside the information processing device D and may be accessed from the information processing device D. The memory unit 2 stores an image of an object, a CAD model, information indicating the position and attitude of the camera, deterioration information, etc., which will be described later.

The input/output unit 3 includes at least one input interface and at least one output interface. For example, the input/output unit 3 can accept inputs such as drawings of an object and multiple images of the object from the outside.

Referring to Figure 2, an outline of a flowchart of operation in the information processing device D of this embodiment is described.

In step S1, the control unit 1 of the information processing device D acquires a CAD model CM from a drawing FI of an object (e.g., a structure). The control unit 1 converts the CAD model CM into a CAD model point cloud CP.

In step S2, the control unit 1 acquires multiple images IM of the object, and then performs SfM (Structure from Motion) processing on the multiple images IM to acquire an SfM point cloud SP and information SC indicating the position and orientation of the camera obtained by the SfM processing. The images IM may be, for example, inspection images of the object. As an alternative, another 3D model creation algorithm may be used instead of the SfM processing.

In step S3, the control unit 1 performs alignment between the CAD model point cloud CP and the SfM point cloud SP using an Iterative Closet Point (ICP) algorithm to obtain a transformation matrix. Alternatively, another alignment algorithm may be used instead of the ICP algorithm.

In step S4, the control unit 1 applies the acquired transformation matrix to the information SC indicating the position and orientation of the camera obtained by the SfM processing. The control unit 1 converts the information SC indicating the position and orientation of the camera into information CC indicating the position and orientation of the camera in the CAD space including the CAD model CM.

In step S5, the control unit 1 extracts deterioration information of the object from the multiple images. The deterioration information includes, for example, information on the deterioration content and deterioration position of the object. The control unit 1 acquires the image DI from which the deterioration information has been extracted.

In step S6, the control unit 1 assigns degradation information to the CAD model CM of the object using information CC indicating the position and orientation of the camera in the CAD space. The control unit 1 acquires a degraded CAD model DC.

In step S7, the control unit 1 performs structural calculations of the object using the degraded CAD model DC.

As described above, the control unit 1 uses multiple planar images to estimate information CC indicating the position and orientation of the camera in CAD space. The control unit 1 further uses information CC indicating the position and orientation of the camera to map the image DI from which degradation information has been extracted onto the CAD model CM. The control unit 1 acquires a deteriorated CAD model DC and realizes a quantitative structural strength assessment of the object.

The flowchart of FIG. 2 is explained in detail with reference to FIG. 3. The reference numbers in FIG. 3 correspond to the reference numbers in FIG. 2.

In step S1, the control unit 1 acquires a CAD model CM from the drawing FI of the object, cuts out a mesh from the CAD model CM, and converts the CAD model CM into a CAD model point cloud CP.

In step S2, the control unit 1 acquires multiple images IM of the object, then performs SfM processing on the multiple images IM to acquire an SfM point cloud SP and information SC indicating the position and attitude of the camera obtained by the SfM processing.

Information SC indicating the position and orientation of the camera is information indicating the position and orientation of the camera when the image IM was captured. Information SC is not information indicating the position and orientation of the camera in CAD space.

An example of the SfM process will be described in detail with reference to Fig. 4. The SfM process is a technique for creating a 3D model from an image. The SfM process is described in, for example, the following literature:
[Non-Patent Document 1] Furukawa, Y., et al. (2010), "Towards Internet-scale Multi-view stereo", 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

In step S11, the control unit 1 extracts feature points from each of the multiple images and performs matching.

In step S12, the control unit 1 selects the image to be restored.

In step S13, the control unit 1 estimates the position and orientation of the camera that captured the selected image.

In step S14, the control unit 1 estimates the three-dimensional position of the object.

In step S15, the control unit 1 performs an optional overall optimization process.

The control unit 1 repeats steps S11 to S15. The control unit 1 outputs information indicating the position and orientation of the camera and information indicating the three-dimensional position of the target object.

Returning to FIG. 3, in step S3, the control unit 1 performs alignment between the CAD model point cloud CP and the SfM point cloud SP, for example, using an ICP algorithm. The control unit 1 obtains a transformation matrix TM from the SfM point cloud SP to the CAD model point cloud CP.

The ICP algorithm is one of the algorithms for aligning multiple 3D point groups. The ICP algorithm is described in, for example, the following document.
[Non-Patent Document 2] Besl, PJ and ND McKay (1992), "A method for registration of 3-D shapes", IEEE Transactions on Pattern Analysis and Machine Intelligence 14(2): 239-256.

The ICP algorithm estimates a rotation matrix R and a translation matrix T that align _Ps with _Pt when two-dimensional point groups _Ps and _Pt acquired from different positions are given. Specifically, the control unit 1 executes the following steps 1 to 4.

Step 1. Using the nearest neighbor detection algorithm described in the following non-patent document 3, the nearest point P _tj is matched to each point P _si of P _s .
[Non-Patent Document 3] Eggert, DW, et al. (1997), "Estimating 3-D rigid body transformations: a comparison of four major algorithms", Machine Vision and Applications 9(5): 272-290.
Step 2. Define an objective function E between the corresponding points determined in step 1 above, and estimate R and T at which E converges.
Step 3. After moving _Ps by R and T obtained in step 2 above, select the corresponding point of _Pt that is closest to each point.
Step 4. Repeat steps 1 to 3 above.

Returning to FIG. 3, in step S4, the control unit 1 multiplies the transformation matrix TM by the information SC indicating the position and orientation of the camera obtained by the SfM processing. The control unit 1 converts the information SC into information CC indicating the position and orientation of the camera in the CAD space.

In step S5, the control unit 1 uses a deep neural network (DNN) to detect the deterioration content and estimate the deterioration position from the multiple images IM. The control unit 1 acquires the image DI from which the deterioration information has been extracted. As an alternative, another machine learning method, deep learning, etc. may be used instead of the DNN.

In step S6, the control unit 1 calculates the three-dimensional position of the pixel in the image DI from which the degradation information was extracted, based on the information CC indicating the position and orientation of the camera in the CAD space. As a result, the control unit 1 assigns the degradation information to the CAD model CM. The control unit 1 acquires the degraded CAD model DC.

In SfM processing, there are often areas that cannot be reconstructed in three dimensions, and the scaled SfM model cannot be used as a CAD model that reflects detailed degradation information. Therefore, with reference to Figure 5, a method for calculating the depth value in calculating the three-dimensional position of a pixel in an image DI from which degradation information has been extracted is described in detail.

As shown in Figure 5, consider the camera coordinate system. The three-dimensional coordinates of a pixel in the image are (x, y, z). The direction perpendicular to the image is the z axis. The direction perpendicular to the image is the depth direction. Therefore, the value in the depth direction is the z value in the camera coordinate system. As shown in Figure 5, the two-dimensional coordinates of the image are (u, v).

The control unit 1 generates a new image seen from the camera position and orientation CC in CAD space. The image generated at this time is a depth map. The depth map holds only the z value in the depth direction.

The control unit 1 compares the depth map with the original image. Pixels in the original image have the z-value of the image coordinates of the corresponding depth map.

The control unit 1 calculates the x and y coordinates of a pixel using the following [Equation 1].

The control unit 1 converts the three-dimensional coordinates (x, y, z) in the camera coordinate system obtained above into three-dimensional coordinates in the world coordinate system. The conversion matrix between the camera coordinate system and the world coordinate system used here is calculated by SfM processing.

As described above, the control unit 1 can reconstruct the pixels in the image DI from which degradation information has been extracted in three dimensions by calculating the three-dimensional coordinates in the world coordinate system of the corresponding pixels in the image.

The above formula (1) assumes a perspective projection model and does not take into account the image projection shift caused by lens distortion. Therefore, 360° images, which are significantly affected by lens distortion, are outside the scope of application of this embodiment.

Returning to Figure 3, in step S7, the control unit 1 performs structural calculations of the object using the degraded CAD model DC.

As described above, according to this embodiment, the control unit 1 of the information processing device D converts the CAD model CM of the object into a CAD model point cloud CP, acquires multiple images of the object, and applies a 3D model creation algorithm to the multiple images to acquire the point cloud and information SC indicating the position and orientation of the camera that captured the images. The control unit 1 also aligns the CAD model point cloud CP with the acquired point cloud to acquire a transformation matrix from the acquired point cloud to the CAD model CM, and converts the information SC indicating the position and orientation of the camera into information CC indicating the position and orientation of the camera in the CAD space using the transformation matrix. The control unit 1 extracts degradation information of the object from the multiple images, and acquires a degraded CAD model by adding degradation information to the CAD model CM using the information CC indicating the position and orientation of the camera in the CAD space. With this configuration, the information processing device D can properly map the captured image of the object to the CAD model, and can properly reflect the degradation position of the object in the CAD model. For example, the information processing device D can create 3D inspection data from an inspection image (excluding 360° images) of a structure and a CAD model CM. As an example, when the 3D inspection data is obtained, a structural calculation can be performed on the 3D data including deterioration information, and the structural strength can be examined, making it possible to perform a more detailed inspection and diagnosis of the structure.

According to this embodiment, the estimation of the position and orientation of the camera when capturing an image in CAD space is performed using the ICP algorithm. This configuration improves the accuracy of the estimation of the position and orientation of the camera.

Furthermore, according to this embodiment, the 3D model creation algorithm includes SfM processing. This configuration improves the accuracy of creating the 3D model.

According to this embodiment, degradation information is extracted from the image using a DNN. This configuration improves the accuracy of extracting degradation information.

According to this embodiment, the control unit 1 assigns degradation information to the CAD model CM by calculating the three-dimensional position of the pixel in the image from which the degradation information was extracted, based on the information CC indicating the position and orientation of the camera in the CAD space. This configuration enables the information processing device D to improve the accuracy of assigning degradation information.

Furthermore, according to this embodiment, the control unit 1 further uses the acquired deteriorated CAD model DC to determine the structural strength of the object. With this configuration, the information processing device D can use the inspection image of the structure to acquire 3D data including deterioration information of the structure and determine the structural strength of the structure. This makes it possible to inspect the structure in more detail.

<Program>
A computer 101 capable of executing program instructions can be used to function as the above-mentioned information processing device D. Fig. 6 is a block diagram showing a schematic configuration of the computer 101 functioning as the information processing device D. Here, the computer 101 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, or the like. The program instructions may be program code, code segments, or the like for performing necessary tasks.

As shown in FIG. 6, the computer 101 includes a processor 110, a ROM (Read Only Memory) 120, a RAM (Random Access Memory) 130, a storage 140, an input section 150, an output section 160, and a communication interface (I/F) 170. Each component is communicatively connected to each other via a bus 180. The processor 110 is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types.

The processor 110 controls each component and executes various arithmetic processing. That is, the processor 110 reads a program from the ROM 120 or the storage 140, and executes the program using the RAM 130 as a working area. The processor 110 controls each component and executes various arithmetic processing according to the program stored in the ROM 120 or the storage 140. In this embodiment, the program related to the present disclosure is stored in the ROM 120 or the storage 140.

The program may be stored in a storage medium readable by computer 101. Using such a storage medium, the program can be installed in computer 101. Here, the storage medium in which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be downloaded from an external device via a network.

ROM 120 stores various programs and various data. RAM 130 temporarily stores programs or data as a working area. Storage 140 is composed of a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and various data including the operating system.

The input unit 150 includes one or more input interfaces that receive input operations from a user and acquire information based on the user's operations. For example, the input unit 150 is a pointing device, a keyboard, a mouse, etc., but is not limited to these.

The output unit 160 includes one or more output interfaces that output information. For example, the output unit 160 is a display that outputs information as a video, or a speaker that outputs information as an audio, but is not limited to these. Note that if the output unit 160 is a touch panel type display, it also functions as the input unit 150.

The communication interface 170 is an interface for communicating with external devices.

All publications, patent applications, and technical standards described in this specification are incorporated by reference into this specification to the same extent as if each individual publication, patent application, and technical standard was specifically and individually indicated to be incorporated by reference.

Although the above-described embodiments have been described as representative examples, it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present disclosure. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications or changes are possible without departing from the scope of the claims. For example, it is possible to combine multiple configuration blocks shown in the configuration diagram of the embodiment into one, or to divide one configuration block.

D Information processing device 1 Control unit 2 Storage unit 3 Input/output unit 101 Computer 110 Processor 120 ROM
130 RAM
140 Storage 150 Input section 160 Output section 170 Communication interface 180 Bus

Claims (8)

An information processing device including a control unit,
Obtaining a CAD model of an object;
converting the CAD model into a CAD model point cloud;
acquiring a plurality of images of the object;
applying a 3D modeling algorithm to the images to obtain a point cloud and information indicative of the position and orientation of a camera that captured the images;
performing an alignment between the CAD model point cloud and the acquired point cloud to obtain a transformation matrix from the acquired point cloud to the CAD model point cloud;
converting the information indicating the position and orientation of the camera into information indicating the position and orientation of the camera in a CAD space using the transformation matrix;
extracting degradation information of the object from the plurality of images;
applying the degradation information to the CAD model using information indicating a position and orientation of a camera in the CAD space to obtain a degraded CAD model;
An information processing device that executes the above. 2. The information processing device according to claim 1,
The position and orientation of the camera when capturing an image in the CAD space is estimated using an Iterative Closest Point (ICP) algorithm. 3. The information processing device according to claim 1,
The information processing device, wherein the 3D model creation algorithm includes SfM (Structure from Motion) processing. 4. The information processing device according to claim 1,
An information processing device, wherein the degradation information is extracted from the image using a deep neural network (DNN). 5. The information processing device according to claim 1,
The control unit of the information processing device adds the degradation information to the CAD model by calculating the three-dimensional position of the pixel in the image from which the degradation information was extracted, based on information indicating the position and orientation of a camera in the CAD space. 6. The information processing device according to claim 5,
The control unit further determines a structural strength of the object using the acquired deteriorated CAD model. An information processing method by an information processing device,
obtaining a CAD model of the object;
converting the CAD model into a CAD model point cloud;
acquiring a plurality of images of the object;
applying a 3D modeling algorithm to the images to obtain a point cloud and information indicative of the position and orientation of a camera that captured the images;
performing a registration between the CAD model point cloud and the acquired point cloud to obtain a transformation matrix from the acquired point cloud to the CAD model point cloud;
converting the information indicating the position and orientation of the camera into information indicating the position and orientation of the camera in a CAD space using the transformation matrix;
extracting degradation information of the object from the plurality of images;
applying the degradation information to the CAD model using information indicating a position and orientation of a camera in the CAD space to obtain a degraded CAD model;
An information processing method comprising: A program for causing a computer to function as an information processing device according to any one of claims 1 to 6.

Images