JP7836005B2 - Learning device, three-dimensional reconstruction device, learning method, three-dimensional reconstruction method, and program - Google Patents

Description

This disclosure relates to a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program.

Conventionally, it is known that a deep learning estimation model (layout estimation network) is used to output features extracted from panoramic images obtained by photographing the interior space of a structure, and these features are then used to reconstruct the interior space of the structure in three dimensions (Non-Patent Documents 1 and 2).

Furthermore, it is known that by inputting not only image features but also shape features, various objects (such as cars, bicycles, airplanes, and chairs in the example in Non-Patent Document 3) can be reconstructed in 3D with high accuracy (Non-Patent Documents 3 and 4).

Sun, C., et al., “HorizonNet: Learning Room Layout With 1D Representation and Pano Stretch Data Augmentation”, CVPR, 2019. Yang, ST., et al., “DuLa-Net: A Dual-Projection Network for Estimating Room Layouts from a Single RGB Panorama”, CVPR, 2019. Yang, X., et al. Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects, BMVC, 2019. Li, Y., et al. DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection, CVPR, 2022.

However, the estimation model described above reconstructs the three-dimensional layout of the internal space of a structure by inputting image features extracted from panoramic images. Therefore, due to the nature of the layout estimation model, it was not possible to input information about the internal space layout from the beginning and perform a three-dimensional reconstruction. Consequently, the estimation model described above had difficulty reconstructing the internal space of a structure in three dimensions with high accuracy compared to the case where image features and shape features were input.

In view of these circumstances, the purpose of this disclosure is to provide a learning device, a three-dimensional reconstruction device, a learning method, a three-dimensional reconstruction method, and a program that can reconstruct the internal space of a structure in three dimensions with high accuracy.

The learning device relating to this disclosure is a learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from a panoramic image of the interior space of a structure, and uses a teacher model that, when the panoramic image and shape information indicating the shape of the interior space are input, outputs processed feature quantities indicating features relating to the correlation of the planes defining the interior space, decodes the processed feature quantities and outputs a feature quantity vector indicating the position of the corners of the planes defining the interior space in the panoramic image, and when image features extracted from the panoramic image and shape features extracted from the shape information are input, the teacher model estimation unit outputs the processed feature quantities, and the panoramic image The system includes a student model estimation unit that outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space when an image is input, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, and outputs the processed feature quantities when image feature quantities extracted from the panoramic image are input, and a student model learning unit that calculates a loss showing the difference between the distribution of processed feature quantities output using the teacher model and the distribution of processed feature quantities output using the student model, and learns the student model as the estimation model so that the loss takes the minimum value with verification data.

The three-dimensional reconstruction apparatus according to this disclosure is a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of a structure, and comprises: an image input unit that receives the panoramic image as input; an estimation unit that inputs the panoramic image to an estimation model learned by the learning device described above and outputs a feature vector indicating the position of the corner of the plane defining the interior space; a three-dimensional coordinate calculation unit that calculates the three-dimensional coordinates of the points constituting the corner based on the feature vector; and a post-processing unit that reconstructs the interior space in three dimensions based on the three-dimensional coordinates.

Furthermore, the learning method relating to this disclosure is a learning method executed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from a panoramic image of the interior space of the structure, wherein when the panoramic image and shape information indicating the shape of the interior space are input, processed feature quantities indicating features relating to the correlation of the planes defining the interior space are output, and the processed feature quantities are decoded to output a feature quantity vector indicating the position of the corner portion of the planes defining the interior space in the panoramic image, and the learning method relating to this disclosure is executed by a training model, wherein when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output. The method includes the steps of: using a student model that, when a panoramic image is input, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, and when image features extracted from the panoramic image are input, outputs the processed feature quantities; and calculating a loss that shows the difference between the distribution of processed feature quantities output using the teacher model and the distribution of processed feature quantities output using the student model, and training the student model as the estimation model so that the loss takes the minimum value on the verification data.

Furthermore, the three-dimensional reconstruction method according to this disclosure is a three-dimensional reconstruction method performed by a three-dimensional reconstruction device that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of a structure, and includes the steps of: receiving the panoramic image as input; inputting the panoramic image to an estimation model learned by the learning method described above and causing it to output a feature vector indicating the position of the corner of the plane defining the interior space; calculating the three-dimensional coordinates of the points constituting the corner based on the feature vector; and reconstructing the interior space in three dimensions based on the three-dimensional coordinates.

Furthermore, the program relating to this disclosure causes the computer to operate as the learning device described above.

Furthermore, the program relating to this disclosure causes the computer to operate as the three-dimensional reconstruction device described above.

According to this disclosure, the learning device, the three-dimensional reconstruction device, the learning method, the three-dimensional reconstruction method, and the program can reconstruct the internal space of a structure in three dimensions with high accuracy.

This figure shows an example of the configuration of a learning device according to one embodiment of the present disclosure. This figure shows an example of a panoramic image. This figure shows an example of a feature vector. This figure shows an example of the configuration of a three-dimensional reconstruction apparatus according to one embodiment of the present disclosure. This figure shows a three-dimensional image of a structure, constructed using the three-dimensional reconstruction device shown in Figure 3. Figure 1 is a flowchart illustrating an example of the operation of the learning device. Figure 3 is a flowchart illustrating an example of the operation of the three-dimensional reconstruction apparatus. This is a hardware block diagram of the learning device.

The following description of this embodiment will be made with reference to the drawings as appropriate. In each drawing, the same or corresponding parts are denoted by the same reference numerals. In the description of this embodiment, the description of the same or corresponding parts will be omitted or simplified as appropriate. The embodiments described below are examples of the configuration of the present disclosure, and the present invention is not limited to the following embodiments.

<Configuration of the learning device>
The learning device 100 learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images taken of the interior space. As shown in Figure 1, the learning device 100 comprises an image input unit 11 for the teacher model, a shape information input unit 12, a teacher model estimation unit 13, an image input unit 14 for the student model, a student model estimation unit 15, and a student model learning unit 16. The learning device 100 is a computer such as a server belonging to a cloud computing system or other computing system.

The teacher model image input unit 11, the shape information input unit 12, and the student model image input unit 14 are configured by an input interface. The input interface may be, for example, a physical key, a capacitive key, a pointing device, a touchscreen integrated with a display, or a microphone. The input interface may also include a communication interface. For example, standards such as Ethernet®, FDDI (Fiber Distributed Data Interface), and Wi-Fi® may be used for the communication interface. The teacher model estimation unit 13, the student model estimation unit 15, and the student model learning unit 16 are configured by a controller. The controller may be configured by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or by a processor, or by including both.

The teacher model image input unit 11 receives input of a panoramic image D1 taken of the interior space of a structure.

Figure 2A shows an example of a panoramic image. In this embodiment, the input to the learning device 100 is a single panoramic image with dimensions of 3 × 512 × 1024 (channels × height × width). As shown in Figure 2A, the panoramic image D1 is an image obtained by photographing the interior space of a structure with a camera or the like at angles from -90° to +90° in the vertical direction and from 0° to 360° in the horizontal direction. Therefore, the panoramic image D1 shows surfaces that define the interior space of the structure (e.g., ceiling, walls, floor).

The teacher model image input unit 11 may receive the panoramic image D1 by any method. For example, the teacher model image input unit 11 may receive the panoramic image D1 output from an external device equipped with a camera. The teacher model image input unit 11 may also receive the panoramic image D1 by reading it from memory.

The shape information input unit 12 receives shape information D2, which indicates the shape of the internal space of the structure. For example, shape information D2 may be a drawing representing the internal shape of the structure. The drawing may include a plan view, a side view, etc.

The teacher model estimation unit 13 outputs processed features when it receives image features extracted from the panoramic image D1 and shape features extracted from shape information D2 using the teacher model. The teacher model is a model that, when it receives the panoramic image D1 and shape information D2 which shows the shape of the internal space, outputs processed features that show features related to the correlation of the planes defining the internal space, decodes these processed features, and outputs a feature vector that shows the position of the corner of the plane that defines the internal space in the panoramic image D1.

The feature vector is a vector indicating the position of the corners of the surfaces that define the interior space. The surfaces that define the interior space include the ceiling, walls, and floor. The corners may be the boundaries between walls, between walls and floors, and between walls and ceilings. Specifically, the feature vector is a vector indicating the position of the boundary between a wall and a ceiling, the position of the boundary between a wall and a floor, and the position of the boundary between walls for each column of the panoramic image. The position of the boundary between walls may be a position where the probability of existence of the boundary between walls is higher than a predetermined value. Alternatively, the feature vector may be a vector indicating the position of the boundary between a wall and a ceiling, the position of the boundary between a wall and a floor, and the probability of existence of the boundary between walls for each column of the panoramic image.

The training model is obtained by creating a neural network that takes a panoramic image D1 and shape information D2 as inputs and outputs a feature vector, and then training this neural network. The training model may also be trained so that the training model loss L_{_T} on the validation data is minimized. The training model loss L_{_S} is a value that represents the difference between the distribution of processed features output by the training model feature processor 133 (which will be explained in detail later), calculated using the loss function, and the distribution of the true values.

The teacher model estimation unit 13 comprises a teacher model image feature extractor 131, a shape feature extractor 132, a teacher model feature processor 133, and a teacher model feature decoder 134.

The image feature extractor 131 for the training model extracts image features from the panoramic image D1, which is received as input by the image input unit 11 for the training model. Image features are quantities that represent image characteristics such as edges and corners within structures, as shown in the panoramic image D1. The image feature extractor 131 for the training model can extract image features using any method; for example, it may use the known Horizon Net to extract image features.

The shape feature extractor 132 extracts shape features from the shape information D2, which represents the shape of the internal space of the structure, and which is received as input by the shape information input unit 12. The shape features can be, for example, three-dimensional coordinates representing the shape of the internal space of the structure. The shape feature extractor 132 can extract shape features using any method.

The training model feature processor 133, upon receiving image features extracted from the panoramic image D1 and shape features extracted from shape information, outputs processed features that show characteristics related to the correlation of surfaces defining the internal space. Specifically, the training model feature processor 133 outputs processed features upon receiving image features extracted by the training model image feature extractor 131 and shape features extracted by the shape feature extractor 132.

More specifically, the training model feature processor 133 can obtain processed features by processing image features and shape features using the methods described in References 1 to 3.
Reference 1: Yu, Z., et al. Multi-modal Factorized Bilinear Pooling with Co-attention Learning for Visual Question Answering, ICCV, 2017.
Reference 2: Yu, Z., et al. Deep Modular Co-Attention Networks for Visual Question Answering, CVPR, 2019.
Reference 3: Ben-younes, H., et al. MUTAN: Multimodal Tucker Fusion for Visual Question Answering, ICCV, 2017.

The training model feature decoder 134 outputs a feature vector indicating the position of the corner of the plane defining the internal space of the structure in the panoramic image D1 by decoding the processed features output by the training model feature processor 133. The training model feature decoder 134 may output the feature vector using, for example, Horizon Net.

The student model image input unit 14 receives a panoramic image D1 taken inside a structure. The panoramic image D1 received by the student model image input unit 14 is the same as the panoramic image D1 received by the teacher model image input unit 11.

The student model estimation unit 15 outputs processed features when image features extracted from the panoramic image D1 are input using the student model. The student model is a model that, when the panoramic image D1 is input, outputs processed features that show features related to the correlation of the planes defining the internal space, and decodes these processed features to output a feature vector that shows the position of the corners of the planes defining the internal space. The student model is obtained by training a neural network that takes the panoramic image D1 as input and outputs a feature vector. The student model may be trained so that the student model loss L _S takes the minimum value on the validation data. The student model loss L _S is the difference between the distribution of processed features output by the student model feature processor 152 (which will be explained in detail later) and the distribution of true values, calculated using the loss function.

The student model estimation unit 15 comprises a student model image feature extractor 151, a student model feature processor 152, and a student model feature decoder 153.

The student model image feature extractor 151 extracts image features from the panoramic image D1, which is received as input by the student model image input unit 14.

The student model feature processor 152 outputs processed features that show features related to the correlation of planes defining the internal space when image features extracted from the panoramic image D1 are input to the student model. Specifically, the student model feature processor 152 outputs processed features when image features extracted by the student model image feature extractor 151 are input to the student model. Specifically, the student model feature processor 152 can output processed features by processing the image features output by the student model feature processor 152 using, for example, the Transformer described in Reference 4.
Reference 4: Ashish, V., et al. Attention Is All You Need, CL, 2017.

The student model feature decoder 153 estimates a feature vector indicating the position of the corner of the plane defining the interior space of the structure by decoding the processed features output by the student model feature processor 152. Similar to the teacher model feature decoder 134, the student model feature decoder 153 may output a feature vector indicating the position of the corner of the interior space in the panoramic image D1 based on the processed features, for example, using Horizon Net.

The student model learning unit 16 learns the student model described above. At this time, the student model learning unit 16 sets a loss function based on the distribution of processed features output using the teacher model by the teacher model feature processor 133 and the distribution of processed features output using the student model by the student model feature processor 152. Specifically, the student model learning unit 16 calculates a loss L feature that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model. Then, the student model learning unit 16 learns the student model as an estimation model so that _{the loss L feature on} the validation data takes its minimum value.

The size of the feature vector output by the estimation model trained by the learning device 100 is 3 × 1 × 10²⁴. As described above, in a configuration where the feature vector indicates the probability of existence of a boundary between walls, the feature vector indicates, for each column of the panoramic image, the boundary position between the ceiling and walls (y _c ), the boundary position between the floor and walls (y _f ), and the probability of existence of a boundary between walls (i.e., a corner) (y _w ). y _w may be a binary vector with labels of 0 or 1, but since the number of 1s becomes sparse (for example, 4 out of 10²⁴), y _w (i) = c ^dx may also be used. Here, i is the i-th column, dx is the distance from the i-th column to the nearest column where a boundary between walls exists, and c is a constant (for example, c = 0.96).

Figure 2B shows an example of a feature vector output when the panoramic image shown in Figure 2A is input to the learning device 100. To make the correspondence between the panoramic image and the feature vector easier to understand, the positions indicated by y _c and y _f are superimposed on the panoramic image. In addition, y _w (i) is shown above the panoramic image. In Figure 2B, y _w (i) is stretched in the height direction for easier viewing.

<Configuration of the three-dimensional reconstruction device>
The three-dimensional reconstruction device 300 reconstructs the interior space of a structure in three dimensions from a panoramic image D1 taken of the interior space. The three-dimensional reconstruction device 300 is a computer such as a server belonging to a cloud computing system or other computing system.

As shown in Figure 3, the three-dimensional reconstruction device 300 comprises a student model storage unit 31, an image input unit 32, an estimation unit 33, a three-dimensional coordinate calculation unit 34, and a post-processing unit 35.

The student model storage unit 31 is composed of memory. The memory may be an HDD (Hard Disk Drive), SSD (Solid State Drive), EEPROM (Electrically Erasable Programmable Read-Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The image input unit 32 is composed of an input interface. The estimation unit 33 and the post-processing unit 35 are composed of a controller. The controller may be composed of dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field-Programmable Gate Array), or it may be composed of a processor, or it may include both.

The student model memory unit 31 stores the student model (estimated model) learned by the learning device 100.

The image input unit 32 receives a panoramic image D1, which is a photograph of the interior space of the structure.

The estimation unit 33 inputs the panoramic image D1 to the student model trained by the learning device 100 described above, and estimates the feature vector by outputting a feature vector indicating the position of the corner of the interior space in the panoramic image D1.

The three-dimensional coordinate calculation unit 34 calculates the three-dimensional coordinates of the points constituting the corner based on the feature vector estimated by the estimation unit 33.

The post-processing unit 35 reconstructs the internal space in three dimensions based on the three-dimensional coordinates calculated by the three-dimensional coordinate calculation unit 34. Specifically, the post-processing unit 35 generates a three-dimensional image D3, as shown in Figure 4, based on the three-dimensional coordinates. The post-processing unit 35 also outputs the three-dimensional image D3 to a display device or the like.

<Operation of the learning device>
Here, the operation of the learning device 100 according to this embodiment will be described with reference to Figure 5. Figure 5 is a flowchart showing an example of the operation of the learning device 100 according to this embodiment. The operation of the learning device 100 described with reference to Figure 5 corresponds to an example of the method performed by the learning device 100 according to this embodiment, which learns an estimation model for three-dimensional reconstruction of the interior space from a panoramic image D1 taken of the interior space of a structure.

First, in steps S11 to S15, the teacher model estimation unit 13 uses the teacher model to output processed features when it receives image features extracted from the panoramic image D1 and shape features extracted from the shape information D2. As described above, when the panoramic image D1 and shape information D2 indicating the shape of the internal space are input, the model outputs processed features indicating the features related to the correlation of the planes defining the internal space, and decodes these processed features to output a feature vector indicating the position of the corners of the planes defining the internal space.

Specifically, in step S11, the image input unit 11 for the training model receives input of a panoramic image D1 taken of the interior space of the structure.

In step S12, the image feature extractor 131 for the training model extracts image features from the panoramic image D1.

In step S13, the shape information input unit 12 receives shape information D2, which indicates the shape of the internal space of the structure.

In step S14, the shape feature extractor 132 extracts shape features from the shape information D2.

In step S15, the teacher model estimation unit 13 receives image features extracted from the panoramic image D1 and shape features extracted from the shape information D2 as input and outputs processed feature quantities that show features related to the correlation of the planes defining the internal space. Specifically, the teacher model feature processor 133 inputs the image features extracted in step 12 and the shape features extracted in step S14 into the teacher model and outputs processed feature quantities.

Next, in steps S16 to S18, the student model estimation unit 15 outputs processed features when image features extracted from the panoramic image D1 are input using the student model. As described above, when the panoramic image D1 is input, the model outputs processed features that show features related to the correlation of the planes defining the internal space, and decodes these processed features to output a feature vector that shows the position of the corners of the planes defining the internal space in the panoramic image D1.

Specifically, in step S16, the student model image input unit 14 receives input of a panoramic image D1 taken inside the structure.

In step S17, the student model image feature extractor 151 extracts image features from the panoramic image D1.

In step S18, when the student model estimation unit 15 receives image features extracted from the panoramic image D1 as input, it outputs processed feature quantities that show features related to the correlation of the planes defining the internal space. Specifically, the student model feature processor 152 inputs the image features extracted in step S17 into the student model and outputs processed feature quantities.

Next, in steps S19 and S20, the student model learning unit 16 learns the student model.

Specifically, in step S19, the student model learning unit 16 calculates a loss L feature, which is the difference between the distribution of processed features output using the teacher model in step S15 and the distribution of processed _features output using the student model in step S18.

In step S20, the student model learning unit 16 learns the student model so that the loss L _feature takes its minimum value on the validation data.

In the above description, the learning device 100 executed steps S16 to S18 after executing steps S11 to S15, but it is not limited to this. For example, the learning device 100 may execute steps S11 to S15 after executing steps S16 to S18. Alternatively, the learning device 100 may execute steps S16 to S18 at the same time as executing steps S11 to S15.

Furthermore, while the learning device 100 performed steps S13 and S14 after performing steps S11 and S12, it is not limited to this. For example, the learning device 100 may perform steps S11 and S12 after performing steps S13 and S14. Alternatively, the learning device 100 may perform steps S13 and S14 at the same time as performing steps S11 and S12.

<Operation of the three-dimensional reconstruction device>
Here, the operation of the three-dimensional reconstruction apparatus 300 according to this embodiment will be described with reference to Figure 6. Figure 6 is a flowchart showing an example of the operation of the three-dimensional reconstruction apparatus 300 according to this embodiment. The operation of the three-dimensional reconstruction apparatus 300 described with reference to Figure 6 corresponds to an example of the method performed by the three-dimensional reconstruction apparatus 300 for three-dimensional reconstruction of the interior space from a panoramic image D1 taken of the interior space of a structure, according to this embodiment.

In step S31, the image input unit 32 receives an input of a panoramic image D1 taken inside the structure.

In step S32, the estimation unit 33 inputs the panoramic image D1 received in step 31 into the student model trained by the learning device 100 described above, and estimates the feature vector by outputting a feature vector indicating the position of the corner of the interior space in the panoramic image D1.

In step S33, the three-dimensional coordinate calculation unit 34 calculates the three-dimensional coordinates of the points constituting the corner portion based on the feature vector estimated by the estimation unit 33.

In step S34, the post-processing unit 35 reconstructs the internal space in three dimensions based on the three-dimensional coordinates estimated in step S34.

As described above, the learning device 100 in this embodiment is a learning device 100 that learns an estimation model for three-dimensional reconstruction of the interior space from a panoramic image D1 of the interior space of a structure, and when the panoramic image D1 and shape information D2 indicating the shape of the interior space are input, it outputs processed feature quantities that show features related to the correlation of the planes defining the interior space, and decodes the processed feature quantities to output a feature quantity vector that shows the position of the corners of the planes defining the interior space in the panoramic image, using a teacher model, and when the image features extracted from the panoramic image D1 and the shape features extracted from the shape information D2 are input The system includes: a teacher model estimation unit 13 that outputs processed features when a panoramic image D1 is input, a student model estimation unit 15 that outputs processed features when image features extracted from the panoramic image D1 are input, using a student model that outputs processed features showing features related to the correlation of planes defining the internal space when a panoramic image D1 is input, decodes the processed features and outputs a feature vector showing the position of the corner of the planes defining the internal space in the panoramic image D1, and calculates a loss L _feature that shows the difference between the distribution of processed features output by the teacher model estimation unit 13 and the distribution of processed features output by the student model estimation unit 15, and learns the student model as an estimation model so that the loss on the verification data is minimized.

As a result, the three-dimensional reconstruction device 300 can use the estimation model learned by the learning device 100 to estimate feature vectors with the same high accuracy as when shape features are used, without using shape features. Therefore, the three-dimensional reconstruction device 300 can calculate the three-dimensional coordinates of points constituting the corners of the internal space of a structure with high accuracy, and can reconstruct the internal space in three dimensions with high accuracy. Consequently, for example, by reflecting the deterioration of surfaces defining the internal space of a structure, detected by workers through inspections, etc., in the three-dimensional image D3, the modeling accuracy of the structure showing deteriorated areas is improved. Furthermore, by reflecting furniture placed in the internal space of a structure, wallpaper applied to surfaces defining the internal space, etc., in the three-dimensional image D3, the modeling accuracy of the structure is improved. In addition, because the three-dimensional reconstruction device 300 does not use shape features, it can estimate three-dimensional coordinates at high speed.

Furthermore, the three-dimensional reconstruction device 300 in this embodiment is a three-dimensional reconstruction device 300 that reconstructs the interior space of a structure in three dimensions from a panoramic image D1 taken of the interior space of the structure, and comprises an image input unit 32 that receives the panoramic image D1 as input, an estimation unit 33 that inputs the panoramic image D1 to an estimation model learned by the learning device 100 described above and outputs a feature vector indicating the position of the corners of the planes defining the interior space, a three-dimensional coordinate calculation unit 34 that calculates the three-dimensional coordinates of the points constituting the corners based on the feature vector, and a post-processing unit 35 that reconstructs the interior space in three dimensions based on the three-dimensional coordinates. As a result, the three-dimensional reconstruction device 300 can calculate the three-dimensional coordinates of the planes defining the interior space with high accuracy without using shape features. Therefore, the three-dimensional reconstruction device 300 can reconstruct the interior space in three dimensions with high accuracy based on the three-dimensional coordinates. Consequently, for example, by reflecting the deterioration of surfaces defining the internal space of a structure, detected by workers through inspections, etc., into the three-dimensional image D3, the modeling accuracy of the structure showing deteriorated areas is improved. Furthermore, by reflecting furniture placed in the internal space of the structure, wallpaper applied to surfaces defining the internal space, etc., into the three-dimensional image D3, the modeling accuracy of the structure is improved. In addition, since the three-dimensional reconstruction device 300 does not use shape features, it can reconstruct the internal space in three dimensions at high speed.

<Program>
The learning device 100 and the three-dimensional reconstruction device 300 described above can be implemented using a computer. Furthermore, programs for functioning as the learning device 100 and the three-dimensional reconstruction device 300 may be provided. These programs may be stored on a storage medium or provided via a network. Figure 7 is a block diagram illustrating the schematic configuration of a computer 401 functioning as the learning device 100. The computer functioning as the three-dimensional reconstruction device 300 is similar. Here, the computer 401 may be a general-purpose computer, a dedicated computer, a workstation, a PC (Personal Computer), an electronic notepad, etc. Program instructions may be program code, code segments, etc., for executing the necessary tasks.

As shown in Figure 7, the computer 401 comprises a processor 410, a ROM (Read Only Memory) 420, a RAM (Random Access Memory) 430, storage 440, an input unit 450, an output unit 460, and a communication interface (I/F) 470. Each component is connected to the others via a bus 480 so as to be able to communicate with each other. The processor 410 is specifically a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), DSP (Digital Signal Processor), SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types.

The processor 410 controls each configuration and performs various arithmetic operations. Specifically, the processor 410 reads a program from the ROM 420 or storage 440 and executes the program using the RAM 430 as a working area. The processor 410 controls each configuration and performs various arithmetic operations according to the program stored in the ROM 420 or storage 440. In the above-described embodiment, the program according to this disclosure is stored in the ROM 420 or storage 440.

The program may be stored on a storage medium readable by computer 401. Using such a storage medium, the program can be installed on computer 401. Here, the storage medium on which the program is stored may be a non-transitory storage medium. The non-transitory storage medium is not particularly limited, but may include, for example, a CD-ROM, DVD-ROM, or USB (Universal Serial Bus) memory. Alternatively, the program may be downloaded from an external device via a network.

ROM 420 stores various programs and data. RAM 430 temporarily stores programs or data as a working area. Storage 440 consists of an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs and data, including the operating system.

The input unit 450 includes one or more input interfaces that receive user input operations and acquire information based on the user operations. For example, the input unit 450 is a pointing device, keyboard, mouse, etc., but is not limited to these.

The output unit 460 includes one or more output interfaces for outputting information. For example, the output unit 460 is a display that outputs information as video, or a speaker that outputs information as audio, but is not limited to these. Furthermore, if the output unit 460 is a touch panel display, it also functions as an input unit 450.

The communication interface 470 is an interface for communicating with external devices.

The following additional information is disclosed regarding the embodiments described above.
[Additional note 1]
A learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure,
The controller is equipped with a controller,
Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output.
Using a student model that, upon inputting the panoramic image, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and then, upon inputting image features extracted from the panoramic image, outputs the processed feature quantities,
A learning device that calculates a loss representing the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and trains the student model as the estimation model so that the loss takes its minimum value on the validation data.
[Additional note 2]
A three-dimensional reconstruction device for three-dimensionally reconstructing the interior space of a structure from a panoramic image of the interior space captured,
The system includes an input interface for receiving the aforementioned panoramic image and a controller.
The aforementioned controller,
The panoramic image is input to the estimation model trained by the learning device described in Appendix 1, and a feature vector indicating the position of the corner of the plane defining the interior space is output.
Based on the aforementioned feature vector, the three-dimensional coordinates of the points constituting the corner portion are calculated.
A three-dimensional reconstruction device that reconstructs the internal space in three dimensions based on the aforementioned three-dimensional coordinates.
[Additional note 3]
A learning method performed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure,
Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, when image features extracted from the panoramic image and shape features extracted from the shape information are input, the processed feature quantities are output.
Using a student model that, upon inputting the panoramic image, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and then, upon inputting image features extracted from the panoramic image, outputs the processed feature quantities,
A learning method comprising: calculating a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model; and training the student model as the estimation model so that the loss takes its minimum value on the validation data.
[Additional note 4]
A three-dimensional reconstruction method performed by a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of the structure,
The panoramic image is received as input.
The estimation model trained by the learning method described in Appendix 3 is input with the panoramic image and outputs a feature vector indicating the position of the corner of the plane defining the interior space.
Based on the aforementioned feature vector, the three-dimensional coordinates of the points constituting the corner portion are calculated.
A three-dimensional reconstruction method for three-dimensionally reconstructing the internal space based on the three-dimensional coordinates.
[Additional note 5]
A non-temporary computer-readable medium containing a program for operating a computer as a learning device as described in Appendix 1.
[Additional note 6]
A non-temporary computer-readable medium storing a program for operating a computer as a three-dimensional reconstruction device as described in Appendix 2.

All documents, patent applications, and technologies described herein are incorporated by reference to the same extent as if each individual document, patent application, and technology were specifically and individually described as being incorporated by reference.

Although the embodiments described above are representative examples, it will be apparent to those skilled in the art that many modifications and substitutions are possible within the spirit and scope of this disclosure. Therefore, the present invention should not be construed as being limited by the embodiments described above, and various modifications or changes are possible without departing from the claims.

11 Image input unit for teacher model 12 Shape information input unit 13 Teacher model estimation unit 14 Image input unit for student model 15 Student model estimation unit 16 Student model learning unit 31 Student model storage unit 32 Image input unit 33 Estimation unit 34 Three-dimensional coordinate calculation unit 35 Post-processing unit 100 Learning device 131 Image feature extractor for teacher model 132 Shape feature extractor 133 Feature processor for teacher model 134 Feature decoder for teacher model 151 Image feature extractor for student model 152 Feature processor for student model 153 Feature decoder for student model 300 Three-dimensional reconstruction device 401 Computer 410 Processor 420 ROM
430 RAM
440 Storage 450 Input section 460 Output section 470 Communication interface 480 Bus

Claims (6)

A learning device for learning an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure,
A teacher model estimation unit that, upon inputting the panoramic image and shape information indicating the shape of the internal space, outputs processed feature quantities showing features relating to the correlation of the planes defining the internal space, decodes the processed feature quantities, and outputs feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, and upon inputting image features extracted from the panoramic image and shape features extracted from the shape information, outputs the processed feature quantities.
A student model estimation unit that, when an image feature extracted from the panoramic image is input, outputs a processed feature quantity showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantity and outputs a feature quantity vector showing the position of the corner of the planes defining the internal space in the panoramic image, and when an image feature quantity extracted from the panoramic image is input, outputs the processed feature quantity,
A student model learning unit calculates a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and trains the student model as the estimation model so that the loss takes the minimum value on the validation data.
A learning device equipped with the following features. A three-dimensional reconstruction device for three-dimensionally reconstructing the interior space of a structure from a panoramic image of the interior space captured,
An image input unit that receives the aforementioned panoramic image input,
An estimation unit that inputs the panoramic image to an estimation model learned by the learning device described in claim 1 and outputs a feature vector indicating the position of the corner of the plane defining the internal space,
A three-dimensional coordinate calculation unit calculates the three-dimensional coordinates of the points constituting the corner portion based on the aforementioned feature vector,
A post-processing unit that reconstructs the internal space in three dimensions based on the three-dimensional coordinates,
A three-dimensional reconstruction device equipped with the following features. A learning method performed by a learning device that learns an estimation model for three-dimensional reconstruction of the interior space of a structure from panoramic images of the interior space of the structure,
Using a training model that, when the panoramic image and shape information indicating the shape of the internal space are input, outputs processed feature quantities showing features relating to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors indicating the positions of the corners of the planes defining the internal space in the panoramic image, the step of outputting the processed feature quantities when image features extracted from the panoramic image and shape features extracted from the shape information are input,
Using a student model that, when the panoramic image is input, outputs processed feature quantities showing characteristics related to the correlation of the planes defining the internal space, decodes the processed feature quantities to output feature vectors showing the positions of the corners of the planes defining the internal space in the panoramic image, the student model outputs the processed feature quantities when image features extracted from the panoramic image are input.
The steps include: calculating a loss that represents the difference between the distribution of processed features output using the teacher model and the distribution of processed features output using the student model, and training the student model as the estimation model so that the loss takes its minimum value on the validation data;
Learning methods that include this. A three-dimensional reconstruction method performed by a three-dimensional reconstruction apparatus that reconstructs the interior space of a structure in three dimensions from a panoramic image of the interior space of the structure,
The step of receiving the aforementioned panoramic image as input,
The steps include inputting the panoramic image into an estimation model trained by the learning method described in claim 3 and outputting a feature vector indicating the position of the corner of the plane defining the interior space,
The steps include: calculating the three-dimensional coordinates of the points constituting the corner portion based on the feature vector;
The steps include: three-dimensional reconstruction of the internal space based on the three-dimensional coordinates;
A three-dimensional reconstruction method, including... A program for operating a computer as a learning device according to claim 1. A program for operating a computer as a three-dimensional reconstruction apparatus as described in claim 2.