JP7540671B2 - Three-dimensional model generating device and three-dimensional model generating program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. Date held: November 30, 2021 Meeting name and website address: Spacely Connect 2021 ~DX Towards solving the problem of labor shortages~ (held online) https://info.spacely.co.jp/5th_anniversary/

Article 30, paragraph 2 of the Patent Act applies. Date held: November 30, 2021 Meeting name and website address: Spacely Connect 2021 ~Towards resolving the issue of DX talent shortages~ (held online) https://info.spacely.co.jp/5th_anniversary/

This disclosure relates to a three-dimensional model generation device and a three-dimensional model generation program.

There is a known technology for creating a three-dimensional model of a real estate property to allow a third party to learn about the interior information of the property. When a real estate property is created as a three-dimensional model using such technology, it is possible to allow a third party to learn detailed information about the interior of the property by displaying the three-dimensional model on a normal display or on VR goggles using VR (Virtual Reality) technology. For example, Patent Document 1 discloses a technology for generating a desired three-dimensional model by analyzing a floor plan of a real estate property and identifying room areas, etc.

JP 2018-81548 A

The above-mentioned technology generates a three-dimensional model by simply identifying room areas, etc., based on a floor plan. However, the indoor conditions of individual real estate properties vary widely, and there is a need to allow third parties to obtain detailed information that is not shown on the floor plan.

The three-dimensional model generation device and three-dimensional model generation program disclosed herein aim to make it possible to generate three-dimensional models of real estate properties that include more detailed information.

A three-dimensional model generating device (1) according to one aspect of the present disclosure includes an indoor image input receiving unit (10) that receives input of an indoor image (RG) captured inside a real estate property, a segmentation unit (12) that performs segmentation on the indoor image (RG) received as input by the indoor image input receiving unit (10), an indoor structure estimation unit (13) that estimates an indoor structure, which is a structure inside the room, based on the result of the segmentation performed on the indoor image (RG) by the segmentation unit (12), and a three-dimensional model generating unit (20) that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimation unit (13).

A three-dimensional model generation program (P) according to one aspect of the present disclosure causes a computer (C) to function as an indoor image input receiving unit (10) that receives input of an indoor image (RG) captured inside a real estate property, a segmentation unit (12) that performs segmentation on the indoor image (RG) received as input by the indoor image input receiving unit (10), an indoor structure estimation unit (13) that estimates the indoor structure, which is the structure of the room, based on the result of the segmentation performed on the indoor image (RG) by the segmentation unit (12), and a three-dimensional model generation unit (20) that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimation unit (13).

According to at least one of these three-dimensional model generation device (1) and three-dimensional model generation program (P), the attributes of each part included in the indoor image (RG) captured inside the real estate property are grasped by performing segmentation on the indoor image (RG) captured inside the real estate property, and the indoor structure is estimated based on the attributes of each part that have been grasped. Then, a three-dimensional model of the real estate property is generated based on the indoor structure estimated in this way. Therefore, compared to a case where a three-dimensional model is generated based on the indoor structure simply estimated from a floor plan or the like, a three-dimensional model that reflects the indoor situation in detail is generated. Therefore, it becomes possible to generate a three-dimensional model of the real estate property that includes more detailed information.

In a three-dimensional model generating device (1) according to one aspect of the present disclosure, the segmentation unit (12) may perform at least two types of segmentation on the indoor image (RG). With this, for example, objects that are difficult to distinguish using a specific type of segmentation can be preferably distinguished by performing another type of segmentation.

In a three-dimensional model generating device (1) according to one aspect of the present disclosure, the segmentation unit (12) may perform at least semantic segmentation and instance segmentation on the indoor image (RG). This makes it possible to suitably distinguish room boundaries, which are difficult to distinguish using semantic segmentation, by performing instance segmentation.

In a three-dimensional model generation device (1) according to one aspect of the present disclosure, the indoor image (RG) includes a celestial sphere image of the room, and a top-view image correction unit (14) may be provided that performs correction on a top-view image (TG) that is an image of the room viewed from above in the celestial sphere image. This makes it possible to correct the positions of walls, corners, etc. of the estimated indoor structure, and therefore to generate a three-dimensional model with higher accuracy. Moreover, since the indoor structure can be corrected from the top-view image (TG), more intuitive operation is possible, improving convenience.

In the three-dimensional model generating device (1) according to one aspect of the present disclosure, the indoor image (RG) includes a two-dimensional image (PG) that is a part of the omnidirectional image of the room, and includes a connecting unit (11) that connects one end (PGa) of the two-dimensional image (PG) to the other end (PGb) opposite to the one end (PGa), and the segmentation unit (12) may perform segmentation on the two-dimensional image (PG) in which the one end (PGa) and the other end (PGb) are connected to each other by the connecting unit (11). The two-dimensional image (PG) included in the indoor image (RG) constitutes a part of the omnidirectional image of the room. For example, the two-dimensional image (PG) may be a panoramic image captured over 360 degrees in the horizontal direction. Such a two-dimensional image (PG) is a band-shaped image with one end (PGa) in the horizontal direction as the base end and the other end (PGb) at a position 360 degrees from the one end (PGa) as the terminal end. At this time, the same location in the room (for example, a location on a straight line extending in the vertical direction) is captured at one end (PGa) and the other end (PGb) of the strip-shaped two-dimensional image (PG). Here, an object that exists so as to overlap one end (PGa) of the two-dimensional image (PG), i.e., the other end (PGb), is displayed in two-dimensional image (PG) divided into one end (PGa) side and the other end (PGb) side. In this case, it becomes difficult for the segmentation unit (12) to appropriately perform segmentation on the object displayed in two-part into one end (PGa) side and the other end (PGb) side. As a result, it becomes difficult to correctly grasp the attributes of the part where the object is displayed on the two-dimensional image (PG), and therefore it becomes difficult to correctly estimate the indoor structure. Therefore, in the above-mentioned three-dimensional model generating device (1), by connecting one end (PGa) and the other end (PGb) of the band-shaped two-dimensional image (PG) to each other to form a ring shape, an object existing so as to overlap one end (PGa) of the two-dimensional image (PG), i.e., the other end (PGb), is no longer in a state of being divided into two, and it becomes easier to perform appropriate segmentation on the object. As a result, it becomes easier to correctly grasp the attributes of the part where the object is displayed on the two-dimensional image (PG), and thus it becomes easier to correctly estimate the indoor structure. In this way, by performing segmentation on the two-dimensional image (PG) in which one end (PGa) and the other end (PGb) are connected to each other, it becomes possible to appropriately perform segmentation on the part where the accuracy of image recognition is likely to decrease due to the cut-off of the two-dimensional image (PG). Note that, although a panoramic image captured over 360 degrees in the horizontal direction is given here as an example of a two-dimensional image (PG), the two-dimensional image (PG) does not necessarily have to be an image of this type, and may be, for example, a panoramic image captured over 360 degrees in a direction intersecting the horizontal direction.

A three-dimensional model generating device (1) according to one aspect of the present disclosure includes a floor plan image input receiving unit (15) that receives an input of a floor plan image (FG) showing the floor plan of a real estate property, a property structure estimating unit (16) that estimates a property structure, which is a structure of a real estate property, based on the floor plan image (FG) input received by the floor plan image input receiving unit (15), and a matching unit (17) that matches the indoor structure estimated by the indoor structure estimating unit (13) with the property structure estimated by the property structure estimating unit (16), and the three-dimensional model generating unit (20) may generate a three-dimensional model of the real estate property based on the result of matching between the indoor structure and the property structure performed by the matching unit (17). According to this, a three-dimensional model of the real estate property is generated based not only on the indoor structure estimated based on the indoor image (RG) but also on the property structure estimated based on the floor plan image (FG). Therefore, information about the overall structure of the real estate property, which is likely to be unclear based on the indoor structure alone, can be supplemented based on the property structure. This makes it possible to generate 3D models of real estate properties with high accuracy.

The three-dimensional model generating device (1) according to one aspect of the present disclosure includes a room type information acquisition unit (19) that acquires room type information related to the room type of the real estate property, and the three-dimensional model generating unit (20) may generate a three-dimensional model of the real estate property based on the result of matching between the interior structure and the property structure performed by the matching unit (17) and the room type information acquired by the room type information acquisition unit (19). In this way, the three-dimensional model of the real estate property is generated based also on the room type information, which is information related to the type and number of rooms that the real estate property has. Therefore, it is possible to generate a three-dimensional model of the real estate property with high accuracy.

In a three-dimensional model generating device (1) according to one aspect of the present disclosure, the indoor image input receiving unit (10) can receive input of a plurality of indoor images (RG) for a real estate property, and may include a relationship estimation unit (18) that estimates at least one of the connection relationship and the positional relationship between the plurality of indoor images (RG) based on the result of matching between the indoor structure and the property structure performed by the matching unit (17). With this, at least one of the connection relationship and the positional relationship between the plurality of indoor images (RG) of the real estate property is estimated with high accuracy based on the result of matching between the indoor structure estimated by the indoor structure estimation unit (13) and the property structure estimated by the property structure estimation unit (16). Therefore, it becomes possible to generate a three-dimensional model of the real estate property with high accuracy.

Note that the reference numerals in parentheses above are intended to indicate the reference numerals of components in the embodiments described below as an example of the present disclosure, and are not intended to limit the present disclosure to the aspects of the embodiments.

In this way, the three-dimensional model generation device and three-dimensional model generation program disclosed herein can generate three-dimensional models of real estate properties that include more detailed information.

FIG. 1 is a block diagram showing a three-dimensional model generating device according to this embodiment. FIG. 2 is a diagram showing an example of a two-dimensional image included in an indoor image of a real estate property. FIG. 3 is a diagram showing an example of a top-view image corresponding to an indoor image of a real estate property. FIG. 4 is a diagram showing a floor plan image of a real estate property. FIG. 5 is a flowchart showing the flow of the three-dimensional model generating process executed by the three-dimensional model generating device. FIG. 6 is a block diagram showing a three-dimensional model generating program.

Below, exemplary embodiments will be described with reference to the drawings. Note that the same or equivalent parts in each drawing are given the same reference numerals, and duplicate explanations will be omitted.

[Overall configuration]
Fig. 1 is a block diagram showing a three-dimensional model generating device 1 according to the present embodiment. The three-dimensional model generating device 1 shown in Fig. 1 is a device for generating a three-dimensional model of a real estate property in order to allow a third party, such as a client, to know information about the interior of the real estate property.

A "real estate property" is a building that has at least a structure such as a wall that separates the inside from the outside, and may be, for example, an apartment, condominium, detached house, office, store, or warehouse. A real estate property has one or more rooms. In the following explanation, an apartment with a room type of 1LDK is used as an example of a real estate property. "Room type" is a classification of real estate properties based on the type and number of rooms. For example, 1LDK refers to a room type that includes a living/dining room and a kitchen in addition to one living room. The room type is not particularly limited, and may be any type such as a studio, 1K, 1DK, 1LDK, 2K, 2DK, or 2LDK.

A "three-dimensional model" is a structural model that is constructed in a virtual three-dimensional space and can be viewed and operated on a computer using various applications. The three-dimensional model can be displayed on a display. For example, the three-dimensional model may be displayed on a normal display, as well as on VR goggles using virtual reality technology. Here, by using a specific application, the view from a desired viewpoint in the three-dimensional model can be displayed on the display. In addition, the viewpoint can be moved freely (or within a certain range of restrictions) as if moving within a real estate property. In particular, if the real estate property has multiple rooms, the viewpoint can be moved to the next room through an appropriate path (e.g., a door, etc.). The three-dimensional model may include individual three-dimensional models (in other words, a three-dimensional model of one space) that are generated individually for each room of the real estate property, and an overall three-dimensional model (in other words, a three-dimensional model in which each space is arranged and connected) that is generated for the entire real estate property by finally combining multiple individual three-dimensional models. The overall three-dimensional model is also called a dollhouse.

The physical configuration of the three-dimensional model generating device 1 will be described. The three-dimensional model generating device 1 is physically configured as a computer C (server) capable of communicating with a user terminal 2 via a network. The three-dimensional model generating device 1 includes a control arithmetic device, a storage device, and an input/output device. The control arithmetic device is configured by a controller such as a CPU (Central Processing Unit), and executes arithmetic processing and controls the storage device and the input/output device. The storage device includes, for example, a main storage device and an auxiliary storage device. The main storage device includes, for example, a RAM (Random Access Memory). The auxiliary storage device includes, for example, a ROM (Read Only Memory). The input/output device includes, for example, an input device that receives data from the outside and transmits it to the storage device, and an output device that outputs the results of calculations performed by the control arithmetic device and stored in the storage device to the outside. The input/output device also includes a communication device that transmits and receives information to and from external devices.

The three-dimensional model generating device 1 executes a predetermined process, for example, by loading a program stored in ROM into RAM and executing the program loaded into RAM with a CPU. Specifically, the three-dimensional model generating device 1 executes a three-dimensional model generation process described below by loading a three-dimensional model generation program P stored in ROM into RAM and executing the three-dimensional model generation program P loaded into RAM with a CPU. Note that the three-dimensional model generating device 1 may have a physical configuration different from the configuration described above.

First, the user terminal 2 will be described. The user terminal 2 is a terminal used by a user. The user terminal 2 is configured as a computer capable of communicating with the three-dimensional model generating device 1 via a network. The user terminal 2 may be, for example, a desktop computer, a notebook computer, a tablet, or a smartphone, and a desktop computer is exemplified here. The user terminal 2 includes a control and arithmetic unit, a storage device, and an input/output device. The user terminal 2 is in a usable state with applications for working with the three-dimensional model generating device 1 to create a three-dimensional model and for displaying the created three-dimensional model on a display.

Next, the functional configuration of the three-dimensional model generation device 1 will be described. Functionally, the three-dimensional model generation device 1 includes an indoor image input reception unit 10, a combination unit 11, a segmentation unit 12, an indoor structure estimation unit 13, a top-view image correction unit 14, a floor plan image input reception unit 15, a property structure estimation unit 16, a matching unit 17, a relationship estimation unit 18, a room type information acquisition unit 19, a three-dimensional model generation unit 20, and a relationship correction unit 21.

The indoor image input receiving unit 10 receives input of indoor images RG. An "indoor image" is an image captured of the interior of a real estate property. In this example, a 1LDK apartment is used as an example of a real estate property, so the indoor image input receiving unit 10 receives input of indoor images RG for each room, such as an indoor image RG of a living room, an indoor image RG of a living/dining room, and an indoor image RG of a kitchen. In other words, the indoor image input receiving unit 10 can receive input of multiple indoor images RG for a real estate property. The indoor image input receiving unit 10 receives input of indoor images RG from, for example, a user terminal 2.

The indoor image RG includes a spherical image of the room. A "spherical image" is an image including all directions that can be seen from the viewpoint. In particular, the indoor image RG includes a two-dimensional image PG. A "two-dimensional image" is a part of the spherical image of the room. FIG. 2 is a diagram showing an example of a two-dimensional image PG included in the indoor image RG of a real estate property. As shown in FIG. 2, the two-dimensional image PG is a panoramic image captured horizontally over 360 degrees. The two-dimensional image PG is a strip-shaped image with one end PGa in the horizontal direction as the base end and the other end PGb, which is 360 degrees from the one end PGa, as the terminal end. At this time, the same place in the room (a place on a straight line extending in the vertical direction) is captured at one end PGa and the other end PGb of the strip-shaped two-dimensional image PG. Here, an object that overlaps one end PGa (the other end PGb) of the two-dimensional image PG is displayed divided into two parts, one end PGa side and the other end PGb side, on the two-dimensional image PG. In FIG. 2, the walls located at both ends of the two-dimensional image PG are displayed divided into two parts, one end PGa side and the other end PGb side.

The fact that the two-dimensional image PG is a strip-shaped image cut at one end PGa and the other end PGb is due to the format of a typical omnidirectional image. A omnidirectional image is generated, for example, by combining (i.e., stitching) image pieces captured in multiple directions (up, down, left, right, etc.). Note that this stitching process can be automated by using a specific application, in which case, for example, a two-dimensional image PG with an aspect ratio of 1:2 is generated. However, depending on the application or the application settings, the aspect ratio is not limited to 1:2. This type of format is also called, for example, an "Equirectangular Image."

The combining unit 11 executes a combining process on the two-dimensional image PG. The "combining process" is a process of combining one end PGa of the two-dimensional image PG with the other end PGb opposite the one end PGa. In the example shown in FIG. 2, the wall is displayed divided into one end PGa and the other end PGb, so that the wall divided into two by the combining process is recognized as a continuous, integrated object. This makes it possible to appropriately recognize the wall in the segmentation described below.

The segmentation unit 12 performs segmentation on the indoor image RG received by the indoor image input receiving unit 10. For example, the segmentation unit 12 may perform segmentation on the two-dimensional image PG included in the indoor image RG. In other words, the segmentation unit 12 may perform segmentation on a portion of an image cut out from a celestial sphere image. Also, the segmentation unit 12 may perform segmentation on the two-dimensional image PG on which a combining process has been executed. "Segmentation" is a process of identifying the attributes (meanings) of each object (part) recognized in an image. For example, the ceiling, floor, wall, window, door, or kitchen included in the indoor image RG can be identified by segmentation.

The segmentation unit 12 may employ a known method for segmentation. For example, the segmentation unit 12 may employ semantic segmentation, instance segmentation, panoptic segmentation, or the like. Here, the segmentation unit 12 may perform at least two types of segmentation on the indoor image RG. Specifically, the segmentation unit 12 may perform at least semantic segmentation and instance segmentation on the indoor image RG. This allows the strengths and weaknesses of each type of segmentation to be mutually complemented. For example, it may be possible to distinguish between boundaries between rooms that are difficult to distinguish using semantic segmentation using instance segmentation.

Segmentation may also be performed on a two-dimensional image PG in which one end PGa and the other end PGb have been joined together by the segmentation unit 12 and joining unit 11 (i.e., a two-dimensional image PG on which joining processing has been performed). In the example shown in FIG. 2, a wall that was displayed divided into one end PGa and the other end PGb is recognized as a single object by the joining processing, and therefore the segmentation unit 12 can easily recognize this wall.

The segmentation unit 12 may also perform segmentation on the floor plan image FG, which will be described later. In this case, the segmentation unit 12 identifies the attributes (meanings) of each room, etc., in the floor plan image FG.

The indoor structure estimation unit 13 estimates the indoor structure of the real estate property. The "indoor structure" is the structure of the room. For example, the indoor structure may include the positional relationships of walls, corners, doors, windows, etc. More specifically, the indoor structure estimation unit 13 estimates the indoor structure, which is the structure of the room, based on the results of segmentation performed on the indoor image RG by the segmentation unit 12. The indoor structure estimation unit 13 may further estimate the room type of the real estate property based on the indoor image RG.

The interior structure estimation unit 13 also performs image processing on the interior image RG. The interior structure estimation unit 13 may estimate the interior structure based on the results of the image processing in addition to the results of the segmentation.

An example of the image processing performed on the indoor image RG is vanishing point correction processing. The "vanishing point correction processing" is a process of detecting a vanishing point in the indoor image RG and correcting the tilt (distortion) of the image using the detected vanishing point. Specifically, the vanishing point correction processing is a process of identifying a plurality of lines (straight lines) that are parallel to each other in the real world and correcting the straight lines to be parallel in the indoor image RG. For example, the straight lines include a plurality of boundary lines that separate a wall and a floor. In the indoor image RG before the vanishing point correction processing, the straight lines that should be parallel are generally tilted to form a vanishing point. One or more vanishing points may be formed in the indoor image RG. Therefore, the indoor structure estimation unit 13 detects the vanishing point using an arbitrary algorithm and performs image conversion processing to correct the tilt (distortion) of the image. Note that a known method can be adopted as the algorithm for detecting the vanishing point, and as an example, RANSAC (Random Sample Consensus), which is a robust estimation algorithm, may be adopted. Also, known methods can be used for image conversion processing, such as homography conversion.

Top-view image correction unit 14 corrects the indoor structure estimated by indoor structure estimation unit 13, and in particular corrects the indoor structure by performing correction on top-view image TG. For example, top-view image correction unit 14 corrects the indoor structure for each room of a real estate property. A "top-view image" is an image of the room viewed from above in a celestial sphere image of the room included in indoor image RG. FIG. 3 is a diagram showing an example of a top-view image TG corresponding to indoor image RG of a real estate property. As shown in FIG. 3, top-view image TG is an image of a plan view looking down on the room generated from indoor image RG. Top-view image correction unit 14 corrects the position of each part in top-view image TG automatically or manually by the user. For example, top-view image correction unit 14 corrects the position of a wall or a corner in top-view image TG.

When the top view image TG is corrected by the top view image correction unit 14, the indoor image RG is corrected accordingly. The indoor structure is then estimated based on the results of segmentation of the corrected indoor image RG. Therefore, by correcting the top view image TG, the estimated indoor structure is corrected.

The floor plan image input receiving unit 15 receives input of a floor plan image FG. A "floor plan image" is a diagram showing the floor plan of a real estate property. FIG. 4 is a diagram showing a floor plan image FG of a real estate property. As shown in FIG. 4, the floor plan image FG is, for example, a plan view showing a schematic view of the entire real estate property. Here, the floor plan image FG of a real estate property is shown, which has a room type of 1LDK and further has a corridor (entrance), a bathroom (dressing room), a toilet, and a balcony. The real estate property related to the floor plan image FG is the same as the real estate property related to the interior image RG. The floor plan image input receiving unit 15 receives input of the floor plan image FG from, for example, the user terminal 2.

The property structure estimation unit 16 estimates the property structure based on the floor plan image FG input by the floor plan image input reception unit 15. The "property structure" is the structure of the real estate property. Specifically, the property structure is the overall structure of the real estate property that can be estimated from the floor plan image FG. The property structure estimation unit 16 may further estimate the room type of the real estate property based on the floor plan image FG.

The matching unit 17 matches the interior structure estimated by the interior structure estimation unit 13 with the property structure estimated by the property structure estimation unit 16. That is, the matching unit 17 compares the interior structure estimated by the interior structure estimation unit 13 with the property structure estimated by the property structure estimation unit 16 to estimate the shape, orientation, and relationship of the ceiling, floor, wall, window, door, kitchen, etc. of the real estate property. For example, the matching unit 17 matches the interior structure with the property structure to estimate the connection between rooms (between doors). This reduces the ambiguity of the information and improves the accuracy of the generated three-dimensional model. Note that if room type information has been acquired by the room type information acquisition unit 19, the matching unit 17 can perform more accurate matching by adding the room type information to the input information.

The relationship estimation unit 18 estimates the relationship between the multiple indoor images RG based on the result of matching between the indoor structure and the property structure performed by the matching unit 17. The relationship between the multiple indoor images RG may be at least one of the connection relationship and the positional relationship between the multiple indoor images RG. That is, when the input of the multiple indoor images RG is accepted by the indoor image input acceptance unit 10, the relationship estimation unit 18 may estimate the connection relationship between these multiple indoor images RG (which indoor image RG and which indoor image RG are connected by which door, etc.). In addition, when the input of the multiple indoor images RG is accepted by the indoor image input acceptance unit 10, the relationship estimation unit 18 may estimate the positional relationship between these multiple indoor images RG (what kind of positional relationship is between a certain indoor image RG and another indoor image RG, etc.). It is particularly preferable that the relationship estimation unit 18 estimates both the connection relationship and the positional relationship between the multiple indoor images RG at the same time.

The room type information acquisition unit 19 acquires room type information. "Room type information" is information related to the room type of a real estate property, in other words, information indicating what the room type of the real estate property is. When the room type has been estimated by the interior structure estimation unit 13, the room type information acquisition unit 19 may acquire room type information from the interior structure estimation unit 13. When the room type has been estimated by the property structure estimation unit 16, the room type information acquisition unit 19 may acquire room type information from the property structure estimation unit 16. The room type information acquisition unit 19 may acquire room type information by, for example, accepting input of room type information from the user terminal 2.

The three-dimensional model generation unit 20 generates a three-dimensional model of a real estate property. As described above, a three-dimensional model is a structural model that is constructed in a virtual three-dimensional space and can be displayed on a display. The three-dimensional model generation unit 20 can generate a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space based on various information related to the real estate property.

For example, the three-dimensional model generating unit 20 may generate a three-dimensional model of the real estate property based on the interior structure estimated by the interior structure estimating unit 13. In other words, the three-dimensional model generating unit 20 may generate a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space from the interior structure estimated from the interior image RG.

The three-dimensional model generating unit 20 may generate a three-dimensional model of the real estate property based on the result of matching between the interior structure and the property structure performed by the matching unit 17. In other words, the three-dimensional model generating unit 20 may generate a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space from the property structure estimated from the floor plan image FG in addition to the interior structure estimated from the interior image RG.

The three-dimensional model generating unit 20 may generate a three-dimensional model of the real estate property based on the result of matching between the interior structure and the property structure performed by the matching unit 17 and the room type information acquired by the room type information acquiring unit 19. In other words, the three-dimensional model generating unit 20 may generate a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space from the room type information of the real estate property, in addition to the interior structure estimated from the interior image RG and the property structure estimated from the floor plan image FG.

In addition, when multiple indoor images RG are received for a real estate property and the relationship between the multiple indoor images RG (at least one of the connection relationship and the positional relationship between the multiple indoor images RG) is estimated by the relationship estimation unit 18, the three-dimensional model generation unit 20 may generate a three-dimensional model of the real estate property based on the result of matching between the indoor structure and the property structure performed by the matching unit 17 and the relationship between the multiple indoor images RG (at least one of the connection relationship and the positional relationship between the multiple indoor images RG) estimated by the relationship estimation unit 18. In other words, the three-dimensional model generation unit 20 may generate a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space from the property structure estimated from the floor plan image FG in addition to the indoor structure estimated from the indoor images RG. Furthermore, at this time, the three-dimensional model generating unit 20 may generate a three-dimensional model of the real estate property based on the result of matching between the interior structure and the property structure performed by the matching unit 17, the relationship between the multiple interior images RG estimated by the relationship estimating unit 18 (at least one of the connection relationship and the positional relationship between the multiple interior images RG), and the room type information acquired by the room type information acquiring unit 19.

The relationship correction unit 21 corrects the structure of the combined interior structures of the rooms of the real estate property as a whole. For example, the relationship correction unit 21 may correct at least one of the connection relationships and the positional relationships between the multiple interior images RG. The relationship correction unit 21 may correct the interior structure by making corrections to a collective top-view image in which top-view images TG of multiple rooms (all or some of the rooms) are displayed together. Alternatively, the relationship correction unit 21 may make corrections to images other than the top-view image TG. The relationship correction unit 21 may correct the structure of the entire real estate property by accepting manual corrections.

[3D model generation process]
Fig. 5 is a flowchart showing the flow of a three-dimensional model generation process executed by the three-dimensional model generation device 1. The three-dimensional model generation process shown in Fig. 5 is started, for example, by the computer C of the three-dimensional model generation device 1 executing a three-dimensional model generation program P.

In step S10, the indoor image input receiving unit 10 of the 3D model generation device 1 receives the input of the indoor image RG. Here, the indoor image input receiving unit 10 receives the input of a plurality of indoor images RG for a real estate property. The indoor image input receiving unit 10 receives the input of the indoor image RG from, for example, the user terminal 2. After that, the 3D model generation process proceeds to step S12.

In step S12, the combining unit 11 of the three-dimensional model generating device 1 executes a combining process on the two-dimensional image PG. Here, since the wall is displayed divided into two parts, one end PGa and the other end PGb (FIG. 2), the combining process recognizes the divided wall as a continuous, integrated object. After that, the three-dimensional model generating process proceeds to step S14.

In step S14, the segmentation unit 12 of the three-dimensional model generating device 1 performs segmentation on the indoor image RG. Here, the segmentation unit 12 performs segmentation on the two-dimensional image PG, of the indoor image RG, on which the combining process has been performed. Also, here, the segmentation unit 12 performs semantic segmentation and instance segmentation on the indoor image RG. After that, the three-dimensional model generation process proceeds to step S16.

In step S16, the interior structure estimation unit 13 of the three-dimensional model generation device 1 estimates the interior structure of the real estate property. More specifically, the interior structure estimation unit 13 estimates the interior structure, which is the structure of the room, based on the result of segmentation performed on the interior image RG by the segmentation unit 12. The interior structure estimation unit 13 may perform image processing on the interior image RG and estimate the interior structure based on the result of the image processing in addition to the result of segmentation. The interior structure estimation unit 13 may also further estimate the room type of the real estate property based on the interior image RG. After that, the three-dimensional model generation process proceeds to step S18.

In step S18, top-view image correction unit 14 of three-dimensional model generation device 1 performs correction on the top-view image TG. Top-view image correction unit 14 corrects the position of each part in the top-view image TG automatically, or manually by the user. When the top-view image TG is corrected by top-view image correction unit 14, the indoor image RG is corrected accordingly. The indoor structure is then estimated based on the results of segmentation of the corrected indoor image RG. Therefore, by correcting the top-view image TG, the estimated indoor structure is corrected. After that, the three-dimensional model generation process proceeds to step S20.

In step S20, the floor plan image input receiving unit 15 of the three-dimensional model generation device 1 receives input of a floor plan image FG. Here, the floor plan image input receiving unit 15 receives input of a floor plan image FG of a real estate property whose room type is 1LDK and which further has a corridor (entrance), a bathroom (dressing room), a toilet, and a balcony. The real estate property related to the floor plan image FG is the same as the real estate property related to the interior image RG. The floor plan image input receiving unit 15 receives input of the floor plan image FG from, for example, the user terminal 2. The three-dimensional model generation process then proceeds to step S22.

In step S22, the property structure estimation unit 16 of the three-dimensional model generation device 1 estimates the property structure based on the floor plan image FG input by the floor plan image input reception unit 15. The "property structure" is the structure of the real estate property. Specifically, the property structure is the overall structure of the real estate property that can be estimated from the floor plan image FG. The property structure estimation unit 16 may further estimate the room type of the real estate property based on the floor plan image FG. The three-dimensional model generation process then proceeds to step S24.

In step S24, the room type information acquisition unit 19 of the three-dimensional model generation device 1 acquires room type information. When the room type has been estimated by the indoor structure estimation unit 13, the room type information acquisition unit 19 may acquire the room type information from the indoor structure estimation unit 13. When the room type has been estimated by the property structure estimation unit 16, the room type information acquisition unit 19 may acquire the room type information from the property structure estimation unit 16. The room type information acquisition unit 19 may acquire the room type information by, for example, accepting input of room type information from the user terminal 2. After that, the three-dimensional model generation process proceeds to step S26.

In step S26, the matching unit 17 of the three-dimensional model generating device 1 matches the interior structure estimated by the interior structure estimation unit 13 with the property structure estimated by the property structure estimation unit 16. Note that here, the relationship correction unit 21 of the three-dimensional model generating device 1 may correct the structure of the entire combination of the interior structures of each room of the real estate property. After that, the three-dimensional model generation process proceeds to step S28.

In step S28, the relationship estimation unit 18 of the three-dimensional model generation device 1 estimates the connection relationship and positional relationship between the multiple indoor images RG based on the result of matching between the indoor structure and the property structure performed by the matching unit 17. Note that the relationship estimation unit 18 may estimate only either the connection relationship or the positional relationship between the multiple indoor images RG. After that, the three-dimensional model generation process proceeds to step S30.

In step S30, the three-dimensional model generating unit 20 of the three-dimensional model generating device 1 generates a three-dimensional model of the real estate property. Here, the three-dimensional model generating unit 20 generates a three-dimensional model of the real estate property based on the result of matching between the interior structure and the property structure performed by the matching unit 17, the relationship between the multiple interior images RG estimated by the relationship estimation unit 18 (at least one of the connection relationship and the positional relationship between the multiple interior images RG), and the room type information acquired by the room type information acquisition unit 19. In other words, the three-dimensional model generating unit 20 generates a three-dimensional model that reproduces the real estate property in a virtual three-dimensional space from the room structure estimated from the interior images RG and the property structure estimated from the floor plan image FG, as well as the room type information of the real estate property. After that, the three-dimensional model generation process proceeds to step S32.

In step S32, the relationship correction unit 21 of the three-dimensional model generation device 1 corrects the structure of the entire combination of the interior structures of each room of the real estate property. For example, the relationship correction unit 21 corrects at least one of the connection relationship and the positional relationship between the multiple interior images RG. This completes the three-dimensional model generation process.

[3D model generation program]
Fig. 6 is a block diagram showing the three-dimensional model generating program P. The three-dimensional model generating program P shown in Fig. 6 is a program for causing a computer C to function as a three-dimensional model generating device 1.

The three-dimensional model generation program P includes a main module MM, an indoor image input reception module M10, a combination module M11, a segmentation module M12, an indoor structure estimation module M13, a top view image correction module M14, a floor plan image input reception module M15, a property structure estimation module M16, a matching module M17, a relationship estimation module M18, a room type information acquisition module M19, a three-dimensional model generation module M20, and a relationship correction module M21. The main module MM is the part that controls the computer C overall. The functions realized by executing each of the indoor image input reception module M10, the combination module M11, the segmentation module M12, the indoor structure estimation module M13, the top view image correction module M14, the floor plan image input reception module M15, the property structure estimation module M16, the matching module M17, the relationship estimation module M18, the room type information acquisition module M19, the three-dimensional model generation module M20, and the relationship correction module M21 are the same as the functions possessed by each of the above-mentioned indoor image input reception unit 10, the combination unit 11, the segmentation unit 12, the indoor structure estimation unit 13, the top view image correction unit 14, the floor plan image input reception unit 15, the property structure estimation unit 16, the matching unit 17, the relationship estimation unit 18, the room type information acquisition unit 19, the three-dimensional model generation unit 20, and the relationship correction unit 21.

The three-dimensional model generation program P does not necessarily have to include at least any module other than the indoor image input reception module M10, the segmentation module M12, the indoor structure estimation module M13, and the three-dimensional model generation module M20.

[Action and Effect]
As described above, the three-dimensional model generation device 1 comprises an indoor image input receiving unit 10 that receives input of an indoor image RG captured inside a real estate property, a segmentation unit 12 that performs segmentation on the indoor image RG received by the indoor image input receiving unit 10, an indoor structure estimation unit 13 that estimates the indoor structure, which is the structure of the room, based on the result of the segmentation performed on the indoor image RG by the segmentation unit 12, and a three-dimensional model generation unit 20 that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimation unit 13.

The three-dimensional model generation program P causes the computer C to function as an indoor image input receiving unit 10 that receives input of an indoor image RG captured inside a real estate property, a segmentation unit 12 that performs segmentation on the indoor image RG received as input by the indoor image input receiving unit 10, an indoor structure estimation unit 13 that estimates the indoor structure, which is the structure inside the room, based on the results of the segmentation performed on the indoor image RG by the segmentation unit 12, and a three-dimensional model generation unit 20 that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimation unit 13.

According to at least one of the three-dimensional model generation device 1 and the three-dimensional model generation program P, the attributes of each part included in the indoor image RG, which is an image of the interior of a real estate property, are grasped by performing segmentation on the indoor image RG, and the indoor structure is estimated based on the grasped attributes of each part. Then, a three-dimensional model of the real estate property is generated based on the indoor structure estimated in this way. Therefore, compared to a case where a three-dimensional model is generated based on the indoor structure simply estimated from a floor plan or the like, a three-dimensional model that reflects the indoor situation in detail is generated. Therefore, it becomes possible to generate a three-dimensional model of the real estate property that includes more detailed information.

In the three-dimensional model generating device 1, the segmentation unit 12 performs at least two types of segmentation on the indoor image RG. This makes it possible to appropriately distinguish objects that are difficult to distinguish using a particular type of segmentation by performing a different type of segmentation.

In the three-dimensional model generating device 1, the segmentation unit 12 performs at least semantic segmentation and instance segmentation on the indoor image RG. This makes it possible to appropriately distinguish room boundaries, which are difficult to distinguish using semantic segmentation, by performing instance segmentation.

In the three-dimensional model generation device 1, the indoor image RG includes a spherical image of the room, and a top-view image correction unit 14 is provided that performs correction on the top-view image TG, which is an image of the room viewed from above in the spherical image. This makes it possible to correct the positions of walls, corners, and the like of the estimated indoor structure, and therefore to generate a more accurate three-dimensional model. Moreover, because the indoor structure can be corrected from the top-view image TG, more intuitive operation is possible, improving convenience.

In the three-dimensional model generating device 1, the indoor image RG includes a two-dimensional image PG that is a part of the omnidirectional image of the room, and includes a connecting unit 11 that connects one end PGa of the two-dimensional image PG to the other end PGb opposite to the one end PGa, and the segmentation unit 12 performs segmentation on the two-dimensional image PG in which the one end PGa and the other end PGb are connected to each other by the connecting unit 11. The two-dimensional image PG included in the indoor image RG constitutes a part of the omnidirectional image of the room. For example, the two-dimensional image PG may be a panoramic image captured over 360 degrees in the horizontal direction. Such a two-dimensional image PG is a band-shaped image with one end PGa in the horizontal direction as the base end and the other end PGb, which is 360 degrees from the one end PGa, as the terminal end. At this time, the one end PGa and the other end PGb of the band-shaped two-dimensional image PG capture the same location in the room, for example, a location on a straight line extending in the vertical direction. Here, an object existing so as to overlap one end PGa of the two-dimensional image PG, i.e., the other end PGb, is displayed in two-dimensional image PG, divided into one end PGa side and the other end PGb side. In this case, it becomes difficult for the segmentation unit 12 to appropriately perform segmentation on the object displayed in two-division into one end PGa side and the other end PGb side. As a result, it becomes difficult to correctly grasp the attributes of the part where the object is displayed on the two-dimensional image PG, and it becomes difficult to correctly estimate the indoor structure. Therefore, in the above-mentioned three-dimensional model generating device 1, by connecting one end PGa and the other end PGb of the belt-shaped two-dimensional image PG to each other to form a ring shape, the object existing so as to overlap one end PGa of the two-dimensional image PG, i.e., the other end PGb, is no longer divided into two, and it becomes easier to appropriately perform segmentation on the object. As a result, the attributes of the part of the two-dimensional image PG where the object is displayed can be easily understood, and the interior structure can be easily estimated. In this way, by performing segmentation on the two-dimensional image PG in which one end PGa and the other end PGb are connected to each other, it is possible to appropriately perform segmentation on parts where the accuracy of image recognition is likely to decrease due to the two-dimensional image PG being cut off. Note that, although a panoramic image captured over 360 degrees in the horizontal direction is given as an example of the two-dimensional image PG here, the two-dimensional image PG does not necessarily have to be an image of this type, and may be, for example, a panoramic image captured over 360 degrees in a direction intersecting the horizontal direction.

The three-dimensional model generating device 1 includes a floor plan image input receiving unit 15 that receives input of a floor plan image FG showing the layout of a real estate property, a property structure estimating unit 16 that estimates the property structure, which is the structure of the real estate property, based on the floor plan image FG input received by the floor plan image input receiving unit 15, and a matching unit 17 that matches the indoor structure estimated by the indoor structure estimating unit 13 with the property structure estimated by the property structure estimating unit 16. The three-dimensional model generating unit 20 generates a three-dimensional model of the real estate property based on the result of matching between the indoor structure and the property structure performed by the matching unit 17. According to this, the three-dimensional model of the real estate property is generated based not only on the indoor structure estimated based on the indoor image RG, but also on the property structure estimated based on the floor plan image FG. Therefore, information about the overall structure of the real estate property, which is likely to be unclear based on the indoor structure alone, can be supplemented based on the property structure. Therefore, it is possible to generate a three-dimensional model of the real estate property with high accuracy.

The three-dimensional model generating device 1 includes a room type information acquisition unit 19 that acquires room type information related to the room types of the real estate property, and the three-dimensional model generating unit 20 generates a three-dimensional model of the real estate property based on the results of matching between the interior structure and the property structure performed by the matching unit 17 and the room type information acquired by the room type information acquisition unit 19. This allows the three-dimensional model of the real estate property to be generated based also on the room type information, which is information related to the type and number of rooms that the real estate property has. This makes it possible to generate a three-dimensional model of the real estate property with high accuracy.

In the three-dimensional model generating device 1, the indoor image input receiving unit 10 can receive input of multiple indoor images RG for a real estate property, and is equipped with a relationship estimation unit 18 that estimates at least one of the connection relationship and the positional relationship between the multiple indoor images RG based on the result of matching between the indoor structure and the property structure performed by the matching unit 17. With this, at least one of the connection relationship and the positional relationship between the multiple indoor images RG of the real estate property is estimated with high accuracy based on the result of matching between the indoor structure estimated by the indoor structure estimation unit 13 and the property structure estimated by the property structure estimation unit 16. Therefore, it becomes possible to generate a three-dimensional model of the real estate property with high accuracy.

[Modifications]
The above-described embodiment can be implemented in various forms that are modified or improved based on the knowledge of those skilled in the art.

For example, in the above-described embodiment, the two-dimensional image PG is a panoramic image captured across 360 degrees in the horizontal direction. However, the two-dimensional image PG does not have to be captured across 360 degrees in the horizontal direction, and may be, for example, a panoramic image captured across 270 degrees in the horizontal direction, a panoramic image captured across 180 degrees in the horizontal direction, a panoramic image captured across 90 degrees in the horizontal direction, or a panoramic image captured across an angle other than the above in the horizontal direction. Furthermore, the two-dimensional image PG does not have to be captured so as to have a wide angle in the horizontal direction, and may be, for example, a panoramic image captured so as to have a wide angle in a direction intersecting the horizontal direction. As an example, the two-dimensional image PG may be a panoramic image captured so as to have a wide angle in the vertical direction.

1 Three-dimensional model generating device 10 Indoor image input receiving unit 11 Combining unit 12 Segmentation unit 13 Indoor structure estimation unit 14 Top-view image correction unit 15 Floor plan image input receiving unit 16 Property structure estimation unit 17 Matching unit 18 Relationship estimation unit 19 Room type information acquisition unit 20 Three-dimensional model generating unit 21 Relationship correction unit C Computer FG Floor plan image P Three-dimensional model generating program PG Two-dimensional image PGa One end PGb Other end RG Indoor image TG Top-view image

Claims (7)

an indoor image input receiving unit that receives an input of an indoor image captured inside a room of a real estate property having a room;
a segmentation unit that performs segmentation on the indoor image input received by the indoor image input receiving unit;
an indoor structure estimation unit that estimates an indoor structure that is a structure within the room based on a result of the segmentation performed on the indoor image by the segmentation unit;
a three-dimensional model generating unit that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimating unit,
The segmentation unit performs at least semantic segmentation and instance segmentation on the indoor image;
A three-dimensional model generating device, wherein the instance segmentation is used to determine the boundaries of the rooms contained in the indoor image. the indoor image includes a celestial sphere image of the room and a two-dimensional image that is a part of the celestial sphere image of the room,
a top-view image correction unit that corrects a top-view image, which is an image of the room viewed from above, in the omnidirectional image;
the top image correction unit corrects a position of a part in the top image;
2. The three-dimensional model generating device according to claim 1, wherein the two-dimensional image is corrected in association with correcting the position of the part in the top view image, and the indoor image is corrected and estimated based on a segmentation result for the corrected two-dimensional image. the indoor image includes a two-dimensional image that is a part of a celestial sphere image of the room,
a connecting portion for connecting one end of the two-dimensional image to another end opposite to the one end,
The three-dimensional model generating device according to claim 1 , wherein the segmentation unit performs segmentation on the two-dimensional image in which the one end and the other end are connected to each other by the connection unit. a floor plan image input receiving unit that receives an input of a floor plan image showing a floor plan of the real estate property;
a property structure estimation unit that estimates a property structure that is a structure of the real estate property based on the floor plan image input received by the floor plan image input receiving unit;
a matching unit that performs matching between the indoor structure estimated by the indoor structure estimation unit and the property structure estimated by the property structure estimation unit,
The three-dimensional model generating device according to claim 1 , wherein the three-dimensional model generating unit generates the three-dimensional model of the real estate property based on a result of matching between the interior structure and the property structure performed by the matching unit. a room type information acquisition unit that acquires room type information related to a room type of the real estate property,
The three-dimensional model generating device according to claim 4, wherein the three-dimensional model generating unit generates the three-dimensional model of the real estate property based on a result of matching between the interior structure and the property structure performed by the matching unit and the room type information acquired by the room type information acquisition unit. The indoor image input receiving unit is capable of receiving input of a plurality of indoor images of the real estate property,
The three-dimensional model generating device according to claim 4 , further comprising a relationship estimation unit that estimates at least one of a connection relationship and a positional relationship between the multiple indoor images based on a result of matching between the indoor structure and the property structure performed by the matching unit. Computer,
an indoor image input receiving unit that receives an input of an indoor image captured inside a room of a real estate property having a room;
a segmentation unit that performs segmentation on the indoor image input received by the indoor image input receiving unit;
an indoor structure estimation unit that estimates an indoor structure that is a structure within the room based on a result of the segmentation performed on the indoor image by the segmentation unit;
a three-dimensional model generating unit that generates a three-dimensional model of the real estate property based on the indoor structure estimated by the indoor structure estimating unit;
The segmentation unit performs at least semantic segmentation and instance segmentation on the indoor image;
A three-dimensional model generation program, wherein the instance segmentation is used to determine the boundaries of the rooms contained in the indoor image.

Images