JP7704404B2 - Map information generating device, location identifying device, map information generating method, location identifying method, program, and recording medium - Google Patents

Description

The present invention relates to a map information generating device, a location identifying device, a map information generating method, a location identifying method, a program, and a recording medium.

Technology has been reported that estimates the position of a terminal from images captured by the terminal (for example, Patent Document 1, etc.).

JP 2017-21427 A

However, in order to more accurately identify the location of the device that captured the image, more accurate map information needs to be used. For this reason, there is a demand for technology that can generate more accurate map information.

The present invention aims to provide a map information generating device, a location identifying device, a map information generating method, a location identifying method, a program, and a recording medium that are capable of generating more accurate map information.

In order to achieve the above object, the map information generating device of the present invention comprises:
The apparatus includes an image acquisition unit, a three-dimensional model generation unit, and a map information generation unit,
The image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
The map information generation unit is a device that associates positions of point cloud data in the first three-dimensional model with positions of point cloud data in the second three-dimensional model, and generates map information of the specific area.

The map information generating method of the present invention includes the steps of:
The method includes an image acquisition step, a three-dimensional model generation step, and a map information generation step,
The image acquisition step acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generating step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
The map information generating step is a method of associating positions of point cloud data in the first three-dimensional model with positions of point cloud data in the second three-dimensional model, and generating map information of the specific area.

According to the present invention, it is possible to generate more accurate map information. Furthermore, according to the present invention, the map information can be used to more accurately identify, for example, the location of the terminal that captured the image.

FIG. 1 is a block diagram showing an example of a configuration of a map information generating device according to the first embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of the map information generating device according to the first embodiment. FIG. 3 is a flowchart showing an example of processing in the map information generating device of the first embodiment. FIG. 4 is a block diagram illustrating an example of a configuration of a position specifying device according to the second embodiment. FIG. 5 is a block diagram illustrating an example of a hardware configuration of a position specifying device according to the second embodiment. FIG. 6 is a flowchart illustrating an example of processing in the position specifying device according to the second embodiment. FIG. 7 is a flowchart showing an example of a process in the position specifying device of the first modified example. FIG. 8 is a schematic diagram showing an example of the positional relationship between the user terminal and the positional information of the virtual image. FIG. 9 is a schematic diagram showing an example in which a composite image output from the map information generating device to a user terminal is displayed on the display of the user terminal.

In the map information generating device of the present invention, for example,
The three-dimensional image information may include a panoramic image obtained by capturing an image of the specific area in a three-dimensional manner.

The map information generating device of the present invention, for example,
Further, a feature point extraction unit is included,
The feature point extraction unit extracts feature points from the two-dimensional image information by machine learning,
The map information generating unit may generate the map information to which the feature points are linked.

The position specifying device of the present invention comprises:
The present invention includes a map information generating unit, a user terminal information acquiring unit, and an identifying unit,
The map information generating unit is a map information generating device of the present invention,
The user terminal information acquisition unit acquires a captured image and sensor information from a user terminal,
The identification unit is a device that identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

In the position specifying device of the present invention, for example,
the map information generating unit is a map information generating device including a feature point extracting unit,
The feature point extraction unit extracts feature points from the captured image by machine learning,
The identification unit identifies a plurality of candidate points as the location of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
A position of the user terminal may be specified from the candidate point, and an imaging direction of the user terminal may be specified based on point cloud data of the map information and the sensor information.

In the map information generating method of the present invention, for example,
The three-dimensional image information may include a panoramic image obtained by capturing an image of the specific area in a three-dimensional manner.

The map information generating method of the present invention includes, for example,
Further, a feature point extraction step is included,
The feature point extraction step extracts feature points from the two-dimensional image information by machine learning,
The map information generating step may generate the map information linked to the feature points.

The position identification method of the present invention includes the steps of:
The method includes a map information generating step, a user terminal information acquiring step, and a specifying step,
The map information generating step is a step of executing each step of the map information generating method of the present invention,
The user terminal information acquisition step acquires a captured image and sensor information from a user terminal,
The identifying step is a method of identifying a position of the user terminal based on the point cloud data of the map information and the captured image, and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.

In the position identification method of the present invention, for example,
the map information generating step is a step of executing each step of a map information generating method including a feature point extracting step,
The feature point extraction step includes extracting feature points from the captured image by machine learning,
The specifying step includes specifying a plurality of candidate points as the position of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
A position of the user terminal may be specified from the candidate point, and an imaging direction of the user terminal may be specified based on point cloud data of the map information and the sensor information.

The program of the present invention is a program for causing a computer to execute each step of the method of the present invention as a procedure.

The recording medium of the present invention is a computer-readable recording medium on which the program of the present invention is recorded.

Next, an embodiment of the present invention will be described with reference to the drawings. The present invention is not limited to the following embodiment. In each of the drawings, the same parts are given the same reference numerals. Furthermore, the explanations of each embodiment can be mutually incorporated unless otherwise specified, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
Fig. 1 is a block diagram showing an example of the configuration of a map information generating device 10 according to the present embodiment. As shown in Fig. 1, the device 10 includes an image acquisition unit 11, a three-dimensional model generating unit 12, and a map information generating unit 13. The device 10 may further include, for example, a feature point extraction unit 14 as an optional configuration.

The device 10 may be, for example, a single device including each of the above-mentioned parts, or a device to which each of the above-mentioned parts can be connected via a communication line network. The device 10 may also be connected to an external device described later via the communication line network. The communication line network is not particularly limited and may be a publicly known network, for example, wired or wireless. Examples of the communication line network include the Internet line, WWW (World Wide Web), telephone line, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), L5G (Local 5G), etc. Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (Ad Hoc communication), infrastructure communication, indirect communication via an access point, and the like. The device 10 may be incorporated into a server as a system. The device 10 may be, for example, a personal computer (PC, for example, desktop type or notebook type) in which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. Furthermore, the device 10 may be in the form of cloud computing or edge computing, for example, in which at least one of the parts is on a server and the other parts are on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a central processing unit (CPU, GPU, etc.) 101, a memory 102, a bus 103, a storage device 104, an input device 105, a display device 106, a communication device 107, an imaging device 108, and an audio input/output device 109. Each part of the device 10 is connected to each other via the bus 103 and each interface (I/F).

The central processing unit 101 is responsible for the overall control of the device 10. In the device 10, the central processing unit 101 executes, for example, the program of the present invention and other programs, and also reads and writes various types of information. Specifically, for example, the central processing unit 101 functions as an image acquisition unit 11, a three-dimensional model generation unit 12, a map information generation unit 13, and a feature point extraction unit 14.

The bus 103 can also be connected to, for example, an external device. Examples of the external device include an external storage device (external database, etc.), a printer, an external input device, an external display device, an external imaging device, etc. The present device 10 can be connected to an external network (the above-mentioned communication line network) by, for example, a communication device 107 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the central processing unit 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 described below, and the central processing unit 101 receives data from the memory 102 and executes the program. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read only memory).

The storage device 104 is also referred to as an auxiliary storage device, for example, in contrast to the main memory. As described above, the storage device 104 stores operating programs including the program of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from the recording medium. The recording medium is not particularly limited, and may be, for example, an internal or external type, such as a hard disk drive (HDD), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, memory card, etc. The storage device 104 may be, for example, a hard disk drive (HDD) in which the recording medium and the drive are integrated, or a solid state drive (SSD).

In the present device 10, the memory 102 and storage device 104 can also store various information such as log information, information acquired from an external database (not shown) or an external device, information generated by the present device 10, and information used by the present device 10 when executing processing. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

The device 10 may further include, for example, an input device 105, a display device 106, an imaging device 108, and an audio input/output device 109. The input device 105 is, for example, a touch panel, a keyboard, a mouse, etc. The display device 106 is, for example, an LED display, a liquid crystal display, etc. The imaging device 108 is, for example, a 2D camera, a 3D camera, an infrared camera, etc. The audio input/output device 109 is, for example, a microphone, a speaker, etc.

Next, an example of the map information generation method of this embodiment will be described based on the flowchart in FIG. 3. The map information generation method of this embodiment is implemented, for example, as follows using the map information generation device 10 in FIG. 1. Note that the map information generation method of this embodiment is not limited to the use of the map information generation device 10 in FIG. 1. Note that in FIG. 3, the steps shown in parentheses are optional steps and do not need to be processed.

First, the image acquisition unit 11 acquires three-dimensional image information and two-dimensional image information that capture a specific area (S11). The specific area is not particularly limited as long as it is indoors. Examples of indoor areas include exhibition halls, event halls, department stores, and schools such as universities. The three-dimensional image information may be, for example, information including an image captured by a three-dimensional camera, specifically, an all-around panoramic image that captures the specific area in a three-dimensional manner. The three-dimensional camera means a camera that can capture not only vertical and horizontal information but also depth (depth), and specifically, for example, a three-dimensional camera such as Matterport. The three-dimensional camera is, for example, an external device. On the other hand, the two-dimensional image information is, for example, information including an image captured by a two-dimensional camera. The two-dimensional camera means a camera that can capture vertical and horizontal information. Each of the images may be, for example, a still image or a video. In addition, each of the images in the three-dimensional image information and the two-dimensional image information may be, for example, a plurality of images, and may be an all-around panoramic image by joining the plurality of images. The image acquisition unit 11 may generate the omnidirectional panoramic image by acquiring, for example, a plurality of the images. On the other hand, the acquisition from the plurality of images is performed, for example, via the communication line network.

Next, the three-dimensional model generating unit 12 generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information (S12). The first and second three-dimensional models are models that indicate the configuration within the specific area and include point cloud data having shape information. The shape information is, for example, information that indicates the three-dimensional shape of the surface within the specific area. The first and second three-dimensional models have, for example, coordinate information based on the point cloud data. The coordinate information may be, for example, two-dimensional coordinate information (x, y) consisting of horizontal (x) and depth (y) values, or may be three-dimensional coordinate information (x, y, z) that also includes a height (z) value. The configuration within the specific area includes, for example, not only the structure within the specific area, but also the position and shape of objects placed in the specific area.

Below, an example of generating the first three-dimensional model from the three-dimensional image information will be specifically described, but the present invention is not limited to this example and known techniques can be used. First, the three-dimensional model generating unit 12 extracts point cloud data from the three-dimensional image information and performs modeling of the point cloud data using a polygon mesh. There are no particular limitations on the extraction of the point cloud data, and it is sufficient to extract locations that can become the shape information as the point cloud data. Then, the three-dimensional model generating unit 12 places an image corresponding to the point cloud data on the surface of the polygon mesh.

The three-dimensional model generating unit 12 may generate the second three-dimensional model from the two-dimensional image information, for example, by SfM (Structure from Motion) processing. However, without being limited to this, the three-dimensional model generating unit 12 may generate the second three-dimensional model from the two-dimensional image information using a known technique other than SfM processing.

Then, the map information generating unit 13 associates the positions of the point cloud data in the first three-dimensional model with the point cloud data in the second three-dimensional model, generates map information for the specific area (S13), and ends (END). The association can also be called, for example, alignment. That is, the map information generating unit 13 matches the coordinates of the first three-dimensional model with the second three-dimensional model. The method of association is not particularly limited. The map information is used, for example, to identify the position of the terminal that captured the image (two-dimensional image).

As described above, the device 10 may further include a feature point extraction unit 14. The feature point extraction unit 14 extracts feature points from the two-dimensional image information by machine learning, for example, before the step (S13). The machine learning is, for example, unsupervised learning, and is machine learning modeled on a neural network. The model, for example, extracts the feature points by inputting an image. The feature points are points that indicate characteristic locations on the image, and the image can be distinguished from other images by the points. The step (S13) may be performed before the step (S13), and may be processed before the step (S12), for example. In the step (S12), for example, the three-dimensional model generation unit 12 may link the positions of the feature points to the second three-dimensional model.

When the feature points are extracted in the step (S14), the map information generating unit 13 may generate the map information to which the feature points are linked in the step (S13), for example. The feature points and the map information may be linked, for example, by linking the second three-dimensional model and the feature points.

Three-dimensional models generated based on SfM processing have a problem in that the accuracy of location information is low. In contrast, according to this embodiment, more accurate map information can be generated by associating each point cloud data using the first three-dimensional model and the second three-dimensional model. In other words, this embodiment improves the accuracy of location information by using the first three-dimensional model generated using an image captured three-dimensionally within the specific area, and enables the location of the terminal to be identified by using the second three-dimensional model generated using an image captured two-dimensionally within the specific area. Furthermore, by extracting the feature points, map information that contributes to more accurate identification of the terminal's location can be generated.

[Embodiment 2]
Fig. 4 is a block diagram showing an example of the configuration of the position identification device 20 of this embodiment. As shown in Fig. 4, the device 20 includes a map information generation unit 21, a user terminal information acquisition unit 22, and an identification unit 23. The device 20 may further include, for example, a virtual image generation unit 24, a map information management unit 25, and a composite image information output unit 26, etc., as optional configurations.

The device 20 may be, for example, a single device including the above-mentioned units, or a device to which the above-mentioned units can be connected via a communication line network. The device 20 may be connected to the above-mentioned external device via the communication line network. The communication line network is not particularly limited and may be, for example, the same as described above. The device 20 may be, for example, a display device with a reflector in which the program of the present invention is installed. Furthermore, the device 20 may be in the form of cloud computing or edge computing, for example, in which at least one of the units is on a server and the other units are on a terminal. The device 20 may be capable of communicating with the map information generating device 10 described in the first embodiment via the communication line network.

Figure 5 shows an example block diagram of the hardware configuration of the device 20. The device 20 includes, for example, a central processing unit (CPU, GPC, etc.) 101, memory 102, bus 103, storage device 104, input device 105, display device 106, communication device 107, imaging device 108, sound input/output device 109, etc. Each part of the device 20 is connected to each other via the bus 103 and each interface (I/F). Unless otherwise specified, the description of each part can be referenced from the description of the hardware configuration of the map information generating device 10 shown in Figure 2.

The central processing unit 101 functions, for example, as a map information generating unit 21, a user terminal information acquiring unit 22, an identifying unit 23, a virtual image generating unit 24, a map information managing unit 25, and a composite image information output unit 26.

Next, an example of the location identification method of this embodiment will be described based on the flowchart in FIG. 6. The location identification method of this embodiment is implemented as follows, for example, using the location identification device 20 in FIG. 4. Note that the location identification method of this embodiment is not limited to the use of the location identification device 20 in FIG. 4.

First, the map information generating unit 21 generates map information for a specific area (S21). Specifically, the map information generating unit 21 includes the various components of the map information generating device 10 described in the first embodiment, and the above description can be used.

Next, the user terminal information acquisition unit 22 acquires the captured image and sensor information from the user terminal (S22). The captured image is an image captured by the user terminal (also referred to as a captured image). The sensor information is information measured by various sensors (e.g., an acceleration sensor, etc.) of the user terminal. The captured image and the sensor information are collectively referred to as user terminal information. The user terminal information acquisition unit 22 may, for example, acquire the sensor information by linking it to the captured image. Here, the user terminal is a terminal of a user in a specific area, and may be, for example, a personal computer (PC, for example, a desktop type or a notebook type), a smartphone, a tablet terminal, etc. The user terminal is, for example, a device used in conjunction with the position identification device 10. The configuration of the user terminal is not particularly limited, and may be, for example, at least capable of capturing the image, measuring the sensor information, and transmitting the image and the sensor information to the position identification device 20.

Then, the identification unit 23 identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information (S23), and then ends (END).

When the map information generating unit 21 is a map information generating device 10 including a feature point extraction unit 14, the identification unit 23 may, for example, in the step (S23), identify the position of the user terminal in a two-stage process. Specifically, the identification unit 23 may identify a plurality of candidate points as the position of the user terminal based on the feature points of the two-dimensional image information and the feature points of the captured image, and identify the position of the user terminal from the candidate points. The identification unit 23 also identifies the imaging direction of the user terminal as described above. The candidate points are identified, for example, by the identification unit 23 calculating the similarity between each feature point of the two-dimensional image information and the feature points of the captured image. The identification unit 23, for example, identifies a point having a feature point of the two-dimensional image information with a high similarity as the candidate point.

According to this embodiment, more accurate map information is generated, so that the position of the user terminal can be identified more accurately. Note that the position of the user terminal here refers to the position of the user terminal at the time the captured image was taken, which corresponds to, for example, the position of the user. Also, the imaging direction of the user terminal refers to the imaging direction of the user terminal at the time the captured image was taken, which corresponds to, for example, the front direction of the user. If the specific area is indoors, the position of the user terminal cannot be identified using a satellite positioning system such as GPS (Global Positioning System). However, according to this embodiment, the position of the indoor user terminal can be identified using the map information.

(Variation 1)
An example of displaying a virtual image on the user terminal will be described.

As described above, the position identification device 20 of this modified example further includes a virtual image generation unit 24, a map information management unit 25, and a composite image information output unit 26.

Next, an example of the position identification method of this modified example will be described based on the flowchart in FIG. 7. The position identification method of this modified example is implemented, for example, as follows using the position identification device 20 of FIG. 4. Note that the position identification method of this modified example is not limited to the use of the position identification device 20 of FIG. 4.

First, the map information generating unit 21 generates map information for a specific area (S31). This step (S31) is similar to the step (S21) shown in FIG. 6, and the above description can be used.

Next, the virtual image generating unit 24 generates a virtual image related to the specific area (S32). The virtual image is an image superimposed on a local image captured by a user terminal in the specific area. The virtual image generating unit 24 may generate, for example, a plurality of virtual images. Specifically, the virtual image includes, for example, a virtual image for each object in the specific area. The object is not particularly limited and can be set arbitrarily, and may be an artificial object or a natural object. Examples of the artificial object include facilities, exhibition booths, tenants, classrooms, etc. Examples of the natural object include mountains, oceans, rocks, etc. The virtual image for each object includes, for example, a position image of the object, and the position images are linked to each other for each object. The position image is, for example, an image related to the object. The generation of the virtual image may mean, for example, reading out the virtual image stored in advance in the memory 102 or the storage device 104, or may mean acquiring the virtual image from an external database via the communication line network. Furthermore, the virtual image generating unit 24 may, for example, acquire information about the object (also called object information) and generate the virtual image based on the object information.

The virtual image for each object generated by the virtual image generating unit 24 may further include a detailed information image of the object. In this case, the position image and the detailed information image of the object are linked to each other for each object. The composite image information output unit 26 described later, for example, further provides a window section in the captured image and superimposes the detailed information image linked to the position image of the object present in the captured image on the window section. The window section can display, for example, a screen in which multiple detailed information images can be transitioned. The detailed information image is, for example, an image with a hierarchical structure of one or more layers. When the detailed information image is an image with a hierarchical structure of one or more layers, the window section can display, for example, the detailed information image on a screen in which images for each layer can be transitioned. Specifically, the detailed information image is, for example, an image based on detailed information of the object, and one detailed information image or two detailed information images may be linked to one position image of the object.

Next, the map information management unit 25 manages the map information, the virtual image, and each piece of coordinate information of the virtual image in association with each other (S33). The position information of the virtual image is, for example, information indicating the position at which the virtual image is displayed. The position at which the virtual image is displayed can be arbitrarily set by, for example, an administrator.

Next, the user terminal information acquisition unit 22 acquires the captured image and sensor information from the user terminal (S34). The step (S34) is similar to the step (S22) shown in FIG. 6, and the above description can be used.

Next, the identification unit 23 identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information (S35). The step (S35) is similar to the step (S23) shown in FIG. 6, and the above description can be used.

Then, the composite image information output unit 26 outputs a composite image in which the virtual image is superimposed on the captured image (S36), and the process ends (END). Specifically, first, the composite image information output unit 26 refers to the map information based on the position and imaging direction of the user terminal and determines whether a position corresponding to the coordinate information of the virtual image exists in the captured image. If the position exists, the composite image information output unit 26 outputs a composite image in which the virtual image is superimposed on the position to the user terminal. The output may be performed, for example, via the communication line network. The output of the composite image by the composite image information output unit 26 may be performed repeatedly, for example.

Below is a description of an example of superimposing the virtual image on the captured image.

Figure 8 is a schematic diagram showing an example of the positional relationship between the user terminal 30 and the positional information of the virtual image. As shown in Figure 8, the composite image information output unit 26 determines, for example, based on the position of the user terminal 30 and the imaging direction X, that the coordinate information of virtual images A to C exists in the imaging direction X.

9 shows an example of a composite image output from the position identification device 20 to the user terminal 30, displayed on the display of the user terminal 30. As shown in FIG. 9, the composite image information output unit 26 generates a composite image by superimposing virtual images A to C (e.g., the position image; hereinafter, also referred to as pins) at positions in the captured image corresponding to objects A to C present in the imaging direction X. The pins A to C may be superimposed, for example, based on the distance between the position of the user terminal 30 and the coordinate information of the pins A to C, with the pins farther from the user terminal 30 being smaller than the other pins. On the other hand, the pins A to C may be superimposed, for example, based on the distance between the position of the user terminal 30 and the coordinate information of the pins A to C, with the pins closer to the user terminal 30 being larger than the other pins. The outer edges of the pins may be colored, for example, according to the distance between the position of the user terminal 30 and the coordinate information of the pins A to C.

Furthermore, as shown in FIG. 9, the detailed information image is specifically displayed in a window section (Window) 1. In the figure, the window section 1 is displayed below the captured image, but this is merely an example and is not limiting. For example, when the present invention is applied to an exhibition, a detailed information image generated based on detailed information about the exhibited products and services, the company of the booth, etc. is displayed in the window section 1.

According to this modified example, since the position of the user terminal can be identified more accurately, the virtual image can be displayed at a more accurate position using AR (Augmented Reality) technology. Note that this modified example is merely an example, and the map information generating device 10 and the map information generating method of the first embodiment and the position identifying device 20 and the position identifying method of the second embodiment are not limited to application to AR technology.

[Embodiment 3]
The program of this embodiment is a program for causing a computer to execute each step of the method of the present invention as a procedure. In the present invention, the "procedure" may be read as "processing". The program of this embodiment may be recorded, for example, in a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples thereof include a read-only memory (ROM), a hard disk (HD), and an optical disk.

The present invention has been described above with reference to the embodiments, but the present invention is not limited to the above embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments may be described as follows, but are not limited to the following supplementary notes.
(Appendix 1)
The apparatus includes an image acquisition unit, a three-dimensional model generation unit, and a map information generation unit,
The image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
The map information generating unit corresponds positions of the point cloud data in the first three-dimensional model and positions of the point cloud data in the second three-dimensional model, and generates map information of the specific area.
(Appendix 2)
2. The map information generating device according to claim 1, wherein the three-dimensional image information includes a panoramic image of the specific area captured three-dimensionally.
(Appendix 3)
Further, a feature point extraction unit is included,
The feature point extraction unit extracts feature points from the two-dimensional image information by machine learning,
3. The map information generating device according to claim 1, wherein the map information generating unit generates the map information to which the feature points are linked.
(Appendix 4)
The present invention includes a map information generating unit, a user terminal information acquiring unit, and an identifying unit,
The map information generating unit is a map information generating device according to any one of Supplementary Note 1 to 3,
The user terminal information acquisition unit acquires a captured image and sensor information from a user terminal,
The identification unit identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 5)
The map information generating unit is a map information generating device according to claim 3,
The feature point extraction unit extracts feature points from the captured image by machine learning,
The identification unit identifies a plurality of candidate points as the location of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
The position identification device according to claim 4, further comprising: identifying a position of the user terminal from the candidate points; and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 6)
The method includes an image acquisition step, a three-dimensional model generation step, and a map information generation step,
The image acquisition step acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generating step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
The map information generating method, wherein the map information generating step corresponds positions of point cloud data in the first three-dimensional model and point cloud data in the second three-dimensional model to generate map information of the specific area.
(Appendix 7)
7. The map information generating method according to claim 6, wherein the three-dimensional image information includes a panoramic image of the specific area captured three-dimensionally.
(Appendix 8)
Further, a feature point extraction step is included,
The feature point extraction step extracts feature points from the two-dimensional image information by machine learning,
The map information generating method according to claim 6 or 7, wherein the map information generating step generates the map information to which the feature points are linked.
(Appendix 9)
The method includes a map information generating step, a user terminal information acquiring step, and a specifying step,
The map information generating step is a step of executing each step of the map information generating method according to any one of Supplementary Notes 6 to 8,
The user terminal information acquisition step acquires a captured image and sensor information from a user terminal,
A position identification method in which the identification process identifies the position of the user terminal based on the point cloud data of the map information and the captured image, and identifies the imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 10)
The map information generating step is a step of executing each step of the map information generating method described in Supplementary Note 8,
The feature point extraction step includes extracting feature points from the captured image by machine learning,
The specifying step includes specifying a plurality of candidate points as the position of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
10. The method of claim 9, further comprising: identifying a position of the user terminal from the candidate points; and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 11)
A program for causing a computer to execute procedures including an image acquisition procedure, a three-dimensional model generation procedure, and a map information generation procedure;
The image acquisition step acquires three-dimensional image information and two-dimensional image information of a specific area,
the three-dimensional model generation step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
The map information generating step associates positions of point cloud data in the first three-dimensional model with positions of point cloud data in the second three-dimensional model, and generates map information of the specific area.
(Appendix 12)
12. The program according to claim 11, wherein the three-dimensional image information includes a full-surround panoramic image captured three-dimensionally within the specific area.
(Appendix 13)
Further, a feature point extraction step is included,
The feature point extraction step includes extracting feature points from the two-dimensional image information by machine learning;
13. The program according to claim 11, wherein the map information generating step generates the map information to which the feature points are linked.
(Appendix 14)
A program for causing a computer to execute procedures including a map information generating procedure, a user terminal information acquiring procedure, and a specifying procedure;
The map information generating step is a step of executing each step of a program according to any one of Supplementary Notes 11 to 13,
The user terminal information acquisition step acquires a captured image and sensor information from a user terminal;
The identification step identifies a position of the user terminal based on the point cloud data of the map information and the captured image, and identifies an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 15)
The map information generating procedure is a procedure for executing each procedure of the program described in Supplementary Note 13,
The feature point extraction step includes extracting feature points from the captured image by machine learning;
the step of identifying a plurality of candidate points as the location of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
The program according to claim 14, further comprising: identifying a position of the user terminal from the candidate points; and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information.
(Appendix 16)
A computer-readable recording medium having a program according to any one of appendices 11 to 15 recorded thereon.

The present invention makes it possible to generate more accurate map information. For this reason, the present invention is particularly useful in fields that use AR technology, such as exhibitions and games.

10 Map information generating device 11 Image acquisition unit 12 Three-dimensional model generating unit 13 Map information generating unit 14 Feature point extraction unit 20 Position identification device 21 Map information generating unit 22 User terminal information acquisition unit 23 Identification unit 24 Virtual image generating unit 25 Map information management unit 26 Synthetic image information output unit 30 User terminal 101 Central processing unit 102 Memory 103 Bus 104 Storage device 105 Input device 106 Display device 107 Communication device 108 Imaging device 109 Sound input/output device

Claims (7)

The present invention includes a map information generating unit, a virtual image generating unit, a map information managing unit, a user terminal information acquiring unit, a specifying unit, and a composite image information output unit,
the map information generation unit includes an image acquisition unit, a three-dimensional model generation unit, and an area map information generation unit;
The image acquisition unit acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generation unit generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
the area map information generation unit associates positions of the point cloud data in the first three-dimensional model with positions of the point cloud data in the second three-dimensional model to generate map information of the specific area;
The virtual image generator generates a virtual image relating to the specific area,
the virtual image is an image to be superimposed on a local captured image captured by a user terminal within the specific area,
the map information management unit manages the map information, the virtual image, and each piece of coordinate information of the virtual image in association with each other;
the user terminal information acquisition unit acquires a captured image and sensor information from the user terminal;
the identification unit identifies a position of the user terminal based on the point cloud data of the map information and the captured image, and identifies an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information;
The composite image information output unit outputs a composite image in which the virtual image is superimposed on the captured image . The position specifying device according to claim 1 , wherein the three-dimensional image information includes a panoramic image obtained by capturing an entire circumference of the specific area three-dimensionally. Further, a feature point extraction unit is included,
The feature point extraction unit extracts feature points from the two-dimensional image information by machine learning,
The position specifying device according to claim 1 , wherein the map information generating unit generates the map information to which the feature points are linked. The feature point extraction unit extracts feature points from the captured image by machine learning,
The identification unit identifies a plurality of candidate points as the location of the user terminal based on feature points of the two-dimensional image information and feature points of the captured image;
The position specifying device according to claim 3 , further comprising: a position specifying unit that specifies a position of the user terminal from the candidate points; and an imaging direction of the user terminal is specified based on the point cloud data of the map information and the sensor information. The method includes a map information generating step, a virtual image generating step, a map information managing step, a user terminal information acquiring step, a specifying step, and a composite image information output step,
The map information generating step includes an image acquiring step, a three-dimensional model generating step, and an area map information generating step,
The image acquisition step acquires three-dimensional image information and two-dimensional image information obtained by capturing an image of a specific area,
the three-dimensional model generating step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
the area map information generating step includes: associating positions of the point cloud data in the first three-dimensional model with positions of the point cloud data in the second three-dimensional model to generate map information of the specific area;
The virtual image generating step generates a virtual image relating to the specific area,
the virtual image is an image to be superimposed on a local captured image captured by a user terminal within the specific area,
The map information management step manages the map information, the virtual image, and each piece of coordinate information of the virtual image in association with each other,
The user terminal information acquisition step acquires a captured image and sensor information from the user terminal,
The identification step includes identifying a position of the user terminal based on the point cloud data of the map information and the captured image, and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information;
The composite image information output step outputs a composite image in which the virtual image is superimposed on the captured image.
Location information identification method . Further, a feature point extraction step is included,
The feature point extraction step extracts feature points from the two-dimensional image information by machine learning,
The position information specifying method according to claim 5 , wherein the map information generating step generates the map information to which the feature points are linked. A program for causing a computer to execute procedures including a map information generating procedure, a virtual image generating procedure, a map information managing procedure, a user terminal information acquiring procedure, a specifying procedure, and a composite image information output procedure ;
the map information generating step includes an image acquiring step, a three-dimensional model generating step, and an area map information generating step;
The image acquisition step acquires three-dimensional image information and two-dimensional image information of a specific area,
the three-dimensional model generation step generates a first three-dimensional model based on the three-dimensional image information and a second three-dimensional model based on the two-dimensional image information;
the first three-dimensional model and the second three-dimensional model are models showing a configuration within the specific area and include point cloud data having shape information;
the area map information generating step includes associating positions of point cloud data in the first three-dimensional model with positions of point cloud data in the second three-dimensional model to generate map information of the specific area;
The virtual image generation step generates a virtual image relating to the specific area;
the virtual image is an image to be superimposed on a local captured image captured by a user terminal within the specific area,
the map information management step includes managing the map information, the virtual image, and each piece of coordinate information of the virtual image in association with each other;
The user terminal information acquisition step acquires a captured image and sensor information from the user terminal;
The identification step includes identifying a position of the user terminal based on the point cloud data of the map information and the captured image, and identifying an imaging direction of the user terminal based on the point cloud data of the map information and the sensor information;
The composite image information output step includes outputting a composite image in which the virtual image is superimposed on the captured image .

Images