JP7611545B2 - Information processing system and information processing method - Google Patents

Description

The present invention relates to an information processing system and information processing method for displaying augmented reality images.

Conventionally, a system is known in which an augmented reality image to be provided to an information terminal is identified based on feature points of a subject contained in a captured image generated by an information terminal by capturing an image of the subject, and the augmented reality image is superimposed on the captured image and displayed on a display unit (see, for example, Patent Document 1).

JP 2019-175384 A

There is a need to display the same augmented reality image on multiple information terminals at the same time so that multiple users of multiple information terminals in the same space can view the same augmented reality image. For example, when multiple users are playing a game while capturing an image of a subject in the same space, an augmented reality image of a common character may be displayed on the information terminal used by each user. With conventional technology, the augmented reality image is displayed only on the information terminal that identifies the feature points of the subject, making it difficult to display the same augmented reality image on multiple information terminals at the same time.

Therefore, the present invention has been made in consideration of these points, and aims to make it possible to display the same augmented reality image at the same time on multiple information terminals.

The information processing system of the first aspect of the present invention comprises one or more information terminal groups including a plurality of information terminals that display an augmented reality image within a space of the same coordinate system, and an information processing device that provides the augmented reality image to the information terminals, and a first information terminal of the plurality of information terminals has a first display processing unit that displays a first augmented reality image on a display unit by superimposing the first augmented reality image on an image of a space viewed by a user of the first information terminal, a first operation receiving unit that creates display coordinate data indicating coordinates within the space when the first augmented reality image is displayed on the display unit in response to receiving an operation for displaying the first augmented reality image on the display unit, and a first transmission processing unit that transmits the display coordinate data to the information processing device, The information processing device has a coordinate receiving unit that receives the display coordinate data transmitted by the first information terminal and registers the received display coordinate data in an in-memory database, and a coordinate transmitting unit that transmits the display coordinate data to a second information terminal of the multiple information terminals that is different from the first information terminal in response to the display coordinate data being registered in the in-memory database, and the second information terminal has a second reception processing unit that receives the display coordinate data transmitted by the coordinate transmitting unit, and a second display processing unit that displays a second augmented reality image at a position on the display unit that corresponds to the coordinates in the space indicated by the display coordinate data, superimposed on an image of the space viewed by a user of the second information terminal.

The information processing device may have a plurality of in-memory databases corresponding to the plurality of spaces, and the coordinate receiving unit may register the display coordinate data in the in-memory database corresponding to the space that includes the first information terminal that transmitted the display coordinate data.

The information processing device may have a plurality of in-memory databases corresponding to the plurality of spaces, and the coordinate transmission unit may transmit to the second information terminal the display coordinate data registered in the in-memory database corresponding to the space in which the second information terminal is included.

The information processing device may further have a storage unit that stores multiple terminal identification information for identifying the multiple information terminals included in the spaces in association with each of the multiple spaces, and the coordinate transmission unit may transmit the display coordinate data to the second information terminal having second terminal identification information associated with the space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data by referring to the storage unit.

The first operation reception unit may create operation content data indicating the type of the first augmented reality image based on the received operation, the first transmission processing unit may further transmit the operation content data to the information processing device, and the coordinate transmission unit may transmit data corresponding to the second augmented reality image corresponding to the operation content data or data for identifying the second augmented reality image to the second information terminal in response to the operation content data being registered in the in-memory database.

The first information terminal further includes a device storage unit that stores a plurality of spatial data indicating feature points of a subject included in a plurality of images that have been created in advance by imaging the space from a plurality of positions and orientations within the space in association with the positions and orientations within the space, and a detailed position identification unit that identifies, among the plurality of spatial data stored in the device storage unit, a position and orientation corresponding to spatial data that is most similar to feature point group data indicating feature points of a subject included in an image created by the first information terminal by imaging the space, and the first operation reception unit may create the display coordinate data based on the relationship between the position and orientation identified by the detailed position identification unit and the position of the received operation.

The first operation reception unit creates display orientation data indicating the orientation of the first augmented reality image when the first augmented reality image is displayed on the display unit based on the relationship between the position and orientation identified by the detailed position identification unit and the position of the accepted operation, the first transmission processing unit further transmits the display orientation data to the information processing device, and the coordinate transmission unit may transmit the display orientation data to the second information terminal in response to the display orientation data being registered in the in-memory database.

The first information terminal may further include a spatial data acquisition unit that acquires the plurality of spatial data from the information processing device in response to the first operation acceptance unit accepting an operation for acquiring the plurality of spatial data from the information processing device, and stores the acquired plurality of spatial data in the terminal storage unit.

The first information terminal may further include a position detection unit that detects the position of the first information terminal, and a spatial data acquisition unit that acquires the plurality of spatial data corresponding to the position detected by the position detection unit from the information processing device and stores the plurality of acquired spatial data in the terminal storage unit.

The information processing device may further include a device storage unit that stores the first information terminal and the second information terminal that belong to a space of the same coordinate system as the multiple information terminals that display an augmented reality image within that space.

The information processing method of the second aspect of the present invention includes the steps of: creating display coordinate data indicating coordinates within the space when the first augmented reality image is displayed on the display unit of a first information terminal belonging to a predetermined space in response to receiving an operation for displaying a first augmented reality image on the display unit of the first information terminal superimposed on an image of the space viewed by a user of the first information terminal; transmitting the display coordinate data to an information processing device by the first information terminal; registering the display coordinate data received from the first information terminal in an in-memory database by the information processing device; transmitting the display coordinate data to a second information terminal belonging to the space different from the first information terminal in response to the display coordinate data being registered in the in-memory database by the information processing device; receiving the display coordinate data transmitted by the information processing device by the second information terminal; and displaying a second augmented reality image corresponding to the first augmented reality image at a position on the display unit of the second information terminal corresponding to the coordinates within the space viewed by the user of the second information terminal indicated by the display coordinate data by the second information terminal.

The present invention has the effect of enabling multiple information terminals to display the same augmented reality image at the same time.

FIG. 1 is a diagram illustrating an overview of an information processing system. FIG. 2 is a diagram for explaining a basic operation of the information processing system. FIG. 11 is a diagram for explaining screens displayed on a plurality of information terminals. FIG. 4 is a diagram showing feature point group data corresponding to a subject in a first space. 1 is a diagram showing an overview of a process in which an information terminal acquires spatial data corresponding to a space to which the information terminal belongs. FIG. 13 is a diagram illustrating an overview of a process for updating spatial data. FIG. 2 is a diagram showing a configuration of an information terminal. FIG. 1 is a diagram illustrating a configuration of an information processing device. FIG. 2 is a diagram showing the configuration of a computer included in a computer group. 11 is a sequence diagram showing a flow until a plurality of information terminals belonging to the same space acquire spatial data. FIG. 11 is a sequence diagram showing a flow until a plurality of information terminals belonging to different spaces acquire spatial data. FIG. 11 is a sequence diagram showing a process flow in which a plurality of information terminals belonging to the same space display an AR image. FIG. 11 is a sequence diagram showing a processing flow in which the information terminal registers feature point group data as spatial data. FIG.

[Overview of Information Processing System S]
(System Configuration)
1 is a diagram showing an overview of an information processing system S. The information processing system S is a system for displaying an augmented reality image (hereinafter, referred to as an "AR image") on a plurality of information terminals 1. The information processing system S includes a plurality of information terminals 1, an information processing device 2, a computer group 3, and a storage device 4.

The information terminal 1 is a terminal having an imaging function, such as a smartphone, tablet, or glasses-type terminal. A user using the information terminal 1 can move to any location. FIG. 1 shows a state in which multiple information terminals 1a (1a-1 to 1a-m, m is a natural number) exist in a first space, and multiple information terminals 1b (1b-1 to 1b-n, n is a natural number) exist in a second space. The first space and the second space correspond to areas set to display the same AR image on the information terminals 1 of users in the same space, such as, for example, a room separated by a wall, a floor in a building, a building, or a town. Multiple information terminals 1 displaying the same AR image in a space of the same coordinate system constitute a group of information terminals belonging to that space.

The information processing device 2 is a server that provides various data to the information terminal 1. The information processing device 2 provides, for example, spatial data used by the information terminal 1 to identify its own position and orientation, and AR image data to be displayed on the information terminal 1. The spatial data includes a plurality of registered feature point group data created in advance, which is composed of feature point groups indicating the characteristics of one or more subjects that are visible when viewing the space from a plurality of positions in the space. In the spatial data, coordinate data indicating the position and orientation at which the space is viewed is associated with the plurality of registered feature point group data. In addition, pixel values indicating color (R value, G value, B value) and coordinates indicating a position in space may be associated with the plurality of feature points included in the registered feature point group data.

The computer group 3 executes a matching process to identify the space to which the information terminal 1 belongs. The computer group 3 acquires captured image data generated by the information terminal 1, or feature point cloud data indicating feature points of the subject contained in the captured image data, and performs a matching process of the acquired captured image data or feature point cloud data with multiple spatial data composed of multiple registered feature point cloud data corresponding to multiple spaces, respectively, to identify the space to which the information terminal 1 belongs. The multiple feature points contained in the feature point cloud data may be associated with pixel values indicating color (R value, G value, B value).

The computer group 3 selects spatial data including a plurality of registered feature point group data corresponding to the space to which the information terminal 1 belongs, and provides the selected spatial data to the information terminal 1. As will be described in detail later, the computer group 3 has a plurality of computers 30 that store spatial data corresponding to each of the plurality of spaces, and the plurality of computers 30 execute matching processing in parallel to identify the space to which the information terminal 1 belongs in a short period of time.

The storage device 4 is a device that stores spatial data corresponding to various spaces. For example, the storage device 4 stores spatial data corresponding to many spaces around the world, and provides each computer 30 with the spatial data corresponding to each of the computers 30 in the computer group 3 out of the multiple spatial data.

(Basic operation)
2 is a diagram for explaining the basic operation of the information processing system S. In FIG. 2, the flow of the operation is shown in which, in response to the operation of a user U1 using an information terminal 1a-1 present in the first space, the same AR image is displayed superimposed on an image of the space viewed by a user U1 of the information terminal 1a-1 and a user U2 of the information terminal 1a-2 present in the same first space. Since the information terminals 1a-1 and 1a-2 face different directions at different positions in the first space, the same AR image is displayed in different directions at different positions on the screens being displayed by each of them.

The image of the space that user U1 and user U2 are viewing is, for example, a captured image generated by information terminal 1 capturing an image of the space in which user U1 and user U2 are present. When information terminal 1 is a glasses-type terminal, the image of the space that user U1 and user U2 are viewing may be an image that is seen by the eyes of user U1 and user U2 due to light that is incident on the eyes of user U1 and user U2 through a transparent lens of the glasses-type terminal. In the following explanation, a case where the image of the space that user U1 and user U2 are viewing is a captured image will be mainly described as an example, but the following configuration and processing are also applied when the image of the space that user U1 and user U2 are viewing is an image that is seen by the eyes of user U1 and user U2 through a lens.

The information terminal 1 acquires from the information processing device 2 spatial data indicating the positions of a group of feature points of a subject visible to the user U at multiple positions in the space in which the user U is present, and stores the spatial data corresponding to the specified space at least while the user U is in the specified space. The group of feature points is a collection of multiple points indicating the characteristics of the shape of the subject, and is composed of multiple points on the contour line of the subject contained in the captured image data generated by capturing an image of the space. In the spatial data, a combination of position and orientation in a three-dimensional space defined by latitude, longitude, altitude, and direction is associated with the group of feature points data.

Figure 3 is a diagram for explaining the screens displayed on multiple information terminals 1. Figure 3(a) is a diagram showing the space photographed by information terminal 1a-1 and information terminal 1a-2 from above. The left-right direction in Figure 2(a) is the X direction, the up-down direction is the Y direction, and the depth direction is the Z direction.

In FIG. 3, it is assumed that user U1 using information terminal 1a-1 and user U2 using information terminal 1a-2 are located at the positions shown in FIG. 3(a). Subjects T1 to T4 in FIG. 3 are subjects that exist in the first space to which information terminal 1a-1 and information terminal 1a-2 belong. Image A (person), image B (bird), and image C (arrow) in FIG. 3 are AR images displayed on information terminal 1a-1 and information terminal 1a-2.

Figure 3(b) shows an example of a screen displayed on information terminal 1a-1. Figure 3(b) shows a state in which user U1 has performed an operation to shoot image C (arrow) from image A (person) toward image B (bird) using user U1's hand H. Figure 3(c) shows an image displayed on information terminal 1a-2 when the operation shown in Figure 3(b) is performed on information terminal 1a-1. In Figure 3(c), image C is displayed in a different position and orientation from that in Figure 3(b). By information terminals 1a-1 and 1a-2 displaying such images, users U1 and U2 can view the same AR image from different orientations on the screens of the information terminals 1 that they each use, superimposed on subjects that are visible at different positions in the same space.

The process flow for the information terminal 1a-1 and information terminal 1a-2 to display the image shown in FIG. 3 will be described below with reference to FIG. 2. The information terminal 1 creates feature point group data based on captured image data generated by the built-in camera by executing application software such as a game or a tourist guide that has a function for displaying AR images.

Figure 4 is a diagram showing feature point cloud data corresponding to subjects T1 to T4 in the first space shown in Figure 3. Figure 4(a) shows the feature point cloud data of subjects T1 to T4 photographed by information terminal 1a-1, and Figure 4(b) shows the feature point cloud data of subjects T1 to T4 photographed by information terminal 1a-2. In this way, the positions of the multiple feature points included in the feature point cloud data differ depending on the position and orientation at which information terminal 1 photographs.

The information terminal 1 compares the created feature point cloud data with a plurality of registered feature point cloud data included in pre-stored spatial data, and identifies, from the plurality of registered feature point cloud data, the registered feature point cloud data that is most similar to the created feature point cloud data. The information terminal 1 identifies the position and orientation corresponding to the identified registered feature point cloud data, thereby identifying the position and orientation of the information terminal 1.

When the information terminal 1a-1 receives a user operation that triggers the display of an AR image, it notifies the information processing device 2 of operation data indicating the content of the operation and position coordinate data indicating the identified position and orientation, as shown in FIG. 2. The information terminal 1a-1 acquires an AR image corresponding to the position and orientation from the information processing device 2, and displays the acquired AR image, for example, as shown in FIG. 3(b).

The information processing device 2 has multiple memory databases corresponding to multiple spaces, and in response to the position coordinate data notified from the information terminal 1a-1 being written to the memory database corresponding to the space to which the information terminal 1a-1 belongs, the information processing device 2 transmits an AR image and display coordinate data indicating the position to display the AR image and the direction in which to move the image to the other information terminal 1, information terminal 1a-2, which belongs to the same space. For example, as shown in FIG. 3(c), the information terminal 1a-2 displays an AR image corresponding to the position and direction indicated by the received display coordinate data.

In a system that displays AR images, such as in this embodiment, a high-speed response is required, so it is desirable for the database to be an in-memory database that is constructed in main memory and can be accessed more quickly than a database stored in a disk storage mechanism. In the following explanation, an example is given in which the database is an in-memory database.

(Provision of spatial data)
5 is a diagram showing an overview of a process in which the information terminal 1 acquires spatial data corresponding to the space to which the information terminal 1 belongs. The information terminal 1 acquires spatial data when triggered by the start of a specific application software, the user U logging in to a specific site, or the information terminal 1 moving to a specific position.

For example, when the information terminal 1 detects that a specific application software has been launched, it creates feature point cloud data based on captured image data generated by a built-in camera. The information terminal 1 transmits startup information indicating that the application software has been launched and the created feature point cloud data to the information processing device 2. In response to receiving the startup information, the information processing device 2 transmits the received feature point cloud data to the computer group 3.

The computer group 3 inputs the received feature point group data to multiple computers 30, and the multiple computers 30 execute a matching process. Among the multiple computers 30, a computer 30 that detects that the pattern of the feature point group indicated by the received feature point group data matches the pattern of the registered feature point group indicated by the spatial data stored in itself notifies the information processing device 2 of the match and transmits the spatial data stored in itself to the information processing device 2. The information processing device 2 transmits the received spatial data to the information terminal 1.

The information terminal 1 may transmit approximate location data indicating the approximate location detected by the GPS receiver together with the startup information to the information processing device 2. In this case, the information processing device 2 transmits feature point group data to the computer 30 corresponding to the space including the location indicated by the received approximate location data, thereby reducing the number of computers 30 that perform the matching process, thereby reducing the load on the computer group 3.

(Spatial data update)
Objects such as buildings, items, and plants in a space change over time. Therefore, the spatial data stored in the storage device 4 does not necessarily correspond to the latest state of the space. In addition, there may be spaces for which no spatial data has been created. However, it is not easy to collect spatial data every time the state of the space changes. Therefore, the information processing system S uses a large amount of feature point group data created by the information terminals 1 to update the spatial data to the latest state.

Figure 6 is a diagram showing an overview of the process of updating spatial data. When the information terminal 1 does not include registered feature point group data that matches the feature point group data created by itself among the multiple registered feature point group data included in the spatial data acquired via the information processing device 2, the information terminal 1 determines that there is an error in the spatial data or that there is no spatial data for the space to which the information terminal 1 belongs. In this case, the information terminal 1 transmits an update request to the information processing device 2 requesting that the registered feature point group data be updated, together with the feature point group data created by itself and position data indicating its own position. The position data is, for example, data indicating latitude, longitude, altitude, and direction.

It is desirable for the position data to be as accurate as possible. Therefore, the information terminal 1 calculates a position corresponding to the feature point group data to be sent together with the update request, for example, based on the distance and direction of movement from the position where the registered feature point group data included in the spatial data most recently matched the feature point group data created by the information terminal 1. The information terminal 1 may send the update request on the condition that the distance from the position where the registered feature point group data included in the spatial data most recently matched the feature point group data created by the information terminal 1 is within a predetermined threshold. The predetermined threshold is determined based on at least one of the accuracy of the position indicated by the spatial data or the accuracy with which the information terminal 1 can identify the distance and direction of movement.

When the information processing device 2 receives an update request, it transmits the received feature point cloud data and position data to the storage device 4. When the information processing device 2 receives feature point cloud data corresponding to the same position and orientation from multiple information terminals 1, it transmits the feature point cloud data that is considered to have higher accuracy. Feature point cloud data that is considered to have higher accuracy is feature point cloud data that includes a larger number of feature points, or feature point cloud data transmitted from an information terminal 1 that uses a sensor with higher accuracy to identify the position, etc.

When the storage device 4 receives the feature point cloud data and position data from the information terminal 1, it associates the feature point cloud data with the position and orientation (i.e., latitude, longitude, altitude and direction) indicated by the received position data and stores it as registered feature point cloud data included in the spatial data. By the storage device 4 updating the registered feature point cloud data included in the spatial data in this manner, the information terminal 1 can identify its own position and orientation using the spatial data corresponding to the latest space, and can display the AR image in an appropriate position.

[Configuration of each part]
(Configuration of information terminal)
7 is a diagram showing the configuration of the information terminal 1. The information terminal 1 has a communication unit 11, an operation unit 12, a position detection unit 13, an imaging unit 14, a display unit 15, a storage unit 16, and a control unit 17. The control unit 17 has a transmission processing unit 171, a reception processing unit 172, an operation acceptance unit 173, an approximate position identification unit 174, an image processing unit 175, a detailed position identification unit 176, a display processing unit 177, and a spatial data acquisition unit 178.

The communication unit 11 has a communication interface for connecting to a communication network. The communication unit 11 transmits and receives various data to and from the information processing device 2 via a wireless communication network such as a mobile wireless communication network or Wi-Fi (registered trademark). The communication unit 11 transmits data input from the transmission processing unit 171, and inputs data received via the communication network to the reception processing unit 172.

The operation unit 12 has a device for detecting an operation by a user U of the information terminal 1. The operation unit 12 has, for example, a touch panel, a trackball, or a mouse. When the operation unit 12 is a touch panel, the operation unit 12 is provided on top of the display unit 15. The operation unit 12 notifies the operation reception unit 173 of operation data indicating at least either the position where the operation was performed or the amount of operation.

The position detection unit 13 has a device for detecting the approximate position of the information terminal 1. The position detection unit 13 has, for example, a GPS (Global Positioning System) receiver, and notifies the approximate position identification unit 174 of position data indicating latitude and longitude detected based on radio waves received from GPS satellites. The position detection unit 13 has at least one of an altimeter, a compass, or a gyro sensor, and may notify the approximate position identification unit 174 of the altitude of the information terminal 1 and the direction in which the information terminal 1 is facing as position data.

The imaging unit 14 is a device, such as a camera, that captures an image of a subject around the information terminal 1 to generate captured image data. The imaging unit 14 generates captured image data by capturing an image of at least a partial area of the space to which the information terminal 1 belongs. The imaging unit 14 may generate captured image data continuously at a predetermined time interval (frame rate). The imaging unit 14 may also be a stereo camera that has multiple image sensors and generates distance data indicating the distance to the subject. In this case, the feature point group data may include information indicating the distance from the information terminal 1 to the subject corresponding to each feature point.

The display unit 15 is a device for displaying various types of information, such as a display. For example, the display unit 15 displays a screen for the user U of the information terminal 1 to operate, or displays an image based on captured image data generated by the imaging unit 14. The display unit 15 also displays an AR image provided by the information processing device 2, superimposed on an image of the space viewed by the user U (for example, a captured image).

The storage unit 16 is a terminal storage unit having storage media such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage unit 16 stores programs executed by the control unit 17. The storage unit 16 also temporarily stores image data displayed by the display unit 15, such as captured image data and AR image data. The storage unit 16 stores a plurality of feature point group data indicating feature point groups of a subject contained in a plurality of images created by capturing an image of a space from a plurality of positions and orientations in the space, in association with the positions and orientations in the space.

The control unit 17 has, for example, a CPU (Central Processing Unit). The control unit 17 executes the programs stored in the storage unit 16, thereby functioning as a transmission processing unit 171, a reception processing unit 172, an operation reception unit 173, an approximate position identification unit 174, an image processing unit 175, a detailed position identification unit 176, a display processing unit 177, and a spatial data acquisition unit 178.

The transmission processing unit 171 transmits various data to the information processing device 2 or other external devices via the communication unit 11. The transmission processing unit 171 transmits, for example, operation data received from the operation reception unit 173 to the information processing device 2. The operation data includes operation content data indicating the content of the operation, display coordinate data indicating the position of the operation (i.e., the position at which the AR image is displayed), and display orientation data indicating the direction in which the AR image is displayed. The operation data may further include speed data indicating the operation speed of the user U, which corresponds to the speed at which the AR image is moved.

When the user U performs an operation to display an AR image on the display unit 15, the transmission processing unit 171 transmits to the information processing device 2, together with the operation content data, display coordinate data indicating the coordinates in space when the AR image is displayed on the display unit 15, display orientation data, and speed data. In addition, the transmission processing unit 171 transmits the approximate position data received from the approximate position identification unit 174 to the information processing device 2.

The transmission processing unit 171 may transmit operation data using pub/sub messages for accessing the in-memory database in order to transmit and receive data asynchronously in real time with the in-memory database held by the information processing device 2.

The transmission processing unit 171 also acquires feature point cloud data composed of multiple feature points of the subject included in the captured image data from the image processing unit 175, and transmits the acquired feature point cloud data to the information processing device 2. The transmission processing unit 171 may transmit the captured image data received from the image processing unit 175 to the information processing device 2 instead of or together with the feature point cloud data. The transmission processing unit 171 writes the feature point cloud data, for example, by sending a Pub message to an in-memory database corresponding to the space to which the approximate position identified by the approximate position identification unit 174 belongs, among multiple in-memory databases stored in the information processing device 2.

In addition, when there is no registered feature point group data whose difference from the feature point group data is equal to or less than a threshold value among the multiple registered feature point group data stored in the memory unit 16, the transmission processing unit 171 transmits the feature point group data acquired from the image processing unit 175 to the information processing device 2 together with an update request to update the registered feature point group data.

The reception processing unit 172 receives various data from the information processing device 2 or other external devices via the communication unit 11. The reception processing unit 172 receives, for example, display coordinate data transmitted to the information processing device 2 by another information terminal 1 belonging to the same space.

When the information terminal 1a-1 transmits display coordinate data to the information processing device 2 in response to a user U1 using the information terminal 1a-1 performing an operation to display an AR image, the reception processing unit 172 of the information terminal 1a-2 receives the display coordinate data transmitted by a coordinate transmission unit 232 possessed by the information processing device 2, which will be described later. The reception processing unit 172 inputs the received display coordinate data to the detailed position identification unit 176. The reception processing unit 172 also receives spatial data corresponding to the space to which the information terminal 1 belongs from the information processing device 2. The reception processing unit 172 inputs the received spatial data to the spatial data acquisition unit 178.

The operation acceptance unit 173 identifies the content of the operation performed by the user U based on the operation data notified from the operation unit 12. The operation acceptance unit 173 creates operation content data indicating the type of AR image to be displayed based on the accepted operation. The operation acceptance unit 173 notifies the information processing device 2 of the content of the operation by inputting the created operation content data to the transmission processing unit 171. For example, in response to accepting an operation for displaying an AR image on the display unit 15 superimposed on an image of the space (e.g., a captured image), the operation acceptance unit 173 creates display coordinate data indicating the coordinates in the space when the AR image is displayed on the display unit 15.

The operation reception unit 173 may acquire detailed position data indicating the detailed position identified by the detailed position identification unit 176, and create display coordinate data based on the position and orientation of the information terminal 1 indicated by the acquired detailed position data and the position operated by the user U on the operation unit 12. In other words, the operation reception unit 173 may create display coordinate data based on the relationship between the position and orientation identified by the detailed position identification unit 176 and the position of the operation accepted by the operation reception unit 173.

In the example shown in FIG. 3B, the operation acceptance unit 173 creates display coordinate data including the X and Z coordinates of the position where the hand H touches the operation unit 12. When the position of the hand H moves, the operation acceptance unit 173 creates display coordinate data at a predetermined time interval. The operation acceptance unit 173 may create display coordinate data including speed data indicating the speed at which the hand H moves based on the display coordinate data created at multiple times. When the Y coordinate of the subject at the position where the hand H touches in FIG. 3B is known, such as when the imaging unit 14 is a stereo camera that can generate distance data, the operation acceptance unit 173 may create display coordinate data including the Y coordinate. The operation acceptance unit 173 inputs the created display coordinate data to the transmission processing unit 171.

The operation reception unit 173 may create display orientation data indicating the orientation of the AR image when the AR image is displayed on the display unit 15, based on the relationship between the position and orientation of the information terminal 1 identified by the detailed position identification unit 176 and the position of the operation accepted by the operation acceptance unit 173. The operation acceptance unit 173 inputs the display orientation data to the transmission processing unit 171.

The approximate position determination unit 174 generates approximate position data indicating the approximate position of the information terminal 1 based on the received radio waves. The approximate position determination unit 174 determines the approximate position of the information terminal 1 based on the position data input from the position detection unit 13. The approximate position data is data indicating a position with an accuracy corresponding to the accuracy of GPS, for example, position data with an error of 10 m or more.

The approximate position identification unit 174 inputs the identified position data to the transmission processing unit 171, thereby notifying the information processing device 2 of the approximate position of the information terminal 1. The approximate position identification unit 174 may periodically notify the information processing device 2 of the approximate position data, or may notify the information processing device 2 of the approximate position data when the user U performs a predetermined operation or when it becomes necessary to obtain registered feature point group data.

The image processing unit 175 creates feature point cloud data consisting of multiple feature points of the subject contained in the captured image data. Specifically, the image processing unit 175 first identifies pixels in the captured image data whose pixel value change from adjacent pixels is equal to or greater than a threshold, thereby identifying the contour line of the subject and the contour line of the uneven portion of the subject. The image processing unit 175 creates feature point cloud data consisting of multiple feature points on the identified contour line.

As in the example shown in FIG. 4, the image processing unit 175 may create feature point cloud data by identifying points where the direction of a contour line changes as feature points. The image processing unit 175 creates feature point cloud data every time new captured image data is input from the imaging unit 14. The image processing unit 175 inputs the created feature point cloud data to the detailed position identification unit 176. The image processing unit 175 inputs the feature point cloud data to the transmission processing unit 171 when it is necessary to acquire spatial data from the information processing device 2 (i.e. when it is necessary to identify the space to which the information terminal 1 belongs).

When the image processing unit 175 receives a notification that the detailed position identification unit 176 cannot identify the position and orientation of the information terminal 1 based on the registered feature point cloud data acquired from the information processing device 2, the image processing unit 175 inputs the created feature point cloud data to the transmission processing unit 171 in order to register the feature point cloud data in the storage device 4, and instructs the transmission processing unit 171 to transmit an update request to the information processing device 2 to update the registered feature point cloud data.

The detailed position identification unit 176 identifies the position and orientation corresponding to the registered feature point cloud data that is most similar to the feature point cloud data input from the image processing unit 175, among the multiple registered feature point cloud data received from the information processing device 2. The detailed position identification unit 176 identifies the position and orientation corresponding to the registered feature point cloud data that is most similar to the feature point cloud data indicating the feature point cloud of the subject included in the captured image data, among the multiple registered feature point cloud data stored in the storage unit 16.

Specifically, the detailed position identification unit 176 first compares the feature point cloud data input from the image processing unit 175 with a plurality of registered feature point cloud data included in the spatial data acquired from the information processing device 2 and stored in the storage unit 16. The detailed position identification unit 176 calculates a correlation value between the feature point cloud data input from the image processing unit 175 and each of the plurality of registered feature point cloud data included in the spatial data, and identifies the registered feature point cloud data for which the calculated correlation value is maximum.

The detailed position identification unit 176 identifies the position and orientation of the information terminal 1 based on coordinate data (e.g., data indicating latitude, longitude, altitude, and direction) associated with the identified registered feature point group data. In this way, the detailed position identification unit 176 can identify the position and orientation of the information terminal 1 with greater accuracy than, for example, the approximate position identification unit 174. The detailed position identification unit 176 identifies the position and orientation of the information terminal 1 with an accuracy of, for example, 1 mm. The detailed position identification unit 176 notifies the reception processing unit 172 and the display processing unit 177 of detailed position data indicating the identified position and orientation.

The display processing unit 177 causes various kinds of information or images to be displayed on the display unit 15. For example, the display processing unit 177 causes the display unit 15 to display an image based on captured image data generated by the imaging unit 14 capturing an image of the space to which the information terminal 1 belongs. The display processing unit 177 also determines the position and orientation for displaying the AR image based on the display coordinate data. The display processing unit 177 causes the AR image to be displayed by superimposing it on an image of the space viewed by the user U (for example, an image based on captured image data generated by the imaging unit 14 capturing an image of the space to which the information terminal 1 belongs) at a position on the display unit 15 corresponding to the coordinates in the space indicated by the display coordinate data.

When a user U performs an operation to display an AR image, the display processing unit 177 creates display coordinate data indicating the position and orientation at which the AR image is to be displayed, based on the detailed position data input from the detailed position identification unit 176 and the operation position data notified from the operation reception unit 173. When a user U of another information terminal 1 belonging to the same space as the information terminal 1 performs an operation to display an AR image, the display processing unit 177 acquires display coordinate data used to display the AR image corresponding to the operation from the information processing device 2.

As shown in FIG. 2, when information terminal 1a-1 and information terminal 1a-2 exist in the first space, the display processing unit 177 of information terminal 1a-1 displays an AR image at a position and orientation based on the display coordinate data notified from the operation reception unit 173. Meanwhile, the display processing unit 177 of information terminal 1a-2 displays an AR image at a position and orientation based on the display coordinate data notified from the information processing device 2. By operating the display processing units 177 of information terminals 1a-1 and 1a-2 in this manner, user U1 using information terminal 1a-1 and user U2 using information terminal 1a-2 can simultaneously view AR images at different positions and orientations depending on the position and orientation of information terminal 1.

The spatial data acquisition unit 178 acquires spatial data including registered feature point group data from the information processing device 2 via the communication unit 11 and the reception processing unit 172. In response to the operation acceptance unit 173 accepting an operation for acquiring multiple registered feature point group data from the information processing device (for example, an operation for starting up a specific application software, an operation for logging in to a specific site), the spatial data acquisition unit 178 acquires spatial data including multiple registered feature point group data corresponding to the space to which the information terminal 1 belongs from the information processing device, and stores the acquired spatial data in the information processing device 16.

The spatial data acquisition unit 178 notifies the detailed position identification unit 176 of the acquired plurality of registered feature point group data. The spatial data acquisition unit 178 notifies the detailed position identification unit 176 of the plurality of registered feature point group data via the storage unit 16, for example, by storing the acquired plurality of registered feature point group data in the storage unit 16.

(Configuration of information processing device 2)
8 is a diagram showing the configuration of the information processing device 2. The information processing device 2 has a communication unit 21, a storage unit 22, and a control unit 23. The control unit 23 has a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, a spatial data transmitting unit 236, a spatial data management unit 237, and a number management unit 238.

The communication unit 21 has a communication interface for connecting to a communication network. The communication unit 21 transmits and receives various data between the information terminal 1, the computer group 3, and the storage device 4.

The storage unit 22 is a device storage unit having a ROM, a RAM, and an SSD (Solid State Drive). The storage unit 22 stores a program executed by the control unit 23. The storage unit 22 also stores a first space database 221 and a second space database 222 as multiple in-memory databases corresponding to multiple spaces. The storage unit 22 stores, for example, multiple in-memory databases (for example, the first space database 221 and the second space database 222) that store display coordinate data or feature point group data corresponding to each of the multiple spaces (for example, the first space and the second space).

As described above, an in-memory database is a database constructed in main memory, and can be accessed more quickly than a database stored in a disk storage mechanism. In FIG. 8, the storage unit 22 illustrates a first space database 221 and a second space database 222, but the storage unit 22 stores databases corresponding to many more spaces.

The storage unit 22 stores a first information terminal and a second information terminal belonging to a space of the same coordinate system as multiple information terminals 1 displaying an AR image within the same space. The first information terminal is the information terminal 1 on which the user U performs an operation to display the AR image, and is information terminal 1a-1 in the example shown in FIG. 2. The second information terminal is another information terminal 1 belonging to the same space as the first information terminal, and is information terminal 1a-2 in the example shown in FIG. 2. The storage unit 22 stores multiple pieces of terminal identification information for identifying the multiple information terminals 1 included in the space, in association with each of the multiple spaces.

Specifically, in the in-memory database stored in the storage unit 22, space identification information for identifying the space to which each of the multiple information terminals 1 belongs is registered in association with terminal identification information for identifying each of the multiple information terminals 1. In addition, in the in-memory database, AR image identification information for identifying an AR image to be displayed on the information terminal 1 belonging to the space corresponding to the space identification information, and display coordinate data indicating the position and orientation at which the AR image is to be displayed are registered in association with the space identification information.

The multiple in-memory databases may store feature point cloud data received from information terminals 1 belonging to the respective corresponding spaces in order to transfer the feature point cloud data to the computer group 3. When the first space database 221 corresponding to the first space receives feature point cloud data from one or more information terminals 1 belonging to the first space, it transfers the received feature point cloud data to a computer 30 that analyzes whether the feature point cloud data matches registered feature point cloud data corresponding to the first space. The in-memory database may relay the feature point cloud data between the information terminal 1 and the computer group 3 based on a pub/sub message sent from the information terminal 1 or the computer group 3 without going through the control unit 23.

The control unit 23 has, for example, a CPU. The control unit 23 executes the programs stored in the storage unit 22 to function as a coordinate receiving unit 231, a coordinate transmitting unit 232, a terminal data receiving unit 233, a feature point group transmitting unit 234, an information receiving unit 235, a spatial data transmitting unit 236, a spatial data management unit 237, and a number management unit 238.

The coordinate receiving unit 231 receives operation content data and display coordinate data transmitted by the information terminal 1 in response to an operation by a user U using the information terminal 1, and registers the received operation content data and display coordinate data in an in-memory database. The coordinate receiving unit 231 registers the display coordinate data in an in-memory database corresponding to the space that includes the information terminal 1 that transmitted the display coordinate data. For example, when the coordinate receiving unit 231 receives display coordinate data from an information terminal 1a-1 that belongs to the first space, it stores the display coordinate data in the first space database 221.

In response to the display coordinate data being registered in the in-memory database by the coordinate receiving unit 231, the coordinate transmitting unit 232 transmits the display coordinate data to a second information terminal, different from the first information terminal, among the multiple information terminals 1. The coordinate transmitting unit 232 transmits the display coordinate data registered in the in-memory database corresponding to the space in which the second information terminal is included to the second information terminal.

By referring to the storage unit 22, the coordinate transmission unit 232 transmits the display coordinate data to a second information terminal having second terminal identification information associated with a space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data. The coordinate transmission unit 232 may transmit the display orientation data to the second information terminal in response to the display orientation data transmitted by the first information terminal being registered in the in-memory database. As a result, multiple information terminals 1 belonging to the same space share the position and display orientation at which the AR image is displayed.

In addition, in response to the operation content data received from the first information terminal being registered in the in-memory database, the coordinate transmission unit 232 transmits data corresponding to the second AR image corresponding to the operation content data or data for identifying the second AR image to the second information terminal. The second AR image is an image corresponding to the first AR image, showing the same object as the first AR image, but with a different orientation or size. This allows multiple information terminals 1 belonging to the same space to share the same AR image.

The coordinate transmission unit 232 transmits the display coordinate data and the AR image stored in the first space database 221 to the information terminal 1a-2. If the coordinate transmission unit 232 has already transmitted an AR image to the information processing device 2, the coordinate transmission unit 232 may transmit AR image identification information for identifying an AR image to be displayed on the information terminal 1a-2 from among the multiple AR images already transmitted, instead of transmitting the AR image.

The terminal data receiving unit 233 receives feature point cloud data from the information terminal 1 via the communication unit 21. The terminal data receiving unit 233 also receives approximate position data indicating the approximate position of the information terminal 1 from the information terminal 1 that transmitted the feature point cloud data. The terminal data receiving unit 233 notifies the feature point cloud transmission unit 234 of the received feature point cloud data and approximate position data. The terminal data receiving unit 233 may store the feature point cloud data and approximate position data in the storage unit 22, and notify the feature point cloud data and approximate position data to the feature point cloud transmission unit 234 via the storage unit 22.

The feature point group transmission unit 234 transmits the feature point group data received from the information terminal 1 to the group of computers 3. For example, as shown in FIG. 5, when the terminal data receiving unit 233 receives feature point group data transmitted by the information terminal 1 together with startup information, the feature point group transmission unit 234 transmits the feature point group data to the group of computers 3 together with a message requesting spatial data.

When the feature point group transmission unit 234 acquires the approximate position data from the terminal data receiving unit 233, it may select one or more computers 30 corresponding to a space including the approximate position indicated by the approximate position data from the multiple computers 30 included in the computer group 3, and transmit the feature point group data to the selected one or more computers 30. In other words, the feature point group transmission unit 234 may transmit the feature point group data to a computer 30 associated with a space corresponding to the data to which the approximate position indicated by the approximate position data transmitted by the information terminal 1 together with the feature point group data belongs.

The feature point group transmission unit 234 may relay the feature point group data between the information terminal 1 and the computer group 3 via an in-memory database corresponding to each of the multiple spaces. The feature point group transmission unit 234 transmits the feature point group data to a computer 30 associated with a space corresponding to the in-memory database in which the feature point group data is written, among the multiple in-memory databases.

The information receiving unit 235 receives, from a computer 30 that has identified the space to which the information terminal 1 belongs, among the one or more computers 30 to which the feature point group transmitting unit 234 has transmitted feature point group data, predetermined information indicating that the space to which the information terminal 1 belongs has been identified. Specifically, the information receiving unit 235 receives, from one computer 30 among the one or more computers 30 to which the feature point group data has been transmitted, predetermined information indicating that the correlation value between any of the multiple registered feature point group data stored in the computer 30 and the feature point group data transmitted by the feature point group transmitting unit 234 is equal to or greater than a threshold value. In other words, the information receiving unit 235 receives, from the computer 30, predetermined information indicating that the feature point group data transmitted by the feature point group transmitting unit 234 is equivalent to the registered feature point group data of a subject included in the space for which the computer 30 is responsible for matching processing.

The information receiving unit 235 may acquire, together with the predetermined information, spatial data including a plurality of registered feature point cloud data corresponding to the space for which the computer 30 is responsible for the matching process. The information receiving unit 235 inputs the acquired spatial data to the spatial data transmitting unit 236 together with the terminal identification information of the information terminal 1 that created the feature point cloud data that the feature point cloud transmitting unit 234 transmitted to the computer group 3. The information receiving unit 235 may also acquire spatial identification information for identifying the space for which the computer 30 is responsible for the matching process, and store the spatial identification information in the storage unit 22 in association with the terminal identification information of the information terminal 1 that created the feature point cloud data that the feature point cloud transmitting unit 234 transmitted to the computer group 3.

The spatial data transmission unit 236 transmits to the information terminal 1 a plurality of registered feature point group data included in the spatial data corresponding to the computer 30 that transmitted the specified information. The spatial data transmission unit 236 transmits the plurality of registered feature point group data to the information terminal 1 corresponding to the terminal identification information acquired from the information reception unit 235.

The spatial data management unit 237 manages the spatial data stored in the computer group 3 or the storage device 4 to keep it up to date. The spatial data management unit 237 keeps the spatial data up to date, for example, by associating the feature point group data received from the information terminal 1 with a position in space and storing it in the storage device 4.

When the spatial data management unit 237 determines that the feature point group data received from the information terminal 1 is the latest feature point group data, it stores the feature point group data received from the information terminal 1 in the storage device 4 as registered feature point group data. When the spatial data management unit 237 receives feature point group data from multiple information terminals 1 together with an update request requesting that the registered feature point group data be updated, for example, the spatial data management unit 237 determines that the feature point group data based on captured image data generated at a later point in time is the latest feature point group data by, for example, referring to the timestamp included in the feature point group data.

When the device reception processing unit receives multiple feature point cloud data from multiple information terminals 1 that have transmitted detailed position data indicating the same position and orientation, the spatial data management unit 237 may store the feature point cloud data with the larger amount of data in the storage device 4 as registered feature point cloud data. The amount of data is, for example, the number of point clouds contained in the feature point cloud data. It is considered that the larger the number of point clouds, the more detailed characteristics of the subject belonging to the space are expressed in the feature point cloud data, and therefore, by the spatial data management unit 237 registering such feature point cloud data, the accuracy with which the information terminal 1 identifies the position and orientation using the registered feature point cloud data is improved.

The spatial data management unit 237 may store the feature point cloud data received from the information terminal 1 as registered feature point cloud data in the storage device 4 when the feature point cloud data associated with the position and orientation indicated by the detailed position data transmitted by the information terminal 1 together with the feature point cloud data is not stored in the storage device 4. The storage device 22 stores the latitude, longitude, altitude, and orientation corresponding to each of the multiple registered feature point cloud data stored in the storage device 4, and the spatial data management unit 237 stores the feature point cloud data in the storage device 4 when the latitude, longitude, altitude, and orientation indicated by the detailed position data are not stored in the storage device 22.

When registered feature point cloud data corresponding to the position and orientation indicated by the position data is stored in the storage device 4, the spatial data management unit 237 may store the feature point cloud data received from the information terminal 1 as registered feature point cloud data in the storage device 4, provided that the amount of data of the feature point cloud data received from the information terminal 1 is greater than the amount of data of the stored registered feature point cloud data.

The memory unit 22 stores the data amount of each of the multiple registered feature point cloud data stored in the storage device 4 together with the latitude, longitude, altitude and direction, and the spatial data management unit 237 compares the data amount of the feature point cloud data transmitted from the information terminal 1 with the data amount of the registered feature point cloud data stored in the memory unit 22 in association with the position and orientation indicated by the detailed position data transmitted from the information terminal 1. If, as a result of the comparison, the spatial data management unit 237 determines that the data amount of the feature point cloud data received from the information terminal 1 is greater, it stores the feature point cloud data received from the information terminal 1 in the storage device 4. By operating in this manner, the spatial data management unit 237 can prevent the registered feature point cloud data from deteriorating by updating the registered feature point cloud data.

The spatial data management unit 237 may store the feature point cloud data in the storage device 4 on the condition that the feature point cloud data received from the information terminal 1 includes data indicating a subject of a pre-registered shape. A subject of a pre-registered shape is, for example, a newly constructed building. By operating in this manner, the spatial data management unit 237 can quickly register registered feature point cloud data when it is known in advance that no registered feature point cloud data exists.

The number management unit 238 determines the number of computers 30 that will execute the matching process to identify the space to which the information terminal 1 belongs, based on preset conditions. The number management unit 238 changes the number of computers 30, for example, based on the amount of registered feature point cloud data stored in the storage device 4. The feature point cloud transmission unit 234 transmits the feature point cloud data received from the multiple information terminals 1 to the number of computers 30 determined by the number management unit 238. By operating in this manner, the number management unit 238 can prevent an increase in the time required for the matching process, even if the amount of registered feature point cloud data increases over time.

The number management unit 238 may change the number of the multiple computers 30 based on the time required from when the feature point group transmission unit 234 transmits the feature point group data to the multiple computers 30 until it acquires the specified information indicating that the space to which the information terminal 1 belongs has been identified (i.e., the time required for the matching process). If the time required from when the feature point group transmission unit 234 transmits the feature point group data to the multiple computers 30 until it acquires the specified information is long, it is considered that there is a shortage of computers 30. Therefore, when the time reaches or exceeds a threshold, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby preventing the time required for the matching process from becoming too long.

However, when the number of information terminals 1 from which the information processing device 2 receives feature point cloud data increases, the amount of feature point cloud data received by the terminal data receiving unit 233 increases. When the amount of feature point cloud data increases, the time required for matching processing increases. Therefore, the number management unit 238 may change the number of multiple computers 30 based on the amount of feature point cloud data received by the terminal data receiving unit 233. The number management unit 238 may change the number of multiple computers based on the number of information terminals 1 that transmit the feature point cloud data.

When the amount of feature point cloud data received by the terminal data receiving unit 233 within a specified time period is equal to or greater than a threshold, or when the number of information terminals 1 receiving feature point cloud data within a specified time period is increased, the number management unit 238 increases the number of computers 30 that execute the matching process, thereby preventing the time required for the matching process from becoming too long. The number management unit 238 may increase the number of computers 30 that execute the matching process when the number of information terminals 1 from which the terminal data receiving unit 233 receives feature point cloud data is equal to or greater than a threshold.

When the information terminal 1 is moving, it is expected that the information terminal 1 will move to a different space, and the frequency of matching processing for identifying the space to which the information terminal 1 belongs will increase. Therefore, the number management unit 238 may change the number of the multiple computers 30 based on whether the approximate position data received from the information terminal 1 by the terminal data receiving unit 233 indicates that the information terminal 1 is moving.

When the number management unit 238 determines that the number of moving information terminals 1 is equal to or greater than a predetermined number, it increases the number of computers 30 that execute the matching process, thereby preventing the time required for the matching process from becoming too long. Conversely, when the number management unit 238 determines that the number of moving information terminals 1 is less than a predetermined number, it decreases the number of computers 30 that execute the matching process, thereby reducing the cost of using the computers 30.

The number management unit 238 may identify the movement speed of the information terminal 1 based on multiple approximate position data transmitted at multiple times, and increase the number of computers 30 when the number of information terminals 1 whose identified movement speed is equal to or greater than a threshold value is equal to or greater than a predetermined number. By operating in this manner, the number of computers 30 can be changed based on the number of information terminals 1 that are highly likely to switch spaces, and therefore the number of computers 30 that execute the matching process can be made more appropriate.

(Configuration of Computer Group 3)
9 is a diagram showing a configuration of the computer 30 included in the computer group 3. The communication unit 31 includes a communication unit 31, a storage unit 32, and a control unit 33. The control unit 33 includes a feature point group acquisition unit 331, a registered data acquisition unit 332, and a data analysis unit 333.

The communication unit 31 has a communication interface for connecting to a communication network. The communication unit 31 transmits and receives various data between the information processing device 2 and the storage device 4.

The memory unit 32 has a ROM, a RAM, and an SSD. The memory unit 32 stores the programs executed by the control unit 33.

The control unit 33 has, for example, a CPU. The control unit 33 executes the programs stored in the storage unit 32 to function as a feature point group acquisition unit 331, a registered data acquisition unit 332, and a data analysis unit 333.

The feature point group acquisition unit 331 acquires feature point group data from the information processing device 2. The feature point group acquisition unit 331 inputs the acquired feature point group data to the data analysis unit 333.

The registered data acquisition unit 332 acquires a plurality of registered feature point cloud data by reading the plurality of registered feature point cloud data corresponding to the space assigned to the computer 30 from the storage device 4 in which the registered feature point cloud data corresponding to the plurality of spaces is stored. The registered data acquisition unit 332 inputs the acquired plurality of registered feature point cloud data to the data analysis unit 333. The registered data acquisition unit 332 may store the acquired plurality of registered feature point cloud data in the storage unit 32.

The spatial identification information corresponding to the space assigned to the computer 30 is stored, for example, in the storage unit 32, and the registered data acquisition unit 332 acquires from the storage unit 4 a plurality of registered feature point group data stored in the storage unit 4 in association with the spatial identification information.

The data analysis unit 333 transmits predetermined information to the information processing device 2 when the difference between the feature point cloud data acquired by the feature point cloud acquisition unit 331 and any of the multiple registered feature point cloud data acquired by the registered data acquisition unit 332 corresponding to the space assigned to each computer 30 is equal to or less than a threshold. As described above, the predetermined information is information indicating that the feature point cloud data received by the registered data acquisition unit 332 is composed of a feature point cloud of a subject included in the space for which the computer 30 is responsible for the matching process.

[Processing flow in information processing system S]
(Process to obtain spatial data)
Fig. 10 is a sequence diagram showing a flow until multiple information terminals 1 belonging to the same space acquire spatial data. The sequence diagram shown in Fig. 10 starts from the point when the information terminal 1 needs to acquire spatial data. The point when the need to acquire spatial data arises is, for example, the point when the user U starts a predetermined application software on the information terminal 1, logs into a predetermined site, moves a predetermined distance or more from the point where the previous spatial data was acquired, or a predetermined time or more has passed since the previous spatial data was acquired.

The first information terminal and second information terminal in FIG. 10 belong to the same first space, and are, for example, information terminal 1a-1 and information terminal 1a-2 shown in FIG. 1. The first information terminal transmits first approximate position data indicating the position of the first information terminal and first feature point cloud data generated based on captured image data to the information processing device 2. The information processing device 2 transmits the first feature point cloud data to a first computer that executes a matching process using registered feature point cloud data of the first space that includes the approximate position indicated by the received first approximate position data. Here, the information processing device 2 may transmit the first feature point cloud data to multiple first computers.

When the first computer determines that the first feature point cloud data matches any of the multiple registered feature point cloud data, it determines that the first information terminal is in the first space corresponding to the matching registered feature point cloud data, and transmits first spatial data including the multiple registered feature point cloud data corresponding to the first space to the information processing device 2. The information processing device 2 transmits the received first spatial data to the first information terminal. The first information terminal then identifies its own detailed location by comparing the feature point cloud data it generated with the multiple registered feature point cloud data it received.

The second information terminal operates in the same manner as the first information terminal, and transmits to the information processing device 2 second approximate position data indicating the position of the second information terminal and second feature point cloud data generated based on the captured image data. Since the second information terminal belongs to the same first space as the first information terminal, the information processing device 2 transmits the second feature point cloud data to the first computer and receives the first spatial data from the first computer. The information processing device 2 transmits the first spatial data to the second information terminal, and the second information terminal identifies its own detailed position by comparing the feature point cloud data it generated with multiple registered feature point cloud data included in the received first spatial data.

Figure 11 is a sequence diagram showing the flow until multiple information terminals 1 belonging to different spaces acquire spatial data. The first information terminal and the third information terminal in Figure 11 belong to different spaces, and are, for example, the information terminal 1a-1 and the information terminal 1b-1 shown in Figure 1. The operation of the first information terminal is the same as the operation shown in Figure 10.

The third information terminal transmits third approximate position data indicating the position of the third information terminal and third feature point cloud data generated based on the captured image data to the information processing device 2. Since the third information terminal belongs to a second space different from the first information terminal, the information processing device 2 transmits the third feature point cloud data to a second computer in charge of the second space corresponding to the third approximate position data, and receives the second space data from the third computer. The information processing device 2 transmits the second space data to the second information terminal, and the second information terminal identifies its own detailed position by comparing the feature point cloud data it generated with multiple registered feature point cloud data included in the received second space data.

(Processing for displaying AR images)
Fig. 12 is a sequence diagram showing a process flow in which multiple information terminals 1 belonging to the same space display an AR image. The sequence diagram shown in Fig. 12 starts from the point in time when a user U1 using a first information terminal performs an operation to display an AR image. The first information terminal identifies the content of the operation performed by the user U1 (S11) and identifies the operation position coordinates indicating the position where the operation was performed (S12). The first information terminal transmits operation content data indicating the content of the operation and coordinate data indicating the operation position to the information processing device 2.

Based on the received operation content data, the information processing device 2 selects AR image data to be displayed on the first information terminal, and transmits the selected AR image data to the first information terminal. Note that if the first information terminal stores the AR image data, the information processing device 2 does not need to transmit the AR image data to the first information terminal.

The information processing device 2 transmits display coordinate data corresponding to the coordinate data received from the first information terminal to a second information terminal belonging to the same space as the first information terminal. The information processing device 2 also transmits the AR image data transmitted to the first information terminal to the second information terminal. The first information terminal and the second information terminal display an AR image based on the received AR image data at a position and orientation corresponding to the display coordinate data (S13, S14). By the first information terminal and the second information terminal operating in this manner, multiple information terminals 1 belonging to the same space can display the same AR image at a position and orientation according to their respective positions and orientations.

(Spatial data registration)
Fig. 13 is a sequence diagram showing the flow of processing in which the information terminal 1 registers feature point group data as spatial data. The sequence diagram shown in Fig. 13 starts from the point in time when the first information terminal compares the feature point group data that it has created with the registered feature point group data.

When the first information terminal determines that there is no spatial data including registered feature point cloud data that matches the first feature point cloud data created by the first information terminal (YES in S21), it determines that the spatial data needs to be updated (S22) and transmits the first feature point cloud data together with an update request to the information processing device 2. Similarly, when the second information terminal determines that there is no spatial data including registered feature point cloud data that matches the second feature point cloud data created by the first information terminal (YES in S23), it transmits the second feature point cloud data together with an update request to update the spatial data to the information processing device 2.

When the information processing device 2 receives the first feature point cloud data and the second feature point cloud data corresponding to the same position and orientation, it determines which feature point cloud data should be registered based on one of the above-mentioned predetermined conditions, and selects one of the feature point cloud data (S24). The information processing device 2 transmits the selected feature point cloud data to the storage device 4, and the storage device 4 registers the received feature point cloud data as spatial data (S25).

[Modification]
In the above explanation, the information processing device 2 relays the feature point group data and the registered feature point group data between the information terminal 1 and the computer group 3 , but the computer group 3 may have the functions of the information processing device 2 .

In the above description, the information processing device 2 provides an AR image to the information terminal 1 in response to the information terminal 1 transmitting display coordinate data to the information processing device 2. However, the information terminal 1 may store multiple AR images in advance. In this case, the information terminal 1 selects an AR image to be displayed from the multiple stored AR images based on an instruction from the information processing device 2, and displays the selected AR image in a position and orientation based on the received display coordinate data.

In addition, in the above explanation, an example was given of the case where the information processing device 2 has an in-memory database and various display coordinate data and feature point group data are exchanged via the in-memory database, but the information processing device 2 may not have an in-memory database, and multiple information terminals 1 may exchange these data via other storage media.

[Effects of the information processing system S]
The information processing device 2 according to this embodiment receives the display coordinate data transmitted by the first information terminal, registers the received display coordinate data in an in-memory database, and transmits the display coordinate data to a second information terminal belonging to the same space as the first information terminal in response to the display coordinate data being registered in the in-memory database. By the information processing device 2 transmitting and receiving the display coordinate data between the multiple information terminals 1 via the in-memory database, it is possible to reduce the time required for the information terminal 1 to display an AR image.

The computer group 3 also acquires feature point cloud data based on the captured image data generated by the information terminal 1, and multiple computers 30, each responsible for a different space, compare the multiple registered feature point cloud data with the acquired feature point cloud data to identify the space to which the information terminal 1 belongs. The computer group 3 then provides the multiple registered feature point cloud data corresponding to the identified space to the information terminal 1. By the computer group 3 operating in this manner, the information terminal 1 can limit the multiple registered feature point cloud data to be compared with the feature point cloud data it created to the multiple registered feature point cloud data corresponding to the space to which it belongs, and can therefore identify its own accurate position and orientation in a short time.

In addition, the information processing device 2 changes the number of computers 30 that perform matching processing with the feature point cloud data generated by the information terminal 1 based on preset conditions. By operating the information processing device 2 in this manner, even if the number of information terminals 1 increases or decreases, or the amount of feature point cloud data transmitted by the information terminal 1 increases or decreases, the time required to transmit appropriate registered feature point cloud data to the information terminal 1 can be maintained within a certain range.

Furthermore, when there is no registered feature point cloud data corresponding to the feature point cloud data generated by the information terminal 1, the information terminal 1 can register the feature point cloud data it has generated in the storage device 4. By registering a large amount of feature point cloud data generated by the information terminals 1 in the storage device 4, the registered feature point cloud data stored in the storage device 4 can be kept up to date. Furthermore, when the information processing device 2 receives update requests for feature point cloud data corresponding to the same position and orientation from multiple information terminals 1, it registers the more reliable feature point cloud data in the storage device 4. By the information processing device 2 operating in this manner, the reliability of the registered feature point cloud data stored in the storage device 4 is improved.

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

REFERENCE SIGNS LIST 1 Information terminal 2 Information processing device 3 Computer group 4 Storage device 11 Communication unit 12 Operation unit 13 Position detection unit 14 Imaging unit 15 Display unit 16 Memory unit 17 Control unit 21 Communication unit 22 Memory unit 23 Control unit 30 Computer 31 Communication unit 32 Memory unit 33 Control unit 171 Transmission processing unit 172 Reception processing unit 173 Operation acceptance unit 174 Approximate position identification unit 175 Image processing unit 176 Detailed position identification unit 177 Display processing unit 178 Spatial data acquisition unit 221 First spatial database 222 Second spatial database 231 Coordinate reception unit 232 Coordinate transmission unit 233 Terminal data reception unit 234 Feature point group transmission unit 235 Information reception unit 236 Spatial data transmission unit 237 Spatial data management unit 238 Number of units management unit 331 Feature point group acquisition unit 332 Registration data acquisition unit 333 Data analysis unit

Claims (8)

A system including one or more information terminal groups including a plurality of information terminals that display an augmented reality image in a space of the same coordinate system, and an information processing device that provides the augmented reality image to the information terminals,
A first information terminal of the plurality of information terminals,
a first display processing unit that displays a first augmented reality image on a display unit by superimposing the first augmented reality image on an image of the space viewed by a user of the first information terminal;
a terminal storage unit that stores a plurality of spatial data indicating feature points of a subject included in a plurality of images that have been created in advance by capturing images of the space from a plurality of positions and orientations in the space, in association with the positions and orientations in the space;
a position detection unit that detects a position of the first information terminal;
a spatial data acquisition unit that acquires from the information processing device the plurality of spatial data corresponding to the position detected by the position detection unit, and stores the acquired plurality of spatial data in the terminal storage unit;
a detailed position specification unit that specifies a position and an orientation corresponding to spatial data that is most similar to feature point group data indicating feature points of a subject included in an image created by the first information terminal by imaging the space, among the plurality of spatial data stored in the terminal storage unit;
a first operation receiving unit that, in response to receiving an operation for displaying the first augmented reality image on the display unit, creates display coordinate data indicating coordinates in the space when the first augmented reality image is to be displayed on the display unit , based on a relationship between the position and orientation specified by the detailed position specifying unit and the position of the received operation;
a first transmission processing unit that transmits the display coordinate data to the information processing device;
having
The information processing device includes:
a coordinate receiving unit that receives the display coordinate data transmitted by the first information terminal and registers the received display coordinate data in an in-memory database;
a coordinate transmission unit that transmits the display coordinate data to a second information terminal different from the first information terminal among the plurality of information terminals in response to the display coordinate data being registered in the in-memory database;
having
The second information terminal is
A second reception processing unit that receives the display coordinate data transmitted by the coordinate transmission unit;
a second display processing unit that displays a second augmented reality image at a position on the display unit that corresponds to the coordinates in the space indicated by the display coordinate data, superimposed on an image of the space viewed by a user of the second information terminal;
An information processing system having the above configuration. the information processing device has a plurality of in-memory databases corresponding to the plurality of spaces,
The coordinate receiving unit registers the display coordinate data in the in-memory database corresponding to the space including the first information terminal that transmitted the display coordinate data.
The information processing system according to claim 1 . the information processing device has a plurality of in-memory databases corresponding to the plurality of spaces,
the coordinate transmission unit transmits to the second information terminal the display coordinate data registered in the in-memory database corresponding to the space in which the second information terminal is included;
3. The information processing system according to claim 1 or 2. The information processing device further includes a storage unit configured to store a plurality of terminal identification information for identifying the plurality of information terminals included in the space in association with each of the plurality of spaces;
The coordinate transmission unit transmits the display coordinate data to the second information terminal having second terminal identification information associated with the space corresponding to the first terminal identification information of the first information terminal that transmitted the display coordinate data, by referring to the storage unit.
The information processing system according to claim 1 . the first operation reception unit creates operation content data indicating a type of the first augmented reality image based on the received operation;
The first transmission processing unit further transmits the operation content data to the information processing device,
The coordinate transmission unit transmits, in response to the operation content data being registered in the in-memory database, data corresponding to the second augmented reality image corresponding to the operation content data or data for identifying the second augmented reality image to the second information terminal.
The information processing system according to claim 1 . the first operation receiving unit creates display orientation data indicating an orientation of the first augmented reality image when the first augmented reality image is displayed on the display unit, based on a relationship between the position and orientation specified by the detailed position specifying unit and a position of the received operation;
The first transmission processing unit further transmits the display orientation data to the information processing device,
The coordinate transmission unit transmits the display orientation data to the second information terminal in response to the display orientation data being registered in the in-memory database.
The information processing system according to claim 1 . The information processing device further includes a device storage unit that stores the first information terminal and the second information terminal belonging to a space of the same coordinate system as the plurality of information terminals that display an augmented reality image in the space.
The information processing system according to claim 1 . A step in which a first information terminal belonging to a predetermined space detects a position of the first information terminal;
a step in which the first information terminal acquires from an information processing device a plurality of spatial data indicating feature points of a subject included in a plurality of images that have been created in advance by capturing images of the space from a plurality of positions and orientations in the space corresponding to the detected position, and stores the acquired plurality of spatial data in a terminal storage unit that stores the acquired plurality of spatial data in association with the positions and orientations in the space;
a step of the first information terminal identifying a position and an orientation corresponding to spatial data that is most similar to feature point group data indicating feature points of a subject included in an image created by the first information terminal by imaging the space, from among the plurality of spatial data stored in the terminal storage unit;
In response to receiving an operation for displaying a first augmented reality image on a display unit of the first information terminal by superimposing the first augmented reality image on an image of a space viewed by a user of the first information terminal, creating display coordinate data indicating coordinates within the space when the first augmented reality image is to be displayed on the display unit , based on a relationship between the specified position and orientation and the position of the received operation;
a step of the first information terminal transmitting the display coordinate data to the information processing device;
a step of registering the display coordinate data received from the first information terminal in an in-memory database by the information processing device;
a step of transmitting the display coordinate data to a second information terminal belonging to the space different from the first information terminal, by the information processing device in response to the display coordinate data being registered in the in-memory database;
a step of receiving, by the second information terminal, the display coordinate data transmitted by the information processing device;
a step of causing the second information terminal to display a second augmented reality image corresponding to the first augmented reality image at a position on a display unit of the second information terminal that corresponds to coordinates in a space visually recognized by a user of the second information terminal, the coordinates being indicated by the display coordinate data;
An information processing method comprising the steps of:

Images