JP7709719B2 - Terminal device, information processing system, information processing method, and computer program - Google Patents

Description

The technology disclosed in this specification relates to a terminal device, an information processing system, an information processing method, and a computer program.

In recent years, Augmented Reality (AR) technology, which uses computers to augment the real-world environment perceived by humans, has become increasingly popular. For example, in a terminal device (such as a smartphone) held by a user and moving with the user, AR-type information presentation, in which a virtual image (computer graphics) is superimposed on an image of the real world generated by an imaging unit and displayed on a display unit, is used in the form of, for example, walking navigation, tourist guides, maintenance and inspection guides, location games, etc.

In AR-type information presentation, information on where and from what distance and angle the imaging unit of the terminal device is looking is identified based on the position and attitude of the terminal device, and the type, size, position, etc. of the virtual image to be displayed on the display unit are determined based on this information. Therefore, in order to display a virtual image that matches the real environment in AR-type information presentation, it is necessary to estimate the position and attitude of the terminal device (more specifically, the imaging unit of the terminal device; the same applies below) with high accuracy.

Conventionally, a method is known in which a terminal device receives signals from a GNSS (Global Navigation Satellite System) such as GPS, GLONASS, Galileo, or QZSS to estimate the position and attitude of the terminal device. Also known is a method in which a marker whose position and attitude in the real environment are known is placed, an image of the marker is captured by an imaging unit of the terminal device to identify information on the relative position and attitude of the terminal device with respect to the marker, and the position and attitude of the terminal device in the real environment is estimated based on the information (see, for example, Patent Document 1).

JP 2021-4894 A

Methods that use GNSS to estimate the position and attitude of a terminal device have the problem that the estimation error is relatively large (e.g., an error of about several meters) and that signals cannot be received well indoors, making them difficult to use indoors. In addition, methods that use markers to estimate the position and attitude of a terminal device have the problem that a large number of markers whose positions and attitudes are known must be installed in advance.

It is also possible to equip the terminal device with a sensor (e.g., LiDAR) and a three-dimensional point cloud map database used for self-position and orientation estimation by an autonomous mobile body, and use these to estimate the position and orientation of the terminal device. However, if such sensors and databases are equipped on a terminal device that is held by a user and moves with the user, there are issues such as the terminal device becoming larger and heavier, and the storage capacity of the terminal device becoming insufficient.

As such, in the past, it was difficult to estimate the position and orientation of a terminal device with high accuracy both indoors and outdoors while preventing the terminal device from becoming too large and heavy and from running out of memory capacity, and without the need to install a large number of markers in advance.

Note that this issue is not limited to estimating the position and orientation of a terminal device for AR-type information presentation, but is also a common issue in estimating the position and orientation of a terminal device for other purposes.

This specification discloses a technology that can solve the above-mentioned problems.

The technology disclosed in this specification can be realized, for example, in the following forms:

(1) The terminal device disclosed in this specification is a device that is held by a user and moves with the user. The terminal device includes an imaging unit, a marker information acquisition unit, an image correspondence information acquisition unit, a relative relationship identification unit, and a terminal position and orientation estimation unit. The marker information acquisition unit is a moving body that moves while identifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is assigned, and acquires marker position and orientation information indicating the position and orientation of the marker unit on the three-dimensional point cloud map for the moving body having a marker unit that functions as a marker. The image correspondence information acquisition unit acquires image correspondence information that associates an image in which the marker unit is captured with a relative position and relative orientation with respect to the marker unit. The relative relationship identification unit identifies relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit based on the image of the marker unit captured by the imaging unit and the image correspondence information. The terminal position and orientation estimation unit estimates the position and orientation of the terminal device on the 3D point cloud map based on the marker position and orientation information and the relative relationship information.

In this way, the terminal device can estimate the position and orientation of the terminal device on the three-dimensional point cloud map based on marker position and orientation information indicating the position and orientation of the marker unit of the moving object on the three-dimensional point cloud map, and relative relationship information indicating the relative position and relative orientation of the terminal device with respect to the marker unit. Therefore, the terminal device can estimate the position and orientation of the terminal device seamlessly, both indoors and outdoors, as long as the marker unit of the moving object is in a position where it can be imaged by the imaging unit, without the need to install a large number of markers whose positions and orientations are known in advance, and with high accuracy, without equipping the terminal device with sensors other than the imaging unit (e.g., LiDAR) or a three-dimensional point cloud map database.

(2) The above terminal device may further include a tracking unit that tracks changes in the position and attitude of the terminal device from the time when the position and attitude of the terminal device are estimated, and an estimation result update unit that updates the estimation results of the position and attitude of the terminal device based on the changes. According to this terminal device, once the position and attitude (initial position and initial attitude) of the terminal device on the three-dimensional point cloud map is estimated based on the marker position and attitude information and the relative relationship information, the terminal device alone can update the estimation results of the position and attitude of the terminal device, regardless of whether the marker unit of the moving body can be imaged by the imaging unit. At this time, since the estimation accuracy of the initial position and initial attitude is high, the estimation accuracy of the position and attitude of the terminal device is maintained at a certain level or higher even after subsequent updates. Therefore, according to this terminal device, the position and attitude of the terminal device that moves independently of the moving body can be continuously estimated with high accuracy.

(3) In the above terminal device, the tracking unit may be configured to track changes in the position and attitude of the terminal device using images captured by the imaging unit. This terminal device can continuously estimate the position and attitude of the terminal device with high accuracy without requiring any sensors other than the imaging unit.

(4) The above terminal device may further include a display unit and an augmented reality processing unit that displays a virtual image, which changes according to the estimation result of the position and orientation of the terminal device, on the display unit by superimposing the virtual image on the real image captured by the imaging unit. According to this terminal device, a virtual image can be displayed based on the position and orientation of the terminal device estimated with high accuracy, and highly accurate AR-type information presentation can be realized.

(5) The information processing system disclosed in this specification includes the above terminal device and the mobile body. The terminal device may further include an estimation result transmission unit that transmits the estimation result of the position and attitude of the terminal device to the mobile body directly or via another device, and the mobile body may include a specific process execution unit that executes specific processing using the estimation result of the position and attitude of the terminal device. According to this information processing system, specific processing can be executed using the position and attitude of the terminal device on a three-dimensional point cloud map that are estimated with high accuracy, and the accuracy of the specific processing can be improved.

(6) In the above information processing system, the specific process may be configured to include a hazard notification process that notifies the terminal device of the presence of a hazard close to the terminal device. According to this information processing system, the presence of a hazard can be notified from a moving body to a terminal device by using the position and orientation of the terminal device on a three-dimensional point cloud map that are estimated with high accuracy, thereby realizing highly accurate hazard notification.

(7) In the above information processing system, the moving body may be an autonomous moving body that moves autonomously using surrounding detection information from a sensor and the three-dimensional point cloud map. According to this information processing system, by moving the marker unit without requiring human driving operation, it is possible to realize highly accurate estimation of the position and attitude of the terminal device over a wide range using the moving marker unit.

(8) The above information processing system may further include a server device, the moving body may transmit the marker position and orientation information to the server device, and the marker information acquisition unit of the terminal device may acquire the marker position and orientation information via the server device. According to this information processing system, even in a terminal device that cannot directly communicate with a moving body, it is possible to realize highly accurate estimation of the position and orientation of the terminal device using the marker unit of the moving body.

The technology disclosed in this specification can be realized in various forms, such as a terminal device, an information processing system, an information processing method, a computer program that realizes these methods, and a non-transitory recording medium on which the computer program is recorded.

FIG. 1 is an explanatory diagram illustrating a schematic configuration of an information processing system according to a first embodiment. FIG. 1 is a block diagram illustrating a configuration of a terminal device 100 according to a first embodiment. FIG. 1 is a block diagram illustrating a schematic configuration of an autonomous moving body 200 according to a first embodiment. FIG. 1 is a block diagram showing a schematic configuration of a server device 300 according to a first embodiment. FIG. 1 is an explanatory diagram showing a flow of an AR-type information presentation process executed in the information processing system 10 of the first embodiment; FIG. 1 is an explanatory diagram conceptually illustrating an AR-type information presentation process executed in the information processing system 10 of the first embodiment; FIG. 1 is an explanatory diagram conceptually illustrating an AR-type information presentation process executed in the information processing system 10 of the first embodiment; A diagram showing how to convert from a user coordinate system to a world coordinate system. FIG. 13 is an explanatory diagram illustrating a schematic configuration of an information processing system 10a according to a second embodiment. FIG. 11 is a block diagram showing a schematic configuration of a terminal device 100a according to a second embodiment. FIG. 11 is a block diagram showing a schematic configuration of an autonomous moving body 200a according to a second embodiment. FIG. 11 is a block diagram showing a schematic configuration of a server device 300a according to a second embodiment. FIG. 11 is an explanatory diagram showing the flow of a hazard notification process executed in the information processing system 10a of the second embodiment; FIG. 11 is an explanatory diagram conceptually illustrating a hazard notification process executed in an information processing system 10a according to a second embodiment;

A. First embodiment:
A-1. Configuration of information processing system 10:
1 is an explanatory diagram that shows a schematic configuration of an information processing system 10 in the first embodiment. The information processing system 10 in this embodiment is a system for estimating the position and attitude (direction) of a terminal device 100 (more specifically, an imaging unit 154 described later, the same applies below) on a three-dimensional point cloud map with high accuracy, and for executing AR-type information presentation on the terminal device 100 based on the estimated position and attitude of the terminal device 100. In this embodiment, an AR-type information presentation that displays guidance both indoors and outdoors in a shopping mall will be described as an example.

The information processing system 10 includes a terminal device 100 used by each of a plurality of users U, a plurality of autonomous mobile bodies 200, and a server device 300. The devices constituting the information processing system 10 are connected to each other so as to be able to communicate with each other via a communication network NET.

The terminal device 100 is a device that is held by the user U and moves with the user U, and is, for example, a handheld device such as a smartphone or a tablet terminal, or a head mounted image display device (Head Mounted Display, HMD). Below, an example in which a smartphone is used as the terminal device 100 will be described.

FIG. 2 is a block diagram showing a schematic configuration of the terminal device 100 in the first embodiment. The terminal device 100 includes a control unit 110, a storage unit 120, a display unit 151, an operation input unit 152, an audio output unit 153, an imaging unit 154, and a communication unit 155. These units are connected to each other via a bus 190 so as to be able to communicate with each other.

The display unit 151 of the terminal device 100 is configured, for example, by a liquid crystal display or an organic EL display, and displays various images and information. The operation input unit 152 is configured, for example, by a button or a microphone, and receives operations and instructions from the user U. The display unit 151 may function as the operation input unit 152 by being equipped with a touch panel. The audio output unit 153 is configured, for example, by a speaker, and outputs (plays) audio. The imaging unit 154 is configured, for example, by a camera, and captures an image of a subject to generate a still image or a moving image representing the subject. In this embodiment, the imaging unit 154 is a camera (rear camera) with a lens arranged on the surface (rear surface) opposite to the display surface of the display unit 151 in the terminal device 100. The communication unit 155 is a communication interface that communicates with other devices in a predetermined communication method via the communication network NET.

The storage unit 120 of the terminal device 100 is composed of, for example, ROM, RAM, etc., and is used to store various programs and data, and as a work area when executing various programs, and as a temporary storage area for data. For example, the storage unit 120 stores an augmented reality processing program CP11 for performing the AR-type information presentation processing described below. The augmented reality processing program CP11 is provided in a state stored in a computer-readable recording medium (not shown), such as a CD-ROM, DVD-ROM, or USB memory, or is downloaded from an external device via the communications network NET and installed in the terminal device 100, thereby being stored in the storage unit 120.

In addition, when the augmented reality processing program CP11 is installed or when the AR-type information presentation process described below is performed, the storage unit 120 stores marker image information MI, marker position and orientation information LI, image correspondence information II, and relative relationship information CI. The contents of this information will be explained in conjunction with the explanation of the AR-type information presentation process described below.

The control unit 110 of the terminal device 100 is configured with, for example, a CPU, and controls the operation of the terminal device 100 by executing a computer program read from the storage unit 120. For example, the control unit 110 functions as an augmented reality processing unit 111 that performs AR-type information presentation processing by reading an augmented reality processing program CP11 from the storage unit 120 and executing it. The augmented reality processing unit 111 includes a terminal position and orientation estimation unit 112, which includes a marker information acquisition unit 113, an image correspondence information acquisition unit 114, a relative relationship identification unit 115, a tracking unit 116, and an estimation result update unit 117. The functions of each of these units will be described in conjunction with the description of the AR-type information presentation processing described below.

The autonomous mobile body 200 is a mobile body that travels autonomously without the need for human driving operation. The autonomous mobile body 200 is, for example, an autonomous vehicle or an autonomous mobile robot. In this embodiment, an example in which an autonomous mobile body 200 is used as an autonomous mobile robot that transports luggage in a shopping mall will be described. The autonomous mobile body 200 is an example of a mobile body within the scope of the claims.

FIG. 3 is a block diagram showing a schematic configuration of an autonomous moving body 200 in the first embodiment. The autonomous moving body 200 includes a control unit 210, a storage unit 220, a sensor 230, a drive unit 250, a communication unit 255, and a marker unit 260.

The sensor 230 of the autonomous moving body 200 is composed of, for example, LiDAR, RADAR, a visible light camera, a far-infrared camera, an ultrasonic sensor, etc., and recognizes the state (position, attitude, shape, moving speed, etc.) of the surrounding environment of the autonomous moving body 200 (other vehicles, pedestrians, bicycles, road obstacles, street trees, road steps, etc.) and the state of the autonomous moving body 200 itself. The drive unit 250 is composed of, for example, a motor, an engine, a steering wheel, a brake, etc., and performs the running operation (acceleration, steering, braking, etc.) of the autonomous moving body 200. The communication unit 255 is a communication interface that communicates with other devices using a specified communication method via the communication network NET. The marker unit 260 is a part that functions as a marker. In this embodiment, the marker unit 260 is a specified planar image attached to the side of the autonomous moving body 200.

The storage unit 220 of the autonomous mobile body 200 is composed of, for example, a ROM, a RAM, a hard disk drive (HDD), etc., and is used to store various programs and data, and as a work area when executing various programs and a temporary storage area for data. For example, the storage unit 220 stores an autonomous movement control program CP21, a position and orientation information provision program CP22, and a three-dimensional point cloud map database MD. The three-dimensional point cloud map database MD contains a high-precision three-dimensional point cloud map composed of a point cloud to which information indicating the three-dimensional absolute positions (latitude, longitude, altitude) of roads, passages, street trees, guardrails, buildings, etc. is added. The three-dimensional point cloud map contained in the three-dimensional point cloud map database MD covers the entire movable range of the autonomous mobile body 200. In the following, the coordinate system of the three-dimensional point cloud map is also referred to as the "world coordinate system", and the position and orientation on the three-dimensional point cloud map are also referred to as the "absolute position" and "absolute orientation".

The control unit 210 of the autonomous mobile body 200 is configured, for example, by a CPU, and controls the operation of the autonomous mobile body 200 by executing a computer program read from the storage unit 220. For example, the control unit 210 functions as an autonomous mobile control unit 212 that controls the autonomous running of the autonomous mobile body 200 by reading and executing the autonomous mobile control program CP21 from the storage unit 220. The autonomous mobile control unit 212 sets a moving route based on an instruction issued by an administrator. In addition, the autonomous mobile control unit 212 estimates the position (absolute position) and attitude (absolute attitude) of the autonomous mobile body 200 (more specifically, the sensor 230 of the autonomous mobile body 200, the same below) on the three-dimensional point cloud map by matching various signals output from the sensor 230 with the three-dimensional point cloud map included in the three-dimensional point cloud map database MD. Furthermore, the autonomous mobile control unit 212 detects and recognizes the state of the surrounding environment based on various signals output from the sensor 230. The autonomous movement control unit 212 controls the driving operation of the drive unit 250 based on this information, thereby causing the autonomous moving body 200 to travel along the set movement path while avoiding collisions with surrounding objects.

The control unit 210 of the autonomous moving body 200 also functions as a position and orientation information providing unit 214 by reading and executing the position and orientation information providing program CP22 from the storage unit 220. The functions of the position and orientation information providing unit 214 will be explained in conjunction with the explanation of the AR-type information presentation process described below.

The server device 300 is a device for acquiring information from the terminal device 100 and the autonomous mobile body 200, and for providing information to the terminal device 100 and the autonomous mobile body 200. FIG. 4 is a block diagram showing a schematic configuration of the server device 300 in the first embodiment. The server device 300 includes a control unit 310, a storage unit 320, a display unit 351, an operation input unit 352, and a communication unit 355. These units are connected to each other via a bus 390 so as to be able to communicate with each other.

The display unit 351 of the server device 300 is, for example, a liquid crystal display or an organic EL display, and displays various images and information. The operation input unit 352 is, for example, a keyboard, mouse, buttons, microphone, etc., and accepts operations and instructions from the administrator. The communication unit 355 is a communication interface that communicates with other devices using a specified communication method via the communication network NET.

The storage unit 320 of the server device 300 is composed of, for example, a ROM, RAM, a hard disk drive (HDD), etc., and is used to store various programs and data, and as a working area when executing various programs, and as a temporary storage area for data. For example, the storage unit 320 stores an augmented reality server program CP31 for performing the AR-type information presentation process described below. The augmented reality server program CP31 is provided in a state stored in a computer-readable recording medium (not shown), such as a CD-ROM, DVD-ROM, or USB memory, or is downloaded from an external device via the communications network NET and installed in the server device 300, thereby being stored in the storage unit 320.

In addition, the storage unit 320 of the server device 300 stores a virtual image database ID and marker position and orientation information LI when the augmented reality server program CP31 is installed or when the AR-type information presentation process described below is performed. In the virtual image database ID, a plurality of virtual images to be provided to the terminal device 100 in the AR-type information presentation process described below are set in association with the position and orientation of the terminal device 100. In this embodiment, the virtual images are images for displaying guidance within a shopping mall (see Figures 6 and 7 described below). The contents of the marker position and orientation information LI will be described in conjunction with the explanation of the AR-type information presentation process described below.

The control unit 310 of the server device 300 is configured, for example, with a CPU, and controls the operation of the server device 300 by executing a computer program read from the storage unit 320. For example, the control unit 310 functions as an augmented reality server processing unit 312 that performs AR-type information presentation processing by reading and executing an augmented reality server program CP31 from the storage unit 320. The augmented reality server processing unit 312 includes a position and orientation server processing unit 314. The functions of each of these units will be described in conjunction with the explanation of the AR-type information presentation processing described below.

A-2. AR-type information presentation processing:
Next, an AR-type information presentation process executed in the information processing system 10 of the first embodiment will be described. The AR-type information presentation process of the present embodiment is a process in which a virtual image is superimposed on an image of a real environment captured by the imaging unit 154 in the terminal device 100 and displayed on the display unit 151. In the present embodiment, an AR-type information presentation that displays guidance both indoors and outdoors in a shopping mall will be described as an example.

Fig. 5 is an explanatory diagram showing the flow of the AR-type information presentation process executed in the information processing system 10 of the first embodiment. Figs. 6 and 7 are explanatory diagrams conceptually showing the AR-type information presentation process executed in the information processing system 10 of the first embodiment. Fig. 6 shows a scene where AR-type information presentation is being performed outdoors in a shopping mall, and Fig. 7 shows a scene where AR-type information presentation is being performed indoors in a shopping mall.

As shown in FIG. 5, when the process starts, the autonomous mobile body 200 starts autonomous movement outdoors and indoors in the shopping mall (S210). That is, the autonomous mobile body 200's autonomous movement control unit 212 (FIG. 3) estimates the position (absolute position) and attitude (absolute attitude) of the autonomous mobile body 200 on the three-dimensional point cloud map included in the three-dimensional point cloud map database MD, while causing the autonomous mobile body 200 to travel autonomously. The autonomous movement control unit 212 estimates the position and attitude of the autonomous mobile body 200 on the three-dimensional point cloud map by matching various signals output from the sensor 230 with the three-dimensional point cloud map, and the estimation accuracy is very high, for example, with an error of about several centimeters.

When the autonomous moving body 200 starts autonomous movement, the position and orientation information providing unit 214 (FIG. 3) of the autonomous moving body 200 transmits marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map to the server device 300 (S220). In the autonomous moving body 200, information indicating the correspondence between the position and orientation of the autonomous moving body 200 (the sensor 230) grasped by the autonomous movement control unit 212 and the position and orientation of the marker unit 260 is stored in advance in the storage unit 220. The position and orientation information providing unit 214 refers to the correspondence, converts the position and orientation of the autonomous moving body 200 grasped by the autonomous movement control unit 212 into the position and orientation of the marker unit 260 to generate marker position and orientation information LI, and transmits the marker position and orientation information LI to the server device 300. The marker position and orientation information LI includes a moving body ID for identifying the autonomous moving body 200, a timestamp indicating the estimated time of the position and orientation, and information specifying the position and orientation (see FIG. 4).

The position and orientation information providing unit 214 of the autonomous moving body 200 determines whether the autonomous movement of the autonomous moving body 200 has ended (S230), and if it is determined that the autonomous movement of the autonomous moving body 200 has not ended (S230: NO), it executes the marker position and orientation information LI transmission process (S220) again. That is, the position and orientation information providing unit 214 repeatedly executes the marker position and orientation information LI transmission process (S220) periodically (for example, at about 1 Hz to 100 Hz) while the autonomous moving body 200 is performing autonomous movement. In this embodiment, many autonomous moving bodies 200 execute such processing both indoors and outdoors in a shopping mall. If it is determined that the autonomous movement of the autonomous moving body 200 has ended (S230: YES), the autonomous moving body 200 ends its operation.

The position and orientation server processing unit 314 (FIG. 4) of the server device 300 receives the marker position and orientation information LI transmitted from the autonomous moving body 200 in S220, and stores the received marker position and orientation information LI in the storage unit 320 (S310). The position and orientation server processing unit 314 accumulates the marker position and orientation information LI transmitted from the multiple autonomous moving bodies 200 from time to time in the storage unit 320.

In addition, the user U of the terminal device 100, indoors and outdoors in a shopping mall, instructs the execution of an application program (augmented reality processing program CP11) for AR-type information presentation, searches for a nearby autonomous moving body 200, and holds the terminal device 100 so that the imaging unit 154 faces the direction of the marker unit 260 of the autonomous moving body 200 (see Figures 6 and 7). Triggered by such an action by the user U, the terminal position and orientation estimation unit 112 (Figure 2) of the terminal device 100 detects an image of the marker unit 260 from the image generated by the imaging unit 154 (S110). As described above, the storage unit 120 of the control unit 110 stores marker image information MI. The marker image information MI is information that specifies the image used as the marker unit 260. The terminal position and orientation estimation unit 112 refers to the marker image information MI and detects the image of the marker unit 260 from the captured image by a known method such as pattern matching. In the example shown in FIG. 6, the marker unit 260 of the autonomous moving body 200 moving near the entrance outside the shopping mall is captured by the imaging unit 154 of the terminal device 100 held by the user U and displayed on the display unit 151. In the example shown in FIG. 7, the marker unit 260 of the autonomous moving body 200 moving inside the shopping mall is captured by the imaging unit 154 of the terminal device 100 held by the user U and displayed on the display unit 151.

The image correspondence information acquisition unit 114 of the terminal device 100 acquires the image correspondence information II from the storage unit 120, and the relative relationship identification unit 115 of the terminal device 100 identifies the relative relationship information CI indicating the relative position and relative attitude of the terminal device 100 (its imaging unit 154) with respect to the marker unit 260 based on the image of the marker unit 260 captured by the imaging unit 154 and the image correspondence information II (S120).

More specifically, the image correspondence information II is information that associates an image in which the marker unit 260 is captured with a relative position and relative orientation with respect to the marker unit 260. The appearance of the marker unit 260 in the image captured by the imaging unit 154 changes depending on the relative position and relative orientation of the imaging unit 154 of the terminal device 100 with respect to the marker unit 260. For example, if the distance between the imaging unit 154 and the marker unit 260 is relatively short, the marker unit 260 appears relatively large in the captured image, and conversely, if the distance between the imaging unit 154 and the marker unit 260 is relatively long, the marker unit 260 appears relatively small in the captured image. In addition, the appearance of the marker unit 260 in the captured image differs between a state in which the imaging unit 154 faces the marker unit 260 and a state in which the imaging unit 154 is positioned diagonally with respect to the marker unit 260. The image correspondence information II is information that associates the relative position and orientation of the imaging unit 154 of the terminal device 100 with the marker unit 260 and the appearance of the marker unit 260 in the captured image. Therefore, the relative relationship identification unit 115 can identify the relative position and orientation of the terminal device 100 (the imaging unit 154 of the terminal device 100) with the marker unit 260 based on the image of the marker unit 260 and the image correspondence information II. Note that in this embodiment, the relative position and orientation of the terminal device 100 with respect to the marker unit 260 are expressed as the position and orientation of the marker unit 260 in a coordinate system used by the terminal device 100 (hereinafter referred to as the "user coordinate system"). The characterized relative relationship information CI is stored in the storage unit 120.

The marker information acquisition unit 113 (FIG. 1) of the terminal device 100 requests the server device 300 for the marker position and orientation information LI, and acquires the marker position and orientation information LI from the server device 300 (S130). In this embodiment, the marker unit 260 of the autonomous moving body 200 includes information for identifying the autonomous moving body 200 (moving body ID). Therefore, the marker information acquisition unit 113 requests the server device 300 for the marker position and orientation information LI by specifying the moving body ID of the autonomous moving body 200 whose relative position and relative orientation were specified in S120, and by specifying the specific time of the relative position and relative orientation. The position and orientation server processing unit 314 (FIG. 4) of the server device 300 selects the marker position and orientation information LI corresponding to the specified moving body ID and time included in the request from the terminal device 100, and transmits it to the terminal device 100 (S320). The marker position and orientation information LI acquired by the terminal device 100 is stored in the storage unit 120.

Next, the terminal position and orientation estimation unit 112 (FIG. 2) of the terminal device 100 estimates the position (absolute position) and orientation (absolute orientation) of the terminal device 100 (the imaging unit 154) on the three-dimensional point cloud map based on the marker position and orientation information LI acquired in S130 and the relative relationship information CI acquired in S120 (S140). As described above, the marker position and orientation information LI is information indicating the position (absolute position) and orientation (absolute orientation) of the marker unit 260 on the three-dimensional point cloud map, and the relative relationship information CI is information indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260. Therefore, the terminal position and orientation estimation unit 112 can estimate the position and orientation of the terminal device 100 on the three-dimensional point cloud map by performing coordinate transformation (transformation from the user coordinate system to the world coordinate system) using both information. The coordinate transformation used at this time will be described in detail later. In the following, the point in time when the estimation in S140 is performed is referred to as the "tracking start point," and the position and orientation of the terminal device 100 estimated in S140 are referred to as the "initial position" and "initial orientation."

Then, the tracking unit 116 (FIG. 2) of the terminal device 100 tracks changes in the position and attitude of the terminal device 100 from the initial position and initial attitude at the tracking start point, and the estimation result update unit 117 updates the estimation results of the position and attitude of the terminal device 100 based on the changes (S150). Tracking of changes in the position and attitude of the terminal device 100 by the tracking unit 116 (hereinafter also referred to as "tracking") is achieved by a known method. For example, the tracking unit 116 tracks changes in the position and attitude of the terminal device 100 using Visual-SLAM, which is one of SLAM (Simultaneous Localization and Mapping) technologies that simultaneously estimates the self-position of a moving object and creates an environmental map. That is, the tracking unit 116 creates a map of the surrounding environment (a map in the user coordinate system) based on the image captured by the imaging unit 154, estimates the amount of movement and the amount of change in attitude of the terminal device 100 in the user coordinate system based on the transition of the feature amount in the image captured by the imaging unit 154, and tracks the position and attitude of the imaging unit 154 on the surrounding environment map. In this way, tracking by the tracking unit 116 can be performed continuously even if the marker unit 260 of the autonomous moving body 200 is not present in the imaging range of the imaging unit 154. In addition, the estimation result update unit 117 updates the estimation result of the position and attitude of the imaging unit 154 of the terminal device 100 on the three-dimensional point cloud map by performing coordinate transformation (transformation from the user coordinate system to the world coordinate system) on the position and attitude of the imaging unit 154 on the surrounding environment map.

Here, the conversion from the user coordinate system to the world coordinate system will be explained. Figure 8 is an explanatory diagram showing the method of conversion from the user coordinate system to the world coordinate system. As described above, the world coordinate system is the coordinate system of the three-dimensional point cloud map, and the user coordinate system is the coordinate system used by the terminal device 100. In addition, hereinafter, the coordinate system having the position of the autonomous moving body 200 as the origin is referred to as the "moving body coordinate system", and the coordinate system having the position of the imaging unit 154 of the terminal device 100 as the origin is referred to as the "imaging unit coordinate system".

8 shows the position of the origin O _V ^W of the moving body coordinate system in the world coordinate system (origin: O ^W ) and the orientation of the moving body coordinate system, while 8 shows the position of the origin O _C ^U of the imaging unit coordinate system in the user coordinate system (origin: O ^U ) and the orientation of the imaging unit coordinate system, the position of the origin O _V ^U of the moving body coordinate system and the orientation of the moving body coordinate system, while 8 shows the position of the origin O _U ^W of the user coordinate system in the world coordinate system and the orientation of the user coordinate system, the position of the origin O _C ^W of the imaging unit coordinate system and the orientation of the imaging unit coordinate system, and the position of the origin O _V ^W of the moving body coordinate system and the orientation of the moving body coordinate system. The origin O ^U of the user coordinate system is the position of the imaging unit 154 at the tracking start point where the estimation was performed in S140. Therefore, at the tracking start point, the imaging unit coordinate system coincides with the user coordinate system. After the tracking start point, as the position and orientation of the imaging unit 154 of the terminal device 100 changes (shown by thin dashed lines in Figure 8), the origin position and orientation of the imaging unit coordinate system change from the origin position and orientation of the user coordinate system.

As shown in equation (1) below, the linear transformation matrix T _OU ^w of a point in the user coordinate system to the world coordinate system can be obtained by subtracting the vector T O U ^V from the origin O _U ^W of the user coordinate system in the world coordinate system to the origin O _V W of the mobile body coordinate system from the vector T _OV ^W from the origin O ^w of the world coordinate system to _the origin _{O V} ^W of the mobile body coordinate system ^.

Furthermore, as shown in the following equation (2), the rotation transformation matrix R _OU ^w of a point in the user coordinate system to the world coordinate system can be obtained by the quaternion product of the amount of rotation of the origin O _V ^W of the moving body coordinate system as viewed from the origin O ^w of the world coordinate system and the inverse quaternion of the amount of rotation of the moving body coordinate system as viewed from the user coordinate system.

By using these transformation matrices, at the tracking start point, the terminal position and orientation estimation unit 112 can estimate the position and orientation of the terminal device 100 on the three-dimensional point cloud map (S140), and after the tracking start point, the estimation result update unit 117 can update the estimation results of the position and orientation of the terminal device 100 on the three-dimensional point cloud map (S150). This allows the terminal device 100 to continuously grasp the position and orientation of the terminal device 100 on the three-dimensional point cloud map (i.e., the absolute position and absolute orientation in the world coordinate system) after the tracking start point.

As shown in FIG. 5, after the tracking start point, the augmented reality processing unit 111 (FIG. 2) of the terminal device 100 requests the server device 300 for a virtual image VI corresponding to the position and orientation of the terminal device 100 on the three-dimensional point cloud map, and acquires the virtual image VI from the server device 300 (S160). That is, the augmented reality processing unit 111 specifies the position and orientation of the terminal device 100 on the three-dimensional point cloud map and requests the virtual image VI from the server device 300. As described above, in the server device 300, a virtual image database ID (FIG. 4) is constructed in which a plurality of virtual images VI are set in association with the position and orientation of the terminal device 100. The augmented reality server processing unit 312 of the server device 300 selects a virtual image VI corresponding to the position and orientation of the terminal device 100 included in the request from the terminal device 100 from the virtual image database ID, and transmits the virtual image VI to the terminal device 100 (S330).

The augmented reality processing unit 111 of the terminal device 100 superimposes the virtual image VI acquired in S160 onto the real image RI captured by the imaging unit 154 and displays it on the display unit 151 (S170). For example, in the example shown in FIG. 6, a virtual image VI for guidance display (such as "Main Entrance," "Parking," and "Cafe") is displayed on the display unit 151 of the terminal device 100 held by the user U, superimposed on a real image RI showing the exterior of the area near the entrance of the shopping mall. Also, in the example shown in FIG. 7, a virtual image VI for guidance display (such as "Downstair," "Event Space," and "Food Court") is displayed on the display unit 151 of the terminal device 100 held by the user U, superimposed on a real image RI showing the interior of the shopping mall building. The user U can grasp the location of each facility in the shopping mall by referring to the virtual image VI displayed on the display unit 151.

After the tracking start point, the augmented reality processing unit 111 (FIG. 2) of the terminal device 100 monitors whether an image of the marker unit 260 has been detected from the image generated by the imaging unit 154 (S180), and monitors whether an instruction to end the AR-type information presentation process has been received (S190). If the image of the marker unit 260 has not been detected (S180: NO) and it is determined that there has been no instruction to end the process (S190: NO), the terminal device 100 repeatedly executes the above-mentioned processes from S150 to S170. That is, since the position and posture of the terminal device 100 change from moment to moment as the user U moves or changes in posture, the estimation results of the position and posture of the terminal device 100 are updated (S150), and the request/acquisition process of the virtual image VI (S160) is executed again according to the update result, and the displayed virtual image VI is updated (S170). As a result, even if the user U moves or changes in posture, an appropriate virtual image VI that matches the real image RI is displayed on the display unit 151 of the terminal device 100.

Also, when it is determined that the image of the marker unit 260 is detected in the image captured by the imaging unit 154 (S180: YES), the terminal device 100 executes the above-mentioned processing from S120 onwards again. That is, based on the image of the marker unit 260 and the image correspondence information II, the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260 are specified (S120), marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map is acquired (S130), and the position and orientation of the terminal device 100 on the three-dimensional point cloud map are estimated (S140). That is, the initial position and initial orientation of the terminal device 100 are updated. In tracking by the terminal device 100 after the tracking start point, there is a risk that estimation errors of the position and orientation of the terminal device 100 will accumulate, but the error is eliminated by updating the initial position and initial orientation of the terminal device 100, so that continuous highly accurate estimation of the position and orientation of the terminal device 100 is realized. Even after the autonomous movement of the autonomous moving body 200 has ended (S230: YES), the marker position and orientation information LI transmitted to the server device 300 immediately before the autonomous movement ended can be used, so estimation or updating of the initial position and initial orientation of the terminal device 100 can be executed using the marker position and orientation information LI. In addition, if the terminal device 100 fails to track or becomes unable to track midway, the server device 300 may present the position of the autonomous moving body 200 near the terminal device 100, or change the route of the autonomous moving body 200 to move it closer to the terminal device 100. Thereafter, the processing from S150 onwards (tracking, etc.) is executed in the same manner.

When it is determined that an instruction to end the AR-type information presentation process has been received during the execution of the above-mentioned process by the terminal device 100 (S190: YES), the augmented reality processing unit 111 of the terminal device 100 ends the AR-type information presentation process. Similarly, when it is determined that an instruction to end the AR-type information presentation process has been received in the server device 300 (S340: YES), the augmented reality server processing unit 312 of the server device 300 ends the AR-type information presentation process.

A-3. Advantages of the first embodiment:
As described above, the terminal device 100 constituting the information processing system 10 of the first embodiment is a device held by the user U and moves together with the user U, and includes the imaging unit 154, the marker information acquisition unit 113, the image correspondence information acquisition unit 114, the relative relationship specification unit 115, and the terminal position and orientation estimation unit 112. The marker information acquisition unit 113 is an autonomous moving body 200 that moves while specifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating a three-dimensional absolute position is added, and acquires marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map for the autonomous moving body 200 having a marker unit 260 that functions as a marker. The image correspondence information acquisition unit 114 acquires image correspondence information II that associates an image in which the marker unit 260 is captured with a relative position and relative orientation with respect to the marker unit 260. The relative relationship specifying unit 115 specifies relative relationship information CI indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260, based on the image of the marker unit 260 captured by the imaging unit 154 and the image correspondence information II. The terminal position and orientation estimation unit 112 estimates the position and orientation of the terminal device 100 on the three-dimensional point cloud map, based on the marker position and orientation information LI and the relative relationship information CI.

In this way, the terminal device 100 of this embodiment can estimate the position and orientation of the terminal device 100 on the three-dimensional point cloud map based on the marker position and orientation information LI indicating the position and orientation of the marker unit 260 of the autonomous moving body 200 on the three-dimensional point cloud map, and the relative relationship information CI indicating the relative position and relative orientation of the terminal device 100 with respect to the marker unit 260. Therefore, according to the terminal device 100 of this embodiment, as long as the marker unit 260 of the autonomous moving body 200 is in a position where it can be imaged by the imaging unit 154, it is possible to seamlessly estimate the position and orientation of the terminal device 100 both indoors and outdoors, without the need to install a large number of markers whose positions and orientations are known in advance, and without equipping the terminal device 100 with sensors other than the imaging unit 154 (e.g., LiDAR) or a three-dimensional point cloud map database.

The terminal device 100 of this embodiment further includes a tracking unit 116 and an estimation result update unit 117. The tracking unit 116 tracks changes in the position and attitude of the terminal device 100 from the time when the position and attitude of the terminal device 100 are estimated (tracking start point). The estimation result update unit 117 updates the estimation results of the position and attitude of the terminal device 100 based on the changes. Therefore, according to the terminal device 100 of this embodiment, once the position and attitude (initial position and initial attitude) of the terminal device 100 on the three-dimensional point cloud map is estimated based on the marker position and attitude information LI and the relative relationship information CI, the terminal device 100 alone can update the estimation results of the position and attitude of the terminal device 100, regardless of whether the marker unit 260 of the autonomous moving body 200 can be imaged by the imaging unit 154. At this time, since the estimation accuracy of the initial position and initial attitude is high, the estimation accuracy of the position and attitude of the terminal device 100 is maintained at a certain level or higher even after subsequent updates. Therefore, according to the terminal device 100 of this embodiment, the position and orientation of the terminal device 100, which moves independently of the autonomous moving body 200, can be continuously estimated with high accuracy.

In the terminal device 100 of this embodiment, the tracking unit 116 tracks changes in the position and attitude of the terminal device 100 using images captured by the imaging unit 154. Therefore, in the terminal device 100 of this embodiment, the position and attitude of the terminal device 100 can be continuously estimated with high accuracy without requiring any sensors other than the imaging unit 154.

The terminal device 100 of this embodiment further includes a display unit 151 and an augmented reality processing unit 111. The augmented reality processing unit 111 superimposes a virtual image VI, which changes according to the estimation result of the position and orientation of the terminal device 100, on a real image RI captured by the imaging unit 154 and displays it on the display unit 151. Therefore, according to the terminal device 100 of this embodiment, it is possible to display the virtual image VI based on the position and orientation of the terminal device 100 estimated with high accuracy, and it is possible to realize highly accurate AR-type information presentation.

The information processing system 10 of this embodiment also includes a terminal device 100 and an autonomous mobile body 200. The autonomous mobile body 200 is a mobile body that moves autonomously using surrounding detection information from the sensor 230 and a three-dimensional point cloud map. Therefore, according to the information processing system 10 of this embodiment, by moving the marker unit 260 without requiring human driving operation, it is possible to realize highly accurate estimation of the position and attitude of the terminal device 100 over a wide range using the moving marker unit 260.

The information processing system 10 of this embodiment further includes a server device 300. The autonomous moving body 200 transmits the marker position and orientation information LI to the server device 300, and the marker information acquisition unit 113 of the terminal device 100 acquires the marker position and orientation information LI via the server device 300. Therefore, according to the information processing system 10 of this embodiment, even in a terminal device 100 that cannot directly communicate with the autonomous moving body 200, it is possible to realize highly accurate estimation of the position and orientation of the terminal device 100 using the marker unit 260 of the autonomous moving body 200.

B. Second embodiment:
Fig. 9 is an explanatory diagram that shows a schematic configuration of the information processing system 10a in the second embodiment, Fig. 10 is a block diagram that shows a schematic configuration of the terminal device 100a in the second embodiment, Fig. 11 is a block diagram that shows a schematic configuration of the autonomous moving body 200a in the second embodiment, and Fig. 12 is a block diagram that shows a schematic configuration of the server device 300a in the second embodiment. In the following, among the configurations of the information processing system 10a, the terminal device 100a, the autonomous moving body 200a, and the server device 300a of the second embodiment and the processing contents by each device, the same configurations and processing contents as those of the first embodiment described above are appropriately omitted by assigning the same reference numerals.

The information processing system 10a of the second embodiment is a system that estimates the position and orientation of the terminal device 100a on a three-dimensional point cloud map with high accuracy, like the information processing system 10 of the first embodiment, but it is a system that uses the estimated position and orientation to perform hazard notification rather than AR-type information presentation. In this embodiment, an example of hazard notification that notifies the terminal device 100a of the presence of a hazard close to the terminal device 100a at an intersection with poor visibility on a public road will be described.

As shown in FIG. 9, the information processing system 10a includes a plurality of terminal devices 100a, a plurality of autonomous moving bodies 200a, and a server device 300a. In the second embodiment, the autonomous moving body 200a is an autonomous vehicle that carries a passenger and autonomously travels on a public road.

As shown in FIG. 10, the storage unit 120 of the terminal device 100a in the second embodiment stores a hazard notification program CP12 for performing the hazard notification process described below. The control unit 110 of the terminal device 100a reads out the hazard notification program CP12 from the storage unit 120 and executes it to function as a hazard notification unit 119 that performs the hazard notification process described below. The hazard notification unit 119 includes a terminal position and orientation estimation unit 112a similar to that of the first embodiment. However, the terminal position and orientation estimation unit 112a in the second embodiment includes an estimation result transmission unit 118. In addition, the storage unit 120 of the terminal device 100a stores hazard information HI during the hazard notification process described below.

As shown in FIG. 11, the storage unit 220 of the autonomous mobile body 200a in the second embodiment stores a hazard notification program CP23 for performing the hazard notification processing described below. The control unit 210 of the autonomous mobile body 200a reads out the hazard notification program CP23 from the storage unit 220 and executes it to function as a hazard notification unit 216 that performs the hazard notification processing described below. In addition, the storage unit 220 of the autonomous mobile body 200a stores terminal position and orientation information TI during the hazard notification processing described below. Note that the hazard notification processing is an example of a specific processing in the claims, and the hazard notification unit 216 is an example of a specific processing execution unit in the claims.

As shown in FIG. 12, the memory unit 320 of the server device 300a in the second embodiment stores a hazard notification server program CP32 for performing the hazard notification processing described below. The control unit 310 of the server device 300a reads out the hazard notification server program CP32 from the memory unit 320 and executes it to function as a hazard notification server processing unit 318 that performs the hazard notification processing described below. The hazard notification server processing unit 318 includes a position and orientation server processing unit 314 similar to that of the first embodiment. In addition, the memory unit 320 of the server device 300a stores terminal position and orientation information TI and hazard information HI during the hazard notification processing described below.

Next, the hazard notification process executed in the information processing system 10a of the second embodiment will be described. The hazard notification process of this embodiment is a process in which the autonomous mobile body 200a notifies the terminal device 100a of the presence of a hazard close to the terminal device 100a, and the hazard is reported by the audio output unit 153 of the terminal device 100a. In this embodiment, the hazard notification process is described as an example in which the terminal device 100a is notified of the presence of a hazard close to the terminal device 100a at an intersection with poor visibility on a public road, etc.

Fig. 13 is an explanatory diagram showing the flow of the hazard notification process executed in the information processing system 10a of the second embodiment. Fig. 14 is an explanatory diagram conceptually showing the hazard notification process executed in the information processing system 10a of the second embodiment. In the example shown in Fig. 14, an autonomous mobile body 200a is traveling in the imaging range of the imaging unit 154 of the terminal device 100a held by a user U at an intersection with poor visibility on a public road. In addition, in the shadow of a building Bu, there is a bicycle as a hazard Ha that is not visible to the user U but can be recognized by the sensor 230 of the autonomous mobile body 200a, and the bicycle is about to jump out in front of the user U.

In the hazard notification process shown in FIG. 13, the processing of S210 and S220 by the autonomous mobile body 200a, the processing of S310 and S320 by the server device 300a, and the processing of S110 to S140 by the terminal device 100a are the same as the processing of each step in the AR-type information presentation process of the first embodiment shown in FIG. 5. Through these processes, the terminal device 100a can estimate with high accuracy the position (initial position) and attitude (initial accuracy) of the terminal device 100a (the imaging unit 154) on the three-dimensional point cloud map.

Then, the estimation result transmission unit 118 (FIG. 10) of the terminal device 100a transmits the terminal position and orientation information TI indicating the estimation result of the position and orientation of the terminal device 100a on the three-dimensional point cloud map to the autonomous mobile body 200a via the server device 300a. That is, the estimation result transmission unit 118 transmits the terminal position and orientation information TI to the server device 300a (S142), the position and orientation server processing unit 314 (FIG. 12) of the server device 300a transmits the received terminal position and orientation information TI to the autonomous mobile body 200a (S322), and the hazard notification unit 216 (FIG. 11) of the autonomous mobile body 200a receives the terminal position and orientation information TI (S222). The terminal position and orientation information TI includes a terminal ID for identifying the terminal device 100a, a timestamp indicating the estimated time of the position and orientation, and information for identifying the position and orientation (see FIG. 12). In this embodiment, the terminal position and orientation information TI is transmitted to an autonomous moving body 200a that is within a predetermined distance from the estimated position of the terminal device 100a (i.e., that is close to the terminal device 100a).

After transmitting the terminal position and attitude information TI, the terminal device 100a updates the estimated results of the position and attitude of the terminal device 100a based on changes in the position and attitude of the terminal device 100a from the initial position and initial attitude (S150), as in the first embodiment, and determines whether hazard information HI has been received (S152).

The hazard notification unit 216 (FIG. 11) of the autonomous mobile body 200a that has received the terminal position and orientation information TI identifies the position and orientation of the terminal device 100a on the three-dimensional point cloud map based on the terminal position and orientation information TI, and searches for hazards Ha around the terminal device 100a (S224). That is, the hazard notification unit 216 determines the presence or absence of a surrounding object that could be a hazard Ha for the terminal device 100a based on the relative positional relationship and relative speed relationship between the surrounding object recognized by the sensor 230 and the terminal device 100a.

When the hazard notification unit 216 of the autonomous mobile body 200a detects a hazard Ha in the vicinity of the terminal device 100a (S224: YES), it creates hazard information HI that identifies the detected hazard Ha, and transmits the hazard information HI to the target autonomous mobile body 200a via the server device 300a. That is, the hazard notification unit 216 transmits the hazard information HI to the server device 300a (S226), the hazard notification server processing unit 318 (FIG. 12) of the server device 300a transmits the received hazard information HI to the autonomous mobile body 200a (S324), and the hazard notification unit 119 (FIG. 10) of the terminal device 100a receives the hazard information HI (S152: YES). The hazard information HI is, for example, information indicating the position, speed, name, etc. of the hazard Ha. If a hazard Ha is not detected in the autonomous moving body 200a (S224: NO), processing of S226 is skipped.

The hazard notification unit 119 of the terminal device 100a that has received the hazard information HI executes a notification process to notify the user U of the presence of the hazard Ha (S154). For example, in the example of FIG. 14, the voice output unit 153 of the terminal device 100a outputs a voice saying "attention!". Through the notification process, the user U of the terminal device 100a can recognize the presence of the hazard Ha that is not visible to the user U but is recognized by the autonomous moving body 200a. In addition, in the notification process, the direction of the hazard Ha as seen from the terminal device 100a (for example, forward) or the name of the hazard Ha (for example, bicycle) may be notified. In addition, instead of or together with the voice notification, an image notification or a vibration notification may be executed. If the terminal device 100a does not receive the hazard information HI (S152: NO), the process of S154 is skipped. Subsequent processing is the same as in the first embodiment.

As described above, the information processing system 10a of the second embodiment includes the terminal device 100a and the autonomous mobile body 200a. The terminal device 100a includes an estimation result transmission unit 118 that transmits the estimation result of the position and attitude of the terminal device 100a to the autonomous mobile body 200a directly or via another device. The autonomous mobile body 200a includes a hazard notification unit 216 that executes a hazard notification process to notify the terminal device 100a of the presence of a hazard Ha close to the terminal device 100a using the estimation result of the position and attitude of the terminal device 100a. Therefore, according to the information processing system 10a of the second embodiment, the autonomous mobile body 200a can notify the terminal device 100a of the presence of a hazard Ha by using the position and attitude of the terminal device 100a on the three-dimensional point cloud map estimated with high accuracy, thereby realizing a highly accurate hazard notification.

C. Variations:
The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the spirit of the invention. For example, the following modifications are also possible.

The configuration of the information processing system 10 in the above embodiment is merely an example, and various modifications are possible. For example, in the above embodiment, the moving body having the marker unit 260 is the autonomous moving body 200 that moves autonomously using surrounding detection information from the sensor 230 and a three-dimensional point cloud map, but as long as it is a moving body that moves while identifying its own position and orientation on the three-dimensional point cloud map, it does not necessarily have to be an autonomous moving body, and may be a moving body that moves according to a driving operation by a person. Also, in the above embodiment, the autonomous moving body 200 estimates its own position and orientation by matching the surrounding recognition result from the sensor 230 with the three-dimensional point cloud map, but it may be combined with other methods (for example, a method using Visual-SLAM or GNSS) to estimate its own position and orientation with higher accuracy.

In the above embodiment, the autonomous moving body 200 has an image attached to the side of the body as the marker unit 260, but the form of the marker unit 260 is not limited to this. For example, the marker unit 260 may be a three-dimensional object, and the shape of the autonomous moving body 200 itself may function as the marker unit 260.

The contents of the AR type information presentation process and the hazard notification process in the above embodiment are merely examples and can be modified in various ways. For example, in the above embodiment, the position and orientation information providing unit 214 of the autonomous mobile body 200 transmits the marker position and orientation information LI indicating the position and orientation of the marker unit 260 on the three-dimensional point cloud map to the server device 300 (S220 in FIG. 5), and the position and orientation server processing unit 314 of the server device 300 transmits the marker position and orientation information LI to the terminal device 100 (S320). However, the position and orientation information providing unit 214 of the autonomous mobile body 200 may transmit information indicating the position and orientation of the autonomous mobile body 200 (the sensor 230) on the three-dimensional point cloud map, instead of the marker position and orientation information LI, to the server device 300, and the position and orientation server processing unit 314 of the server device 300 may generate the marker position and orientation information LI indicating the position and orientation of the marker unit 260 based on the information and transmit it to the terminal device 100. In addition, in the above embodiment, the exchange of information such as the marker position and orientation information LI between the autonomous moving body 200 and the terminal device 100 is performed via the server device 300, but the exchange of information may be performed directly between the autonomous moving body 200 and the terminal device 100.

In the above embodiment, tracking of changes in the position and attitude of the terminal device 100 after the tracking start point is performed using Visual-SLAM, but tracking may also be performed using other SLAM such as LiDAR-SLAM, or other known methods such as a method using GNSS.

In the above first embodiment, an AR-type information presentation that displays guidance both indoors and outdoors in a shopping mall has been described as an example, but the technology disclosed in this specification can also be applied to AR-type information presentation for other purposes (e.g., walking navigation, tourist information, maintenance and inspection guides, location games, etc.).

In the second embodiment, the autonomous mobile body 200a that has received the terminal position and orientation information TI executes a hazard notification process to notify the terminal device 100a of the presence of a hazard Ha close to the terminal device 100a using the estimation results of the position and orientation of the terminal device 100a. However, the autonomous mobile body 200a that has received the terminal position and orientation information TI may execute other processes (for example, precise guidance processing of the mobile body) using the estimation results of the position and orientation of the terminal device 100a. Conversely, when a hazard for the autonomous mobile body 200a is recognized by the imaging unit 154 of the terminal device 100a, the terminal device 100a may notify the autonomous mobile body 200a of information on the hazard together with the terminal position and orientation information TI. This allows the terminal device 100a to supplement the recognition of the hazard by the autonomous mobile body 200a, thereby improving the safety of the traveling of the autonomous mobile body 200a.

In the above embodiment, some of the configurations realized by hardware may be replaced by software, and conversely, some of the configurations realized by software may be replaced by hardware.

10: Information processing system 100: Terminal device 110: Control unit 111: Augmented reality processing unit 112: Terminal position and orientation estimation unit 113: Marker information acquisition unit 114: Image correspondence information acquisition unit 115: Relative relationship determination unit 116: Tracking unit 117: Estimation result update unit 118: Estimation result transmission unit 119: Hazard notification unit 120: Memory unit 151: Display unit 152: Operation input unit 153: Audio output unit 154: Imaging unit 155: Communication unit 190: Bus 200: Autonomous moving body 210: Control unit 212: Autonomous movement control unit 214: Position and orientation information provision unit 216: Hazard notification unit 220: Memory unit 230: Sensor 250: Drive unit 255: Communication unit 260: Marker unit 300: Server device 310: Control unit 312: Augmented reality server processing unit 314: Position and orientation server processing unit 318: Hazard notification server processing unit 320: Memory unit 351: Display unit 352: Operation input unit 355: Communication unit 390: Bus Ha: Hazard RI: Real image U: User VI: Virtual image

Claims (10)

A terminal device carried by a user and moving with the user,
An imaging unit;
a marker information acquisition unit that acquires marker position and orientation information indicating the position and orientation of a marker unit on a three-dimensional point cloud map, the marker information acquisition unit being configured to acquire marker position and orientation information indicating the position and orientation of a marker unit on the ...
an image correspondence information acquisition unit that acquires image correspondence information that associates an image of the marker unit with a relative position and a relative orientation of the imaging unit with respect to the marker unit;
a relative relationship specifying unit that specifies relative relationship information indicating a relative position and a relative attitude of the terminal device with respect to the marker unit based on an image of the marker unit captured by the imaging unit and the image correspondence information;
a terminal position and orientation estimation unit that estimates a position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information;
A terminal device comprising: The terminal device according to claim 1, further comprising:
a tracking unit that tracks changes in the position and attitude of the terminal device from the time when the position and attitude of the terminal device are estimated;
an estimation result update unit that updates an estimation result of the position and attitude of the terminal device based on the change;
A terminal device comprising: The terminal device according to claim 2,
The tracking unit tracks changes in position and attitude of the terminal device using images captured by the imaging unit. The terminal device according to any one of claims 1 to 3, further comprising:
A display unit;
an augmented reality processing unit that displays, on the display unit, a virtual image that changes according to the estimation result of the position and attitude of the terminal device, superimposed on a real image captured by the imaging unit;
A terminal device comprising: A terminal device according to any one of claims 1 to 4,
The moving body;
Equipped with
The terminal device further includes an estimation result transmission unit that transmits an estimation result of a position and an attitude of the terminal device to the moving object directly or via another device,
The mobile object includes a specific process execution unit that executes a specific process using an estimation result of the position and attitude of the terminal device. 6. The information processing system according to claim 5,
The specific processing includes a hazard notification process for notifying the terminal device of the presence of a hazard in the vicinity of the terminal device. 7. The information processing system according to claim 5,
The information processing system, wherein the moving body is an autonomous moving body that moves autonomously using surrounding detection information by a sensor and the three-dimensional point cloud map. The information processing system according to any one of claims 5 to 7, further comprising:
A server device is provided,
the moving object transmits the marker position and orientation information to the server device;
The marker information acquisition unit of the terminal device acquires the marker position and orientation information via the server device. An information processing method for estimating a position and an attitude of a terminal device having an imaging unit and held by a user and moving together with the user, comprising:
A step of acquiring marker position and orientation information indicating a position and orientation of a marker unit on a three-dimensional point cloud map, the marker unit being a moving body that moves while identifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating three-dimensional absolute positions is assigned, the moving body having a marker unit that functions as a marker;
acquiring image correspondence information that associates an image of the marker unit with a relative position and a relative orientation of the imaging unit with respect to the marker unit;
determining relative relationship information indicating a relative position and a relative attitude of the terminal device with respect to the marker unit based on an image of the marker unit captured by the imaging unit and the image correspondence information;
estimating a position and orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information;
An information processing method comprising: A computer program for estimating a position and an attitude of a terminal device having an imaging unit and held by a user and moving together with the user,
The terminal device,
A process of acquiring marker position and orientation information indicating a position and orientation of a marker unit on a three-dimensional point cloud map, the marker unit being a moving body that moves while identifying its own position and orientation on a three-dimensional point cloud map composed of a point cloud to which information indicating three-dimensional absolute positions is assigned, the moving body having a marker unit that functions as a marker;
A process of acquiring image correspondence information that associates an image of the marker unit with a relative position and a relative orientation of the imaging unit with respect to the marker unit;
A process of identifying relative relationship information indicating a relative position and a relative attitude of the terminal device with respect to the marker unit based on an image of the marker unit captured by the imaging unit and the image correspondence information;
A process of estimating a position and an orientation of the terminal device on the three-dimensional point cloud map based on the marker position and orientation information and the relative relationship information;
A computer program that executes the following: