JP7523055B2 - Terminal device, navigation method, and navigation program - Google Patents

Description

The present disclosure relates to a terminal device, a navigation method, and a navigation program.

Patent document 1 describes a mobile terminal with a navigation function that includes a camera and a display for displaying an image captured by the camera. The mobile terminal includes a first detection means for detecting the current position of the mobile terminal, a second detection means for detecting the orientation of the camera, a setting means for setting a destination, an acquisition means for acquiring information about a route from the current position to the destination from a server based on the destination set by the setting means and the current position detected by the first detection means, a calculation means for calculating a direction to travel on an image captured by the camera based on the information about the route acquired by the acquisition means and the orientation of the camera detected by the second detection means, and a means for adding an indicator indicating the direction to travel calculated by the calculation means to the image captured by the camera and displaying the indicator on the display.

JP 2006-194665 A

The present disclosure provides a terminal device, a navigation method, and a navigation program that can navigate a user appropriately.

The present disclosure provides a terminal device capable of performing navigation, comprising a processor, a screen, a camera, and a memory, wherein the processor superimposes an image captured by the camera and a navigation object on the screen, and when it detects that a user of the terminal device is using an elevator means on the navigation route, it displays information on the screen indicating that elevator means is in use and hides the navigation object.
The present disclosure also provides a terminal device capable of performing navigation, comprising a processor, a screen, a camera, and a memory, wherein the processor superimposes an image captured by the camera and a navigation object on the screen, calculates a relative position of the terminal device, which is pointed toward a navigation start position that is at a predetermined height from a floor surface, with respect to the floor surface, and displays the navigation object so that it is at the predetermined height from the floor surface displayed on the screen.
The present disclosure also provides a terminal device capable of performing navigation, comprising a processor, a screen, a camera, and a memory, wherein the processor superimposes an image captured by the camera and a navigation object on the screen, and when a plane is detected by the camera while a navigation function is being executed, determines whether the plane corresponds to a floor surface, and if the plane corresponds to a floor surface, calculates a relative position of the terminal device with respect to the plane, and changes the height of the navigation object to be within a certain height range with respect to the height of the camera relative to the plane.
The present disclosure also provides a terminal device capable of performing navigation, comprising a processor, a screen, a camera, and a memory, wherein the processor superimposes an image captured by the camera and a navigation object on the screen, extracts feature point information from the image captured by the camera and stores it in the memory, obtains the feature point information from the memory at a predetermined timing, and adjusts a display position of the navigation object on the screen based on the obtained feature point information.

The present disclosure also provides a navigation method by a terminal device capable of performing navigation , the terminal device having a processor, a screen, and a camera, the navigation method comprising the steps of: causing the processor to superimpose an image captured by the camera and a navigation object on the screen; and, when the processor detects that a user of the terminal device is using an elevator means on the navigation route, causing the processor to display, on the screen, information indicating that elevator means is in use and hiding the navigation object.
The present disclosure also provides a navigation method using a terminal device, the terminal device having a processor, a screen, and a camera, the navigation method including the steps of: causing the processor to superimpose an image captured by the camera and a navigation object on the screen; and causing the processor to calculate a relative position of the terminal device, which is pointed toward a navigation start position that is at a predetermined height from a floor surface, with respect to the floor surface, and displaying the navigation object so that it is at the predetermined height from the floor surface displayed on the screen.
The present disclosure also provides a navigation method by a terminal device, the terminal device having a processor, a screen, and a camera, the navigation method comprising the steps of: causing the processor to superimpose an image captured by the camera and a navigation object on the screen; determining, when the processor detects a plane by the camera while a navigation function is being executed, whether the plane corresponds to a floor surface; and, if the plane corresponds to a floor surface, calculating a relative position of the terminal device with respect to the plane, and changing the height of the navigation object so that it falls within a certain height range with respect to the height of the camera relative to the plane.
The present disclosure also provides a navigation method by a terminal device, the terminal device including a processor, a screen, a camera , and a memory , the navigation method including the steps of: the processor superimposing an image captured by the camera and a navigation object on the screen; the processor extracting feature point information from the image captured by the camera and storing it in the memory; the processor acquiring the feature point information from the memory at a predetermined timing; and adjusting a display position of the navigation object on the screen based on the acquired feature point information.

The present disclosure also provides a navigation program that causes a terminal device capable of performing navigation, the terminal device having a processor, a screen, and a camera, to execute a function of superimposing an image captured by the camera and a navigation object on the screen, and a function of, when it is detected that a user of the terminal device is using an elevator means on the navigation route, displaying information on the screen indicating that elevator means is in use and hiding the navigation object.
The present disclosure also provides a navigation program that causes a terminal device having a processor, a screen, and a camera to execute a function of superimposing an image captured by the camera and a navigation object on the screen, and a function of calculating a relative position of the terminal device, which is pointed toward a navigation start position that is at a predetermined height from a floor surface, with respect to the floor surface, and displaying the navigation object so that it is at a predetermined height from the floor surface displayed on the screen.
The present disclosure also provides a navigation program that causes a terminal device having a processor, a screen, and a camera to execute the following functions: a function of superimposing an image captured by the camera and a navigation object on the screen; a function of determining whether or not a plane detected by the camera while a navigation function is being executed corresponds to a floor surface; and a function of calculating a relative position of the terminal device with respect to the plane if the plane corresponds to the floor surface, and changing the height of the navigation object so that it falls within a certain height range with respect to the height of the camera relative to the plane.
The present disclosure also provides a navigation program that causes a terminal device having a processor, a screen, a camera , and a memory to execute the following functions: a function of superimposing an image captured by the camera and a navigation object on the screen; a function of extracting feature point information from the image captured by the camera and storing it in the memory; and a function of the processor acquiring the feature point information from the memory at a predetermined timing, and adjusting a display position of the navigation object on the screen based on the acquired feature point information.

This disclosure allows users to navigate appropriately.

FIG. 1 is a conceptual diagram illustrating a terminal device 100 according to an embodiment of the present disclosure. FIG. 1 is a conceptual diagram showing an example of a system configuration of a terminal device 100. FIG. 1 is a conceptual diagram showing a method for starting AR navigation using a terminal device 100 according to an embodiment of the present disclosure. 13 is a display example of the display 106 when the terminal device 100 acquires an ID from the digital signage 10. An example of a destination list 204 displayed on the display 106 13 is a display example of the display 106 when detecting the initial height of the terminal device 100. 1 is a display example of the display 106 when the current height and orientation of the terminal device 100 are identified. An example of a display on the display 106 when AR navigation starts 1 is a display example on the display 106 when a user who had been pointing the camera 103 of the terminal device 100 at the digital signage 10 turns the camera 103 around by 180 degrees. 13 is a display example of the first mode of vertical movement warning on the display 106. 13 is a display example of the second mode of vertical movement warning on the display 106. 13 is a display example of the third mode of the vertical movement warning on the display 106. 5A and 5B are conceptual diagrams illustrating orientation correction in an AR space of the terminal device 100, (a) before correction, and (b) after correction. Example of display on the display 106 when AR navigation is completed FIG. 1 is a block diagram showing an example of internal processing of the terminal device 100 when performing initial height detection. FIG. 1 is a block diagram showing an example of internal processing of the terminal device 100 when performing height correction processing. FIG. 11 is a block diagram showing an example of internal processing of the terminal device 100 when performing a second mode of height correction processing; FIG. 1 is a block diagram showing an example of internal processing of the terminal device 100 when issuing a vertical movement warning. FIG. 1 is a block diagram showing an example of internal processing of the terminal device 100 when performing a first mode of vertical movement correction (orientation correction); FIG. 11 is a block diagram showing an example of internal processing of the terminal device 100 when performing a second mode of vertical movement correction (position correction).

(Background to this disclosure)
There is known a navigation system that uses a positioning system such as GPS to guide a person, a vehicle, etc. to a destination. In such a navigation system, a device receives radio waves transmitted from an artificial satellite in space, calculates the current location of the device, and guides (navigates) a user of the device (a person, a vehicle, etc.) to the destination.

However, there are places where the radio waves used to identify the current location of a device have difficulty reaching the device, or where the radio waves do not reach the device at all, such as inside train stations or underground shopping malls. In such places, conventional navigation systems have been unable to correctly guide users carrying the device.

The following embodiments therefore describe embodiments of a terminal device, a navigation method, and a navigation program that can navigate a user appropriately even indoors or in other places where radio waves for identifying the current location are difficult or impossible to reach.

Below, with reference to the drawings as appropriate, a detailed description will be given of an embodiment specifically disclosing the terminal device, navigation method, and navigation program according to the present disclosure. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters and duplicate description of substantially identical configurations may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

FIG. 1 is a conceptual diagram illustrating a terminal device 100 according to an embodiment of the present disclosure. FIG. 2 is a conceptual diagram illustrating an example system configuration of the terminal device 100.

The terminal device 100 is a device that can be moved with the user, and may be, for example, a smartphone, a tablet terminal, a notebook computer, a head-mounted display, etc. In this example, the following explanation will be given on the assumption that the terminal device 100 is a smartphone.

The terminal device 100 in this example includes a processor 101, a memory 102, a camera 103, an acceleration sensor 104, an air pressure sensor 105, a display 106, and a network interface 107.

The processor 101 is configured using, for example, a CPU or FPGA, and works with the memory 102 to perform various processes and controls in the terminal device 100. Specifically, the processor 101 references a program stored in the memory 102 and executes the program to realize various functions of the terminal device 100. Note that the program for the terminal device 100, which is a smartphone, may be a smartphone app.

The memory 102 has, for example, a RAM as a work memory used when executing the processing of the processor 101, and a ROM that stores a program that defines the processing of the processor 101. The RAM temporarily stores data generated or acquired by the processor 101. The ROM has written therein a program that defines the processing of the processor 101. Program data for causing the terminal device 100, which is a smartphone, to execute various functions and data used by the program may be stored in the memory 102.

The memory 102 may further store AR navigation data, which will be described later. The AR navigation data may be pre-stored in the memory 102, or the AR navigation data may be downloaded from the server 20 or the like and stored in the memory 102.

The camera 103 may be an imaging camera that captures an image of a subject (such as a person, a building, or a landscape) included in the desired imaging range of the terminal device 100. The camera 103 may also be capable of reading an identification code such as a barcode or a two-dimensional code. When providing navigation to a user using the terminal device 100, the terminal device 100 may superimpose an AR object for navigation (described below) on the image captured by the camera 103 and display it on the display 106 (AR navigation).

The acceleration sensor 104 detects the acceleration of the terminal device 100. The items measured by the acceleration sensor 104 may include gravitational acceleration, and the acceleration sensor 104 can detect human movement, vibration, impact, and the like. For example, the acceleration sensor 104 may be a three-axis acceleration sensor having X, Y, and Z axes. To obtain more precise sensor information, the acceleration sensor 104 may be a six-axis sensor capable of detecting not only acceleration but also angular velocity.

The air pressure sensor 105 detects the air pressure around the terminal device 100. By measuring the air pressure around the terminal device 100 with the air pressure sensor 105, the terminal device 100 can obtain the altitude of the terminal device 100's current position.

Display 106 displays information to the user of terminal device 100. Display 106 may be a touch panel display that has both an input function, whereby the user inputs information by touching with a finger or a stylus, and a display function, whereby information is displayed. If terminal device 100 is, for example, a notebook computer, display 106 may correspond to both an input means, such as a keyboard or a mouse, and a display means, such as an LCD monitor. Display 106 and display means, such as the notebook computer's LCD monitor, are sometimes referred to as "screens."

The network interface 107 is an interface for communicating with the outside of the terminal device 100. For example, if the terminal device 100 is a smartphone, as shown in FIG. 2, the terminal device 100 can communicate with an external device such as a server 20 via a base station BS. The network in this case may be a mobile network, Bluetooth (registered trademark), or Wi-Fi. However, there is no intention to exclude network interfaces other than these.

When the terminal device 100 is a notebook PC or the like, the network interface 107 may be, for example, a wired LAN adapter or a wireless LAN adapter. Also, the communication line may be, for example, an IP network.

The above-mentioned components of the terminal device 100 are not necessarily all essential, and some components may be omitted. Conversely, the terminal device 100 may include other components. For example, the terminal device 100 may include various sensors such as a proximity sensor and an ambient light sensor.

FIG. 3 is a conceptual diagram showing a method for starting AR navigation using a terminal device 100 according to an embodiment of the present disclosure. For navigation (guiding a user) in the present disclosure, AR (Augmented Reality) technology is used. An AR object (e.g., an arrow indicating the direction of travel toward the destination) is superimposed on an image captured by the camera 103 and displayed on the screen (display 106) of the terminal device 100.

As described above, the navigation in this disclosure is premised on implementation in places such as train stations where it is difficult or impossible to receive radio waves for a positioning system such as GPS. Even if radio waves for the positioning system cannot be received and the current position and current orientation of the terminal device 100 based on the positioning system cannot be identified, for example, LinkRay (registered trademark) is used to determine the initial height and orientation of the terminal device 100 relative to the starting point of the AR navigation.

LinkRay (registered trademark) is the name of a service that uses a camera 103 mounted on a smartphone, tablet terminal, or the like, and a dedicated application to rapidly receive visible light signals (specifically, signals on which the information or data to be transmitted is superimposed) transmitted by rapidly blinking an LED (Light Emitting Diode) light source. The dedicated application may be pre-installed in an executable manner on the terminal device 100 and stored in the memory 102. With LinkRay (registered trademark), when a user holds the terminal device 100 over an LED light source that is the target of the LinkRay (registered trademark) service, the terminal device 100 can obtain the information or data transmitted from the LED light source.

As shown in FIG. 3, an electronic signboard 10 is placed as a light source device on a wall or a pillar in a station. The electronic signboard 10 is placed almost perpendicular to the floor surface FL. A mark 11 is displayed on the electronic signboard 10. A user of the terminal device 100 points the camera 103 of the terminal device 100 at the electronic signboard 10. Then, the terminal device 100 receives a visible light signal from the electronic signboard 10 by the above-mentioned LinkRay (registered trademark) technology. This visible light signal contains an ID, which is identification information. The ID, which is identification information, is different for each light source device (electronic signboard 10, etc.). For convenience of explanation, the ID of the electronic signboard 10 shown in FIG. 3 is assumed to be 111111. The downloading of information from the server 20 to the terminal device 100 will be described later.

Figure 4 is an example of a display on the display 106 when the terminal device 100 acquires an ID from the electronic signage 10. A mark 201, a download progress display bar 202, and a rectangular frame 203 are displayed on the screen (display 106). The mark 201 is similar to the mark 11 in Figure 1, so a description thereof will be omitted. The download progress display bar 202 indicates to the user the progress of acquiring the ID from the electronic signage 10. The rectangular frame 203 is an alignment frame displayed so that the user can correctly align the terminal device 100 with respect to the mark 201. Note that the fact that the terminal device 100 has detected the mark 201 for acquiring the ID may be indicated to the user by, for example, changing the color of the rectangular frame 203 to yellow.

The terminal device 100 that has acquired the ID transmits the above-mentioned identification information (ID) to a LinkRay platform (server) (not shown). Content destination information such as a URL is transmitted from the LinkRay platform (server) to the terminal device 100. This content destination information may include multiple types of information, each indicating one or multiple destinations, with the position of the electronic signboard 10 as the starting position of navigation.

Figure 5 is an example of a destination list 204 displayed on the display 106. Information for displaying this destination list 204 may be included in the content destination information described above. In addition, since the content destination information is acquired based on the ID of the electronic signage 10, all of the destinations included in the destination list 204 may be destinations when the location where the electronic signage 10 is installed is set as the starting location of navigation. The user taps the display 106 (touch panel display) of the terminal device 100 to select the destination to which they wish to go. The terminal device 100 then receives (downloads) the specified content from the server 20 based on the acquired content destination information. For ease of explanation, the explanation will be given on the assumption that the user has selected "XXX Center 1F" as the destination for AR navigation.

(Contents downloaded from the server 20 to the terminal device 100)
The terminal device 100 can download designated content from the server 20 based on an ID, which is identification information that differs for each light source device (such as the digital signage 10). The downloaded content (hereinafter, sometimes referred to as DL content) may include, for example, the following information:
Feature amount data (DATA1) indicating starting point information (such as the mark 201) in a light source device (such as the electronic signboard 10) identified by an ID.
Position information data (DATA2) indicating the installation height and installation direction of the origin information (such as the mark 201) in the light source device (such as the electronic signboard 10) identified by the ID.
AR navigation data (DATA3) from a light source device (such as the electronic signboard 10) identified by the ID to a destination.

(Other embodiments for content acquisition)
The acquisition of AR navigation data using LinkRay (registered trademark) has been described. However, for example, by arranging information that substitutes for the above-mentioned ID, such as a two-dimensional code, on a fixed position such as a wall or a pillar that is the navigation start point, the terminal device 100 can acquire the desired AR navigation data from the server 20 in the same manner as described above. In addition, a short-range communication device (an example of an information communication device) that transmits and receives information or data using short-range communication other than the above may be grounded on a fixed position such as a wall or a pillar that is the navigation start point. The short-range communication may be, for example, a beacon terminal. The beacon terminal transmits a signal containing information equivalent to the above-mentioned ID using, for example, BLE (Bluetooth (registered trademark) Low Energy), which is a communication standard. The beacon terminal transmits a signal containing information equivalent to the ID once every few seconds within a radius of several tens of centimeters to several tens of meters starting from the installation position of the terminal itself.

(Initial Height Detection of Terminal Device 100)
6 is a display example on the display 106 when detecting the initial height of the terminal device 100. Following the object selection (see FIG. 5), the user points the camera 103 of the terminal device 100 toward the mark 201 again (see FIG. 6). The terminal device 100 then acquires the mark 201 (starting point information) by the camera 103 (taking an image of the mark in this example), and calculates the relative position of the terminal device 100 with respect to the floor surface FL (see FIG. 3) based on the distortion of the shape of the mark 201 in the captured image. This relative position can be calculated using the position of the mark 201 displayed on the electronic signage 10. The mark 201 is displayed at a predetermined height from the floor surface FL. The position where the mark 201 is displayed is set as the AR navigation start position. The positional relationship between this AR navigation start position (the position where the mark 201 is displayed) and the floor surface FL is determined in advance. On the other hand, the terminal device 100 can specify the relative positional relationship between the position where the mark 201 is displayed and the camera 103 based on the distortion of the shape of the mark 201 in the captured image. Therefore, the terminal device 100 can calculate the current height (from the floor surface FL) and orientation of the terminal device 100 based on the AR navigation start position whose positional relationship with the floor surface FL is fixed and the above-mentioned distortion of the shape of the mark 201.

In the above example, the mark 201 is used as information (starting point information) that serves as a starting point for calculating the relative position of the terminal device 100 with respect to the floor surface FL (see FIG. 3). However, the starting point information may be information other than the mark 201. For example, the shape or pattern of an object placed at the navigation start position, an identification code placed on the object, etc. may also be the starting point information for calculating the relative position of the terminal device 100 with respect to the floor surface FL.

Figure 15 is a block diagram showing an example of internal processing of the terminal device 100 when performing the above-mentioned initial height detection. In the figure, "screen" corresponds to the display 106, "camera" corresponds to the camera 103, "6-axis sensor" corresponds to the acceleration sensor 104, and "server" corresponds to the server 20. The "AR engine" may be implemented in the processor 101. Also, a chip for the AR engine or the like may be implemented in the terminal device 100 and may be part of the processor 101. The above-mentioned correspondences are the same for Figures 15 to 20, so explanations of the above-mentioned correspondences will be omitted for Figures 16 to 20.

The DL content acquired from the server 20 may include mark height information (included in DATA2) and mark feature information (included in DATA1). The AR engine draws the route based on the image information acquired by the camera. This route drawing may be a process of superimposing an AR object for navigation (e.g., reference numeral 207 in FIG. 9) on the image information acquired by the camera. The AR object for navigation may be generated by the AR engine based on AR navigation data (included in DATA3) included in the DL content.

In order to display the navigation AR object on the screen (display 106) at a predetermined height from, for example, the floor surface FL, the terminal device 100 detects the height of the terminal device 100. In order to detect the height, the AR engine detects a vertical plane. A vertical plane is detected from image information captured by the camera, and the AR engine identifies the position of the mark 201 in the vertical plane based on the mark feature amount acquired from the server 20. The terminal device 100 can calculate how high the position of the mark 201 identified from the image information is from the floor surface FL using the mark height information acquired from the server 20. In addition, since the terminal device 100 is moved by, for example, holding it with the user, the amount of movement of the terminal device 100 is calculated by a means such as a six-axis sensor provided in the terminal device 100, and the current height of the terminal device 100 is detected by adding the amount of movement of the terminal device 100 to the height of the above-mentioned mark 201. In this manner, the initial height of the terminal device 100 can be detected. In addition, the AR engine performs the process of superimposing the navigation AR object at a predetermined height from the floor surface FL in the image, based on the height of the terminal device 100, during the route drawing process.

Figure 7 is an example of a display on the display 106 when the current height and orientation of the terminal device 100 have been identified. A destination box 205 containing the selected destination, the character string "XXX Center 1F", is displayed at the top of the screen (display 106). In addition, a message image 206 indicating the start of AR navigation is displayed in the center of the screen. When the user slowly shakes the terminal device 100, the acceleration sensor 104 detects the vibration of the terminal device 100 and AR navigation begins.

Figure 8 is an example of what is displayed on the display 106 when AR navigation starts. The destination box 205 displays the distance to the AR navigation destination "XXX Center 1st Floor" (217 m) and the estimated time it will take to arrive there on foot (3 minutes). Note that the information displayed in the destination box 205 is the same for Figures 9 and onwards, so further explanation will be omitted.

A navigation AR object 207 is displayed near the center of the screen of the display 106. In this example, the navigation AR object 207 is composed of a wide line (see FIG. 9 onwards) and an arrow. However, the shape of the navigation AR object 207 is not limited to the above-mentioned form. The navigation AR object 207 is displayed superimposed on the image captured by the camera 103.

(Height correction process)
9 shows an example of a display on the display 106 when a user who had been pointing the camera 103 of the terminal device 100 at the digital signage 10 turns the camera 103 around by 180 degrees. The terminal device 100 displays the navigation AR object 207 at a predetermined height h from the floor surface FL displayed on the screen. Note that h=0 may be used.

Unlike the above-mentioned initial height detection of the terminal device 100 using the mark 201, once AR navigation has started, the display height of the navigation AR object 207 cannot be adjusted using the mark 201 as a height reference. This is because the terminal device 100 has moved from its initial position to another position. After AR navigation has started, the display height of the navigation AR object 207 is corrected based on a plane included in the image captured by the camera.

During the execution of AR navigation, the AR engine included in the terminal device 100 detects a plane based on the image captured by the camera 103. This plane may be a horizontal plane, but it does not have to be strictly horizontal and may be a gentle slope, etc. An algorithm for detecting such a plane may use existing technology. In this disclosure, as an example, feature points included in the image captured by the camera 103 are used. A feature point is a part that has distinctive features compared to other parts in the captured image. If this part moves in a plane as the terminal device 100 moves, it can be assumed that it is part of a plane. Intuitive examples of feature points include patterns and stains on walls and floors, but they do not necessarily have to be parts that can be recognized by the naked eye. Procedures for detecting and tracking feature points are known in the field of image recognition, so detailed explanations will be omitted.

Here, the horizontal plane reflected in the image captured by the camera 103 is not limited to only one surface. As shown in FIG. 9, in addition to the floor surface FL, the object OBJ, which may be, for example, a bench, may have a horizontal surface (a seat surface in this example). Therefore, the terminal device 100 performs a floor surface identification process to identify a surface corresponding to the floor surface FL from among the planes detected by the AR engine. That is, the processor 101 determines whether or not the detected plane corresponds to the floor surface. By this determination, the floor surface FL in the image is determined to correspond to the floor surface, and the upper surface of the object OBJ in the image is determined not to correspond to the floor surface. Note that, as a criterion for determining whether or not a certain plane corresponds to a floor surface, for example, a plane having a predetermined area or more may be determined to be a floor surface, or other criteria of existing technology may be used.

When the plane corresponds to the floor surface FL, the terminal device 100 calculates the relative position of the terminal device 100 with respect to the plane based on the captured image. Based on this relative position, the terminal device 100 displays the navigation AR object at a predetermined height h from the plane displayed on the screen (display 106).

Figure 16 is a block diagram showing an example of internal processing of the terminal device 100 when performing the height correction process. Explanation of the same parts as in Figure 15 will be omitted. As shown on the right side of Figure 16, the AR engine detects a plane (e.g., a horizontal plane) based on image information captured by the camera. The terminal device 100 identifies a plane that corresponds to the floor surface from among the detected planes. The terminal device 100 calculates a height correction amount from information about the plane identified as corresponding to the floor surface and information indicating the current route height (display height of the navigation AR object) in the AR engine that draws the route. Then, in the route drawing process, the AR engine corrects the display height of the navigation AR object based on the calculated height correction amount.

(Second aspect of height correction process)
Here, a second aspect of the display height correction of the navigation AR object will be described. In the second aspect, a height threshold is used.

For example, if the terminal device 100 is a smartphone, the user holds the smartphone in their hand and walks towards their destination. In this case, it is highly unlikely that the height of the camera (height of the camera viewpoint) of the terminal device 100 is 10 cm above the floor surface FL. This is because it would be unnatural for the user to walk towards their destination in a position that is as if they are stroking the floor surface FL with the terminal device 100. In other words, assuming that the user moves around while carrying the terminal device 100, the height of the camera 103 as seen from the floor surface FL is considered to be within a predetermined height range. Conversely, the height of the floor surface FL as seen from the height of the camera 103 is also considered to be within a predetermined height range.

Therefore, in a second aspect of the height correction process, the terminal device 100 (processor 101) may change the height of the navigation AR object 207 so that the height of the navigation AR object 207 falls within a certain height range relative to the height of the camera 103.

Figure 17 is a block diagram showing an example of internal processing of the terminal device 100 when performing the second aspect of the height correction process described above. Explanations of parts similar to those in Figures 15 and 16 will be omitted. The difference from the block diagram shown in Figure 16 is that in the height correction amount calculation process, height threshold information (information indicating a certain height range) is obtained from the server 20, and the height correction amount is calculated using this height threshold information as well. Note that the height threshold information may be included in the DL content described above, may be obtained separately by the terminal device 100 from the server 20, or may be stored in advance in the memory 102 of the terminal device 100.

(AR space fixed)
Next, the handling of the feature points extracted from the captured image of the camera 103 will be described. The terminal device 100 may extract feature points from the captured image captured by the camera 103 and perform image recognition. If the extracted feature point information is stored in a storage means, the feature point information can be used as a navigation landmark when performing AR navigation on the same route next time. Therefore, the terminal device 100 (processor 101) extracts feature point information from the captured image by the camera 103, stores it in the memory 102, and outputs the feature point information at a predetermined timing. The output of the feature point information may be outputting a file including the feature point information, and the file may include information indicating the position of the feature point in the AR space (such as a coordinate value in a coordinate system). The output of the feature point information may also be uploading the feature point information as described above to the server 20.

The output feature point information is downloaded (acquired) by a second terminal device (which may be the same terminal device as the terminal device 100 or a different terminal device) when performing AR navigation on the same route next time. The second terminal device adjusts the display position of the navigation AR object 207 on the screen (display 106, etc.) based on the acquired feature point information. That is, the terminal device 100 searches for a feature point corresponding to the acquired feature point information in the image captured by the camera 103 (matching process). When a feature point corresponding to the acquired feature point information is found, the terminal device 100 sets the coordinate value (position) of the found feature point in the AR space to the above-mentioned coordinate value indicated by the output feature point information. That is, the terminal device 100 can identify the position of the found feature point based on the output feature point information, thereby reducing errors during navigation.

Note that the error between the position of the terminal device 100 in the AR space and the position of the terminal device 100 in the real space is small near the navigation start position where AR navigation is started (near the electronic signage 10 in the above example). In other words, it is conceivable that feature point information is gradually saved from areas closer to the navigation start position toward more distant areas. Therefore, the specified timing at which the terminal device 100 (processor 101) outputs the feature point information may be after the terminal device 100 has been moved a specified distance from the navigation start position based on the display of the navigation AR object 207. This specified distance may increase each time AR navigation is performed along the navigation route.

In addition, from the viewpoint of reducing the amount of stored data for feature point information, the stored and output feature point information may be limited to only feature point information in the vicinity of the navigation start position. In this case, the stored and output feature point information can also be used for AR navigation to another destination that starts from the same navigation start position (e.g., the same electronic signage 10).

The saved and output feature point information can also be used in AR navigation that starts from a different navigation start position. Suppose there are two navigation routes using AR navigation, route A and route B. If route A and route B share a common route in part, feature point information about feature points that exist on the common route may be shared between AR navigation using route A and AR navigation using route B.

(First mode of vertical movement warning)
FIG. 10 is a display example by the display 106 in the first mode of vertical movement alert. When navigating a user in a station premises, etc., the user does not necessarily move on a plane, but may use an elevator. For example, the navigation start position is on the first floor of the station, and the destination is on the third floor of the station. The elevator is typically a staircase, an escalator, an elevator, etc. Here, in a mechanism for estimating the current position based on the feature points of an image, it is difficult to accurately track the change in height when the elevator is used. In the case of stairs, the boundary or shadow of the step is mistakenly recognized as a feature point, resulting in an incorrect plane being estimated, and in the case of an elevator or escalator, feature points such as the walls and floors in the elevator room or the steps of the escalator do not appear to move, so it is not possible to recognize that a movement has occurred.

In the present disclosure, when the terminal device 100 detects an elevator (stairs, escalator, elevator, etc.) on the AR navigation route, information is displayed on the screen (display 106) encouraging the use of the elevator. This information display can alert the user of the terminal device 100 to the need to move up and down.

Figure 10 shows an example of a display on the display 106 when the terminal device 100 detects an elevator on the AR navigation route. A descending staircase is displayed on the screen (display 106). At this point, a navigation AR object 207 may be superimposed and displayed along the descending staircase based on the AR navigation data (DATA3) included in the DL content. Information encouraging the use of the elevator may be the navigation AR object 207 displayed along the stairs, escalator, etc., or may be message information such as "Please go down the stairs" or a rectangular box containing the message information.

(Second mode of vertical movement warning)
11 is a display example on the display 106 in the second mode of vertical movement alert. When the terminal device 100 (processor 101) detects that the user of the terminal device 100 is using the lifting means, the terminal device 100 displays information 210 indicating that the user is ascending or descending on the screen (display 106) and hides the navigation AR object 207. Note that the information 210 indicating that the user is ascending or descending and the information encouraging the user to use the lifting means may be the same information.

As described above, the navigation in this disclosure is premised on being performed in places such as train stations where it is difficult or impossible to receive radio waves for a positioning system such as GPS. The navigation in this disclosure identifies the current location of the terminal device 100 based on an image captured by the camera 103. Therefore, when a user carrying the terminal device 100 moves not only forward but also up and down, an error may occur in the display position of the navigation AR object 207 displayed on the screen (display 106). Therefore, when the user of the terminal device 100 uses the lifting means, the navigation AR object 207, which may cause an error, is hidden from the screen (display 106), and instead, information 210 indicating that the device is lifting or descending is displayed.

(Third mode of vertical movement warning)
12 is a display example on the display 106 in the third mode of vertical movement alert. When the terminal device 100 (processor 101) detects that the user of the terminal device 100 has finished using the lifting means, the terminal device 100 hides the information 210 indicating that the lifting means is in progress and redisplays the navigation AR object 207. Then, the AR navigation is continued using the redisplayed navigation AR object 207.

Figure 18 is a block diagram showing an example of internal processing of the terminal device 100 when issuing a vertical movement warning. Explanations of parts similar to those in Figures 15 to 17 will be omitted. In the figure, "air pressure sensor" corresponds to the air pressure sensor 105.

The AR engine detects the amount of movement of the terminal device 100 based on the captured image captured by the camera. This amount of movement may be calculated, for example, based on the movement distance within the image of a feature point extracted from the captured image. The terminal device 100 calculates the current location of the terminal device 100 based on the route information acquired from the server 20 (which may be included in DATA3) and the detected amount of movement of the terminal device 100.

In addition, in the position correction amount calculation process, the terminal device 100 (the processor 101) calculates the position correction amount of the navigation AR object 207 based on various information. The various information may be atmospheric pressure information acquired from an atmospheric pressure sensor, movement amount information of the terminal device 100 acquired from a six-axis sensor, calculated current location information of the terminal device 100, and vertical movement start point information (which may be included in DATA3) and vertical movement height information (which may be included in DATA3) acquired from the server 20, etc.

The AR engine corrects the position of the navigation AR object 207 based on the calculated position correction amount and draws the route (see FIG. 10). That is, the terminal device 100 displays the navigation AR object 207, whose position has been corrected, on the screen (display 106) by superimposing it on the image captured by the camera. Since the position of the navigation AR object 207 has been corrected taking into account the start point (vertical movement start point information) and height (vertical movement height information) of the ascending/descending means such as stairs, the navigation AR object 207 is displayed so as to follow the ascending/descending means such as stairs, as shown in FIG. 10.

Furthermore, when the terminal device 100 detects an elevator on the AR navigation route, when it detects that the user of the terminal device 100 is using the elevator, or when it detects that the user of the terminal device 100 has finished using the elevator, it displays the above-mentioned vertical movement warning (first, second or third mode). In the second mode, the navigation AR object 207 related to the route drawing process is hidden. In the third mode, the information 210 related to the warning drawing process indicating that ascent or descent is in progress is hidden.

(Detection of lifting means, detection of lifting/lowering, detection of lifting/lowering)
The terminal device 100 can detect the upward and downward movement of the terminal device 100 based on the atmospheric pressure information from the atmospheric pressure sensor and the movement amount information of the terminal device 100 from the six-axis sensor. When the terminal device 100 detects the upward and downward movement of the terminal device 100 of a certain distance or more, it may determine that the user of the terminal device 100 is using the elevator means (is ascending or descending). In this case, the terminal device 100 displays information 210 indicating that the terminal device 100 is ascending or descending on the screen (display 106) and does not display the navigation AR object 207.

Conversely, when the terminal device 100 detects that the terminal device 100, which had been moving vertically, has stopped moving vertically, the terminal device 100 may determine that the user of the terminal device 100 has finished using the lifting means (after lifting). In this case, the terminal device 100 hides the information 210 indicating that lifting is in progress from the screen (display 106) and displays the navigation AR object 207 again.

If the current location calculated by the current location calculation process is within a predetermined distance before the location of the elevator included in the route information (which can be identified by the vertical movement start point information), the terminal device 100 may determine that it has detected the elevator on the AR navigation route. In this case, the terminal device 100 displays information on the screen encouraging the use of the elevator.

If the current location calculated by the current location calculation process overlaps with the location of the elevator included in the route information (which can be identified by the vertical movement start point information), the terminal device 100 may determine that the user of the terminal device 100 is using the elevator (is ascending or descending). In this case, the terminal device 100 displays information 210 indicating that the user is ascending or descending, and does not display the navigation AR object 207.

If the current location calculated by the current location calculation process is behind the location of the elevator included in the route information (not shown in FIG. 18, can be identified by the vertical movement end point information that may be included in DATA3 obtained from the server), the terminal device 100 may determine that the user of the terminal device 100 has finished using the elevator (after ascending or descending). In this case, the terminal device 100 hides the information 210 indicating that ascending or descending is in progress from the screen (display 106) and displays the navigation AR object 207 again.

(First aspect of vertical movement correction)
As described above, the navigation in the present disclosure is premised on being performed in a station premises or the like where it is difficult or impossible to receive radio waves for a positioning system such as GPS. In addition, the navigation in the present disclosure identifies the current position of the terminal device 100 based on an image captured by the camera 103. Therefore, as the terminal device 100 moves, an error occurs between the actual position or orientation of the terminal device 100 and the position or orientation in the AR space, and this error may gradually increase.

When a user carrying the terminal device 100 uses stairs, an escalator, or the like, the orientation (travel direction) of the stairs or escalator is predetermined. Information about the orientation of the stairs or escalator may be included in the AR navigation data (DATA3) in the DL content. Thus, when the terminal device 100 (its processor 101) detects that the user of the terminal device 100 is using an elevator, it corrects the orientation of the terminal device 100 in the AR space based on a predetermined travel direction that is assumed when using the elevator.

Figure 13 is a conceptual diagram showing orientation correction in an AR space by the terminal device 100, (a) before correction, and (b) after correction.

The AR navigation data includes data indicating stairs ST. For example, orientation information indicating the orientation of stairs ST may be included in the AR navigation data. Furthermore, up-down movement start point information and up-down movement end point information for stairs ST may be included in the AR navigation data. It is also possible to calculate orientation information for stairs ST from the up-down movement start point information and up-down movement end point information for stairs ST. The orientation of stairs ST corresponds to the expected direction of travel when the user of terminal device 100 uses stairs ST.

The hollow arrows shown in Figures 13(a) and (b) indicate the orientation in the AR space of the terminal device 100 carried by the user walking up the stairs ST. This orientation is the orientation that the terminal recognizes as the correct direction of movement in the AR space (in this example, the orientation of the stairs ST within the AR space). At the time of Figure 13(a), the orientation of the terminal device 100 in the AR space is shifted diagonally relative to the orientation of the stairs ST.

Therefore, while moving up the stairs ST, the terminal device 100 corrects the orientation of the terminal device 100 in the AR space to a predetermined travel direction that is assumed when the user carrying the terminal device 100 uses the stairs ST. At the point in time shown in FIG. 13(b), which shows the state after this correction process, the orientation of the terminal device 100 in the AR space has been appropriately corrected based on the orientation of the stairs ST.

Figure 19 is a block diagram showing an example of internal processing of the terminal device 100 when performing the first aspect of vertical movement correction (direction correction). The same parts as those in Figures 15 to 18 will not be described. In general, when vertical movement occurs, the direction of movement immediately after the vertical movement ends (the direction of movement immediately after the top of the climb, or the direction of movement immediately after the bottom of the climb) is limited. For example, in the case of an elevator, movement can only occur in the direction of movement out of the elevator door. In addition, in the case of stairs and escalators, the direction of movement is limited, although there is a deviation in the width direction when climbing stairs diagonally. Therefore, in this aspect, by utilizing this property, when the end of vertical movement is detected using a six-axis sensor, air pressure sensor, etc., the direction of the terminal device 100 in the AR coordinate space is corrected based on the known direction of the lifting means. As a result, the error between the actual direction of the terminal device 100 and the direction in the AR space is reduced, and AR navigation can be continued appropriately. To detect the end of vertical movement, it is advisable to use a sensor other than an image (such as a six-axis sensor, a barometer, a gyro sensor, or an acceleration sensor). As mentioned above, when detecting movement using an image, the feature points on stairs and the like may be misrecognized, and movement itself may not be detected on elevators and escalators.

(Second aspect of vertical movement correction)
As with the first aspect of vertical movement correction described above, in AR navigation that identifies the current position of the terminal device 100 based on an image captured by the camera 103 rather than a position identification system such as GPS, an error may occur between the actual position of the terminal device 100 and its position in the AR space as the terminal device 100 moves, and this error may gradually increase.

When a user carrying the terminal device 100 uses stairs, an escalator, an elevator, or the like, the position and height of the exit of the stairs, escalator, or elevator are predetermined. The end point (position) information and height information of these ascending and descending means may be included in the AR navigation data (DATA3) in the DL content. Thus, when the terminal device 100 (its processor 101) detects that the user of the terminal device 100 has finished using the ascending and descending means, it corrects the current position of the terminal device 100 in the AR space based on coordinate values indicating a predetermined location. More specifically, the predetermined location may be the end point of the ascending and descending means (the exit of the stairs, escalator, elevator, etc.). The coordinate values indicating the predetermined location may be absolute coordinate values.

Figure 20 is a block diagram showing an example of internal processing of the terminal device 100 when performing the second aspect of vertical movement correction (position correction). The same parts as those in Figures 15 to 19 will not be described. In general, when vertical movement occurs, the position immediately after the vertical movement ends (the place where the up or down movement ends) is limited. For example, in the case of an elevator, both the height and horizontal positions are limited to the position where the elevator door exits. In addition, in the case of stairs or escalators, the height is determined to be the height at which the stairs or the like are climbed, and the horizontal position is also limited, although there is a deviation in the width direction of the stairs or the like. Therefore, in this aspect, by utilizing this property, when the end of the vertical movement is detected using a six-axis sensor, a pressure sensor, or the like, the current position of the terminal device 100 in the AR coordinate space is corrected based on the position of the known lifting means. This reduces the error between the actual position of the terminal device 100 and the position in the AR space, and AR navigation can be continued appropriately. In addition, in cases where a shift in the width direction may occur, such as on stairs, the coordinates at the start of the vertical movement can be used as the horizontal coordinates only, or the coordinates of an arbitrary position within the width of the stairs (for example, near the center of the stairs) can be used as the horizontal coordinates. In addition, a sensor other than an image (such as a six-axis sensor, a barometer, a gyro sensor, or an acceleration sensor) can be used to detect the end of the vertical movement. As described above, this is because detection of movement using images can result in erroneous recognition of feature points on stairs, etc., or the movement itself cannot be detected on elevators and escalators.

Figure 14 shows an example of what is displayed on the display 106 when AR navigation ends. A message 211 is displayed on the screen (display 106) indicating that the user has arrived near the destination.

(Modification)
In the above, an example has been described in which the optical signal 12 (see FIG. 3) transmitted from the electronic signboard 10 is received by the camera 103 and an ID is acquired. However, the terminal device 100 may acquire the ID by other methods. For example, instead of the electronic signboard 10, a poster on which a two-dimensional code is printed may be installed at a predetermined location. Then, the terminal device 100 may read the two-dimensional code with the camera 103 and acquire the ID from the two-dimensional code. In this case, a predetermined mark 11 may be printed on the poster. Even when the optical signal 12 is used, the source of the signal does not need to be in the form of an electronic signboard, and may be, for example, a lighting fixture. Also, a signboard may be covered with a cover or poster on which the mark 11 is printed, using a lighting fixture that emits the optical signal 12 as a backlight. That is, in the present disclosure, the target to which the camera 103 is directed may be any installed object. Also, the position of the terminal device 100 at the start of navigation may be acquired from a lighting fixture, and the orientation of the terminal device 100 at the start of navigation may be acquired from an electronic signboard, and each piece of information may be acquired using a different installed object.

In the above description, the only information transmitted from the electronic signboard 10 is the ID, and other information is obtained from the server 20 using the ID, but information (such as the DL content described above) may be stored in the electronic signboard 10 itself, and all or part of the information may be transmitted to the terminal device 100 via wireless communication, etc.

Although the above description of the present disclosure has been given as a system intended for indoor navigation, it may also be used for outdoor navigation. When applying the present disclosure to a route that travels both outdoors and indoors, the current position may be corrected using a position and orientation calculated using information that can only be obtained outdoors, such as a GPS. It is also possible to apply part or all of the present disclosure not only to AR navigation for pedestrians, but also to other navigation systems, such as a two-dimensional car navigation system.

In addition, in the above, the installation height information acquired as DATA2 may be the exact height of the mark 11, or may be the approximate height of the electronic signboard 10. In the former case, the position of the mark 11 needs to be measured accurately, which makes installation of the electronic signboard 10 cumbersome, but the height can be used as the height of the terminal device 100 as is. In the latter case, the effort required to install the electronic signboard 10 is reduced, but the accuracy of the initial height of the terminal device 100 decreases, but as described above, in this disclosure, the floor surface FL is detected and the display height is adjusted as needed, so there is no significant disruption to the appearance of the navigation.

As described above, when the processor detects that the user of the terminal device is using the lifting means, it displays information on the screen indicating that lifting is in progress and hides the navigation object. This allows the terminal device to continue navigation by means other than the navigation object, which may cause errors, when using the lifting means.

When the processor detects that the user of the terminal device is using the lifting means, it corrects the orientation of the terminal device in space based on a predetermined direction of travel that is expected when the lifting means is used. This makes it possible to correct errors in the orientation of the terminal device by using the lifting means that is in a fixed position.

When the processor detects that the user of the terminal device has finished using the lifting means, it makes the information indicating that the lifting means is in progress invisible and displays the navigation object. This allows navigation to be continued even after the user has dismounted from the lifting means.

When the processor detects that the user of the terminal device has finished using the lifting means, it corrects the current position of the terminal device in space based on coordinate values indicating a predetermined location. This makes it possible to correct errors in the current position of the terminal device by using the lifting means that is in a fixed position.

The processor calculates the relative position of the terminal device, which is facing a navigation start position at a predetermined height from a floor surface, with respect to the floor surface, and displays the navigation object so that it is at the predetermined height from the floor surface displayed on the screen. This makes it possible to superimpose and display the navigation object at the predetermined height based on the navigation start position, which is a fixed position.

The processor calculates the relative position of the terminal device with respect to the floor surface based on start point information acquired by the camera and placed at the navigation start position. The start point information may be any one of the shape, pattern, mark, and identification code of an object. This makes it possible to superimpose a navigation object at a predetermined height based on various start point information that can be acquired from the camera.

When a plane is detected by the camera while a navigation function is being executed, the processor determines whether the plane corresponds to a floor surface, and if the plane corresponds to a floor surface, calculates the relative position of the terminal device with respect to the plane and displays the navigation object at a predetermined height from the plane displayed on the screen. This makes it possible to display the navigation object at a constant height based on a plane such as a horizontal plane included in the image captured by the camera even during navigation.

The processor changes the height of the navigation object so that the height of the navigation object falls within a certain range of height relative to the height of the camera. This makes it possible to prevent the navigation object from being displayed at an unnatural height, assuming that the user is carrying the terminal device.

The processor extracts feature point information from the image captured by the camera, stores the feature point information in the memory, and outputs the feature point information at a predetermined timing. This allows the next and subsequent navigations to be performed with less error by using the extracted feature point information as a landmark.

The predetermined timing may be after the terminal device has been moved a predetermined distance from the navigation start position based on the display of the navigation object. This makes it possible to gradually expand the area in which feature point information can be used farther away from the navigation start position.

The processor outputs the feature point information as a file or uploads it to a server. This allows a terminal device other than the terminal device that performed the initial navigation to use the stored feature point information.

The processor acquires the output feature point information and adjusts the display position of the navigation object on the screen based on the acquired feature point information. This makes it possible to utilize feature point information saved in navigation performed by a different terminal device.

Although various embodiments have been described above with reference to the drawings, it goes without saying that the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, amendments, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure. Furthermore, the components in the various embodiments described above may be combined in any manner as long as they do not deviate from the spirit of the invention.

The present disclosure is useful as a terminal device, a navigation method, and a navigation program that can appropriately navigate a user.

10 Digital signboard 11 Mark 20 Server 100 Terminal device 101 Processor 102 Memory 103 Camera 104 Acceleration sensor 105 Barometric pressure sensor 106 Display 107 Network interface 201 Mark 202 Download progress display bar 203 Rectangular frame 204 Destination list 205 Destination box 206 Message image 207 Navigation AR object BS Base station FL Floor surface ST Stairs

Claims (20)

A terminal device capable of performing navigation,
A processor, a screen, a camera, and a memory;
The processor,
The image captured by the camera and a navigation object are displayed in a superimposed manner on the screen;
when it is detected that the user of the terminal device is using an elevator means on the navigation route, information indicating that the elevator means is being used is displayed on the screen, and the navigation object is hidden.
Terminal device. When the processor detects an elevator means on the navigation route, the processor displays information on the screen encouraging the user to use the elevator means.
The terminal device according to claim 1. when detecting that the user of the terminal device is using the lifting means, the processor corrects the orientation of the terminal device in space based on a predetermined traveling direction assumed when the user uses the lifting means;
The terminal device according to claim 1. when the processor detects that the user of the terminal device has finished using the elevator means, the processor makes the information indicating that the elevator means is in progress invisible and makes the navigation object visible.
The terminal device according to any one of claims 1 to 3. when the processor detects that the user of the terminal device has finished using the lifting means, the processor corrects the current position of the terminal device in space based on coordinate values indicating a predetermined location.
The terminal device according to any one of claims 1 to 4. A terminal device capable of performing navigation,
A processor, a screen, a camera, and a memory;
The processor,
The image captured by the camera and a navigation object are displayed in a superimposed manner on the screen;
calculating a relative position of the terminal device, which is directed toward a navigation start position at a predetermined height from a floor surface, with respect to the floor surface, and displaying the navigation object on the screen so that the navigation object is at the predetermined height from the floor surface;
Terminal device. The processor calculates a relative position of the terminal device with respect to the floor surface based on start point information acquired by the camera and placed at the navigation start position.
The terminal device according to claim 6. The terminal device according to claim 7, wherein the origin information is any one of the shape, pattern, mark, and identification code of an object. A terminal device capable of performing navigation,
A processor, a screen, a camera, and a memory;
The processor,
The image captured by the camera and a navigation object are displayed in a superimposed manner on the screen;
When a plane is detected by the camera during execution of a navigation function, it is determined whether or not the plane corresponds to a floor surface;
if the plane corresponds to a floor surface, calculating a relative position of the terminal device with respect to the plane, and changing a height of the navigation object so that the height falls within a certain height range with respect to a height of the camera with respect to the plane;
Terminal device. A terminal device capable of performing navigation,
A processor, a screen, a camera, and a memory;
The processor,
The image captured by the camera and a navigation object are displayed in a superimposed manner on the screen;
extracting feature point information from the image captured by the camera and storing the information in the memory;
acquiring the feature point information from the memory at a predetermined timing, and adjusting a display position of the navigation object on the screen based on the acquired feature point information.
Terminal device. the predetermined timing is after the terminal device is moved a predetermined distance from a navigation start position based on the display of the navigation object.
The terminal device according to claim 10. The terminal device according to claim 10 or 11, wherein the processor outputs the feature point information as a file or uploads it to a server. A navigation method by a terminal device capable of executing navigation , comprising:
The terminal device includes a processor, a screen, and a camera;
a step of causing the processor to superimpose an image captured by the camera and a navigation object on the screen;
a step of displaying, when the processor detects that the user of the terminal device is using an elevator means on the navigation route, information indicating that the elevator means is being used on the screen and hiding the navigation object;
having
Navigation methods. A navigation method by a terminal device, comprising:
The terminal device includes a processor, a screen, and a camera;
a step of causing the processor to superimpose an image captured by the camera and a navigation object on the screen;
a step of the processor calculating a relative position of the terminal device, which is directed toward a navigation start position at a predetermined height from a floor surface, with respect to the floor surface, and displaying the navigation object so as to be at the predetermined height from the floor surface displayed on the screen;
having
Navigation methods. A navigation method by a terminal device, comprising:
The terminal device includes a processor, a screen, and a camera;
a step of causing the processor to superimpose an image captured by the camera and a navigation object on the screen;
When the processor detects a plane by the camera during execution of a navigation function, the processor determines whether the plane corresponds to a floor surface;
if the plane corresponds to a floor surface, calculating a relative position of the terminal device with respect to the plane, and changing a height of the navigation object so that the height falls within a certain height range with respect to a height of the camera with respect to the plane;
having
Navigation methods. A navigation method by a terminal device, comprising:
The terminal device includes a processor, a screen, a camera , and a memory ;
a step of causing the processor to superimpose an image captured by the camera and a navigation object on the screen;
The processor extracts feature point information from the image captured by the camera and stores the feature point information in the memory;
a step of the processor acquiring the feature point information from the memory at a predetermined timing, and adjusting a display position of the navigation object on the screen based on the acquired feature point information;
having
Navigation methods. 1. A navigation program comprising:
A terminal device that is equipped with a processor, a screen, and a camera and is capable of performing navigation ,
a function of displaying an image captured by the camera and a navigation object superimposed on the screen;
a function of displaying information indicating that the user of the terminal device is using an elevator on the navigation route on the screen and hiding the navigation object when the user of the terminal device detects that the user is using an elevator on the navigation route;
Execute the
Navigation program. 1. A navigation program comprising:
A terminal device having a processor, a screen, and a camera,
a function of displaying an image captured by the camera and a navigation object superimposed on the screen;
a function of calculating a relative position of the terminal device, which is directed toward a navigation start position at a predetermined height from a floor surface, with respect to the floor surface, and displaying the navigation object so that the navigation object is at the predetermined height from the floor surface displayed on the screen;
Execute the
Navigation program. 1. A navigation program comprising:
A terminal device having a processor, a screen, and a camera,
a function of displaying an image captured by the camera and a navigation object superimposed on the screen;
a function of determining whether or not a plane corresponds to a floor surface when a plane is detected by the camera during execution of a navigation function;
a function of calculating a relative position of the terminal device with respect to the plane when the plane corresponds to a floor surface, and changing a height of the navigation object so that the height falls within a certain height range with respect to a height of the camera with respect to the plane;
Execute the
Navigation program. 1. A navigation program comprising:
A terminal device having a processor, a screen, a camera , and a memory ,
a function of displaying an image captured by the camera and a navigation object superimposed on the screen;
A function of extracting feature point information from an image captured by the camera and storing the information in the memory;
a function of the processor acquiring the feature point information from the memory at a predetermined timing, and adjusting a display position of the navigation object on the screen based on the acquired feature point information;
Execute the
Navigation program.

Images