JP6699709B2 - Information processing device and program - Google Patents

Description

  The present invention relates to an information processing device and a program.

  Patent Documents 1 and 2 disclose cameras for capturing a background generated by computer graphics by an operation similar to that of an actual background.

  Patent Document 3 discloses a technique of determining the front-back relationship in the depth direction between a subject and a virtual object appearing in a captured image, and generating an image in which the virtual object is combined with the captured image.

JP 2012-165447 A JP, 2008-288922, A JP2012-58968A

  An object of the present invention is to provide information for guiding a user to an object.

According to a first aspect of the present invention, a target object image associated with a target object and guide information for guiding the target object to a user are displayed on a display unit, and the relationship between the target image and the guide information is displayed. depending on, have a control means for changing the display mode of the guidance information, the guidance information is direction image pointing the direction to the object the position of the user as a reference, mark image as a mark of the object, And at least one of supplementary information for the user to reach the target object, wherein the control unit further provides the supplementary information to the target object while maintaining a predetermined character size. If the supplementary information cannot be displayed without overlapping with the image, the supplementary information is output as voice information, and the supplementary information is displayed without overlapping with the object image while maintaining the predetermined character size. When it is possible, the display mode of the supplementary information is changed so that the supplementary information and the object image do not overlap .

  The invention according to claim 2 is characterized in that the control means changes at least one of a display position, a display color, and a display size of the guidance information as the display mode. It is the information processing device described.

The invention according to claim 3 is characterized in that the control means further causes the display means to display each piece of information included in the supplementary information so that the supplementary information and another image do not overlap each other. The information processing apparatus according to claim 1 or 2 .

According to a fourth aspect of the present invention, the control means causes the display means to display the target object image associated with the target target object , and causes the display means to display the direction image indicating the direction to the target object based on the position of the user. have a said control means is further the case where the direction information can not be displayed without overlapping with the object image, and outputs the direction information as audio information, the object image and superimposing the direction information An information processing device that changes the display mode of the direction information so that the direction information and the object image do not overlap each other when they can be displayed without being displayed .

The invention according to claim 5 is the information processing apparatus according to claim 4 , wherein the control means further causes the display means to display a landmark image serving as a landmark of the object.

The invention according to claim 6 is the information processing apparatus according to claim 5 , wherein the control unit causes the display unit to display the target image by superimposing the mark image on the target image.

In the invention according to claim 7 , when the background image is displayed at the display position of the landmark image, the control means superimposes the landmark image on the background image and causes the display means to display the superimposed image. The information processing apparatus according to claim 6 , characterized in that.

The invention according to claim 8 causes a computer to display an object image associated with a target object and guide information for guiding the object to a user on the display means, and the object image and the guide information. In accordance with the relationship between the guide information and the control means for changing the display mode of the guide information. Which is at least one of supplementary information for the user to reach the target object, and the control means further maintains the predetermined character size while maintaining the supplementary information. When it cannot be displayed without overlapping with the object image, the supplementary information is output as voice information, and the supplementary information is not overlapped with the object image while maintaining the predetermined character size. Is a program for changing the display mode of the supplementary information so that the supplementary information and the object image do not overlap each other.

According to a ninth aspect of the present invention, the computer causes the display unit to display the target object image associated with the target target object, and displays on the display unit a direction image indicating a direction toward the target object based on the position of the user. If the direction information cannot be displayed without overlapping the object image, the direction information is output as audio information, and the direction information is used as the target information. A program for changing the display mode of the direction information so that the direction information and the object image do not overlap each other when the object image can be displayed without overlapping .

According to the invention according to claim 1 9, it is possible to provide information to guide the user to the object.

It is a block diagram which shows the structure of the information processing system which concerns on 1st Embodiment of this invention. It is a block diagram which shows the structure of a terminal device. It is a block diagram which shows the structure of a server. FIG. 3 is a schematic view showing the appearance of the image forming apparatus. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. FIG. 3 is a schematic diagram showing an image forming apparatus and a user. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. FIG. 3 is a diagram showing a positional relationship between an image forming apparatus and a user. It is a figure which shows a screen. It is a figure which shows a screen. FIG. 3 is a diagram showing a positional relationship between an image forming apparatus and a user. It is a figure which shows the positional relationship between a user and an object. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen. It is a figure which shows a screen.

<First Embodiment>
An information processing system according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 shows an example of an information processing system according to the first embodiment.

  The information processing system according to the first embodiment includes, for example, the terminal device 10 and the server 12. In the example shown in FIG. 1, the terminal device 10 and the server 12 have a function of communicating with each other via a communication path N such as a network. The information processing system may include a plurality of terminal devices 10 and a plurality of servers 12.

  The terminal device 10 is a device such as a personal computer (PC), a tablet PC, a smartphone, and a mobile phone, and has a function of transmitting and receiving data to and from other devices. The terminal device 10 may be a wearable terminal (wristwatch type terminal, wristband type terminal, eyeglass type terminal, ring type terminal, contact lens type terminal, implantable terminal, hearable terminal, etc.).

  The server 12 is a device that manages data and user information.

  In the first embodiment, as an example, various types of information are displayed on the terminal device 10 by using AR (Augmented Reality) technology (augmented reality technology). For example, the information associated with the shooting target is displayed on the terminal device 10. AR technology known as AR technology is used. For example, a position AR technique using position information, an image AR technique using image analysis, or the like is used. As the image AR technology, for example, a marker type AR technology that uses a marker such as a two-dimensional barcode, a markerless AR technology that uses an image recognition technology without using a marker, or the like is used.

  Hereinafter, the configuration of the terminal device 10 will be described in detail with reference to FIG.

  The communication unit 14 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 14 may be a communication interface having a wireless communication function or may be a communication interface having a wired communication function. The communication unit 14 supports, for example, one or a plurality of types of communication methods, and may communicate with the communication partner according to a communication method suitable for the communication partner (that is, a communication method supported by the communication partner). .. The communication method is, for example, infrared communication, visible light communication, Wi-Fi (registered trademark) communication, close proximity wireless communication (for example, NFC (Near Field Communication)), or the like. For close proximity wireless communication, Felica (registered trademark), Bluetooth (registered trademark), RFID (Radio Frequency Identifier), or the like is used. Of course, another type of wireless communication may be used as the close proximity wireless communication. The communication unit 14 may switch the communication method and the frequency band according to the communication partner, or may switch the communication method and the frequency band according to the surrounding environment.

  The UI unit 16 is a user interface unit and includes a display unit and an operation unit. The display unit is a display device such as a liquid crystal display. The operation unit is an input device such as a touch panel or a keyboard. Of course, it may be a user interface that also serves as a display unit and an operation unit (including, for example, a touch-type display and a device that electronically displays a keyboard or the like on the display). The UI unit 16 may include a sound collecting unit such as a microphone and a sound generating unit such as a speaker. In this case, the information may be input to the terminal device 10 by voice input, or the information may be output by voice.

  The camera 18 as a photographing unit generates image data (for example, still image data or moving image data) by photographing a photographing target. The image is displayed on the display unit of the UI unit 16, for example. The image displayed on the display unit may be operated by the user. Image data captured by an external camera may be sent to the terminal device 10 via the communication path and the image may be displayed on the display unit of the UI unit 16. Also in this case, the image may be operated by the user.

  The storage unit 20 is a storage device such as a hard disk or a memory (for example, SSD). In the storage unit 20, for example, display information management information, various data, various programs (for example, OS (Operating System), various application programs (application software), etc.), information indicating the address of the terminal device 10 (terminal Address information), information indicating the address of the server 12 (server address information), and the like are stored. The storage unit 20 may store address information associated with other users and information indicating device addresses of other devices (device address information). Of course, they may be stored in separate storage devices, or may be stored in one storage device.

  The display information management information will be described below.

  The display information management information is information for managing the information displayed on the display unit of the UI unit 16 by using the AR technology. The display information management information includes, for example, position AR management information used in the position AR technology and image AR management information used in the image AR technology.

  The position AR management information is, for example, information indicating a correspondence between information indicating a position (position information) and information (display information, content information) displayed on the display unit regarding the position. That is, the display information is associated with the position. One piece of position information is associated with one or more pieces of display information. The position information may be information indicating an absolute position or information indicating a relative position. The position information is, for example, information indicating longitude and latitude. Information indicating the height may be included in the position information.

  The image AR management information is, for example, information indicating a correspondence between information for identifying an object (object identification information) and information (display information) displayed on the display unit regarding the object. is there. That is, the display information is associated with the target object. One or more pieces of display information are associated with one piece of object identification information. The target identification information is, for example, the name or ID of the target, information indicating the type of target, an image associated with the target (target image), information indicating the address of the target (target address information), target Information indicating the shape of an object, and the like. Information indicating the position where the target object is installed (positional information) may be used as the target object identification information. In this case, it can be said that the position AR management information and the image AR management information are the same management information.

  The object is, for example, a device, a building (for example, a store, a building, a station, a hospital, a tower, another facility, etc.), another object (for example, a traffic light, a sign, a signboard, other structures or works, etc.), Roads, animals, etc.

  The device is, for example, an image forming apparatus having an image forming function, a PC, a tablet PC, a smartphone, a mobile phone, a robot (humanoid robot, animal robot other than human, robot other than these), projector, liquid crystal display, and the like. Display device, recording device, reproducing device, imaging device such as camera, refrigerator, rice cooker, microwave oven, coffee maker, vacuum cleaner, washing machine, air conditioner, lighting device, clock, surveillance camera, automobile, motorcycle, aircraft (for example, It is a device such as an unmanned aerial vehicle (so-called drone), a game machine, various sensing devices (for example, temperature sensor, humidity sensor, voltage sensor, current sensor, etc.). The device may be a device that outputs output to the user (for example, an image forming apparatus or a PC), or a device that does not output output to the user (for example, sensing device). The category of the device concept may include all devices. For example, information equipment, video equipment, audio equipment, and other equipment may be included in the category of the equipment according to the first embodiment.

  When the object is a device, the model number of the device and information for managing the device (for example, asset management number) may be used as the object identification information of the device. When the object is a device, the display information may include, as information about the device, information indicating a function of the device, information indicating a state of the device, and the like. When the object is a store, the display information may include, as information about the store, information about products sold at the store, for example.

  The object image may be, for example, an image showing the outside of the object (for example, the housing of the device or the entire building), or the inside of the object (for example, the inside of the object can be seen from the outside by opening the housing of the device). Image showing the inside of a building) or an image showing a state in which an object is covered with an object such as a sheet.

  For example, the terminal device 10 acquires the position information, the display information, and the target object identification information and registers them in the display information management information. These pieces of information may be acquired by using the Internet or the like, or may be input by a user, an administrator, or the like. Further, the terminal device 10 may update the display information management information at an arbitrary timing, regularly, or at a timing designated by the user, the administrator, or the like. Of course, the external device such as the server 12 may generate or update the display information management information without the terminal device 10 itself generating or updating the display information management information. In this case, the display information management information generated or updated by the external device is transmitted from the external device to the terminal device 10 and stored in the storage unit 20.

  The control unit 22 has a function of controlling the operation of each unit of the terminal device 10. The control unit 22 performs, for example, control of communication by the communication unit 14, control of display of information on the display unit of the UI unit 16, writing of data to the storage unit 20, reading of data from the storage unit 20, and the like. .. The control unit 22 also includes an identifying unit 24.

  The identifying unit 24 has a function of identifying the display information displayed on the display unit of the UI unit 16 by referring to the display information management information.

  For example, when the position AR technology is used, the identifying unit 24 uses the GPS (Global Positioning System), wireless communication, various sensors, and the like mounted on the terminal device 10 to identify the terminal device 10 (user). By acquiring position information indicating the current position (for example, latitude, longitude, height, etc.), and further using a geomagnetic sensor (so-called electronic compass), an acceleration sensor, a gyro sensor, etc. mounted on the terminal device 10, The azimuth information indicating the azimuth of the camera 18 (user) of the terminal device 10 is acquired. Next, the identifying unit 24 refers to the management information for the position AR to associate the current position (for example, latitude, longitude, height, etc.) of the terminal device 10 (user) with the display information. Based on (for example, latitude, longitude, height, etc.) and the orientation of the camera 18, the display information displayed on the display unit is specified, and the display position of the display information on the display unit is determined. The identifying unit 24 identifies, for example, display information associated with a position where the distance to the terminal device 10 (user) is equal to or less than a predetermined threshold value.

  When the image AR technique is used, the identifying unit 24 identifies the object by the image AR technique and refers to the image AR management information to identify the display information associated with the object. When the marker type AR technology is used, the identifying unit 24 identifies the target object based on the marker provided on the target object, and uses the identification result to associate the target object with the target object. Specify the displayed information. When the markerless AR technology is used, the identifying unit 24 identifies the target object based on the shape (for example, appearance) of the target object, and uses the identification result to associate the target object with the target object. Specify the displayed information. Information about the identified object may be stored in the storage unit 20.

  Note that the object may be identified without using the AR technique. For example, if the device is connected to the network, the device may be identified based on the IP address, or the device ID may be read to identify the device. Further, when the device has various wireless communication functions such as infrared communication, visible light communication, Wi-Fi, and Bluetooth, even if the device ID is acquired and the device is identified by using the wireless communication function. Good.

  The control unit 22 displays the display information specified by the specifying unit 24 on the display unit of the UI unit 16. At this time, the control unit 22 controls the display form of the plurality of display information so that a part or all of each display information included in the plurality of display information is displayed on the display unit. That is, the control unit 22 controls the display form of the plurality of display information so that each display information is at least partially displayed on the display unit. Further, the control unit 22 may output the display information as audio information. In this case, the display information is output as voice from the speaker.

  Hereinafter, the configuration of the server 12 will be described in detail with reference to FIG. FIG. 3 shows the configuration of the server 12.

  The communication unit 26 is a communication interface and has a function of transmitting data to another device and a function of receiving data from another device. The communication unit 26 may be a communication interface having a wireless communication function or may be a communication interface having a wired communication function.

  The storage unit 28 is a storage device such as a hard disk or a memory (for example, SSD). The storage unit 28 stores, for example, display information management information, various data, various programs, terminal address information of the terminal device 10, server address information of the server 12, and the like. The storage unit 28 may store address information associated with each user, device address information of a device, information about an object identified by the identification process, and the like. Of course, they may be stored in separate storage devices, or may be stored in one storage device.

  The display information management information stored in the storage unit 28 is the same information as the display information management information stored in the storage unit 20 of the terminal device 10.

  The control unit 30 has a function of controlling the operation of each unit of the server 12. The control unit 30 controls communication by the communication unit 26, writes data to the storage unit 28, reads data from the storage unit 28, and the like. The control unit 30 also includes a specifying unit 32. The identifying unit 32 has the same function as the identifying unit 24 of the terminal device 10.

  The display information management information may be stored in the storage unit 20 of the terminal device 10 or may be stored in the storage unit 28 of the server 12. When the display information management information is stored in the storage unit 20 of the terminal device 10, the display information management information does not have to be stored in the storage unit 28 of the server 12. When the display information management information is stored in the storage unit 28 of the server 12, the display information management information does not have to be stored in the storage unit 20 of the terminal device 10.

  Further, when the identifying unit 24 is provided in the terminal device 10, the identifying unit 32 may not be provided in the server 12. Similarly, when the specifying unit 32 is provided in the server 12, the specifying unit 24 may not be provided in the terminal device 10.

  The processing by the identifying unit may be performed by the terminal device 10 or the server 12. For example, when the terminal device 10 is provided with the identifying unit 24, if the display information management information is not stored in the storage unit 20 of the terminal device 10 but is stored in the storage unit 28 of the server 12, The identifying unit 24 of the device 10 accesses the server 12 via the communication path N and refers to the display information management information stored in the server 12. When the display information management information is stored in both the storage unit 20 of the terminal device 10 and the storage unit 28 of the server 12, the identifying unit 24 of the terminal device 10 refers to predetermined display information management information. Of course, when the display information management information is not stored in the storage unit 28 of the server 12 but is stored in the storage unit 20 of the terminal device 10, the specifying unit 24 of the terminal device 10 stores the display information management information in the storage unit 20 of its own device. Refer to the stored display information management information. The same applies when the identifying unit 32 of the server 12 executes the process.

  Hereinafter, the acquisition processing of the object identification information will be described with reference to FIG. FIG. 4 schematically shows the outer appearance of the image forming apparatus 34 as an example of the device. The image forming apparatus 34 is, for example, a device having at least one of a print function, a scan function, a copy function, and a facsimile function. Here, a process for acquiring the device identification information as the object identification information of the image forming apparatus 34 by using the marker type AR technology will be described. A marker 36 such as a two-dimensional barcode is provided on the housing of the image forming apparatus 34. The marker 36 is information in which the device identification information of the image forming apparatus 34 is coded. When the user photographs the marker 36 with the camera 18 provided in the terminal device 10, image data representing the marker 36 is generated. The identifying unit 24 of the terminal device 10 extracts the device identification information by applying the decoding process to the marker image represented by the image data. As a result, the image forming apparatus 34 is identified. Next, the identifying unit 24 identifies the display information associated with the extracted device identification information (object identification information) by referring to the image AR management information. The control unit 22 of the terminal device 10 causes the display unit of the UI unit 16 to display the display information. For example, in the image AR management information, the device identification information of the image forming apparatus 34 corresponds to information indicating the state of the image forming apparatus 34, information indicating the function of the image forming apparatus 34, etc. as an example of display information. The information is attached and the information is displayed on the display unit of the UI unit 16.

  The identifying unit 32 of the server 12 may extract the device identification information by applying a decoding process to the image data representing the marker 36. In this case, the image data is transmitted from the terminal device 10 to the server 12, and the server 12 applies the decoding process to the image data. Further, the specifying unit 32 of the server 12 may specify the display information. In this case, the display information is transmitted from the server 12 to the terminal device 10 and displayed on the display unit of the UI unit 16.

  The marker 36 may include coded information about the image forming apparatus 34 (for example, information indicating a function of the image forming apparatus 34). In this case, by applying the decoding process to the image data representing the marker 36, the device identification information of the image forming apparatus 34 is extracted and the information indicating the function of the image forming apparatus 34 is also extracted.

  When the device identification information is acquired by applying the markerless AR technology, for example, the user uses the camera 18 to photograph the whole or part of the appearance of the target object (for example, the image forming apparatus 34). Of course, it is useful to obtain information for identifying the object such as the name of the image forming apparatus 34 (for example, product name), model number, and asset management number by photographing. Appearance image data representing all or part of the appearance of the image forming apparatus 34 is generated by photographing. The identifying unit 24 of the terminal device 10 identifies the image forming apparatus 34 based on the appearance image data. For example, in the storage unit 20 of the terminal device 10, for each object, the appearance image data showing the whole or a part of the appearance of the object and the object identification information of the object are associated with the appearance image. Correlation information is stored. The specifying unit 24 compares the appearance image data generated by photographing with each appearance image data included in the appearance image association information, and specifies the object identification information of the image forming apparatus 34 based on the comparison result. .. For example, the identifying unit 24 extracts the appearance feature of the image forming apparatus 34 from the appearance image data generated by shooting, and in the appearance image data group included in the appearance image association information, the same or similar to the appearance feature. The appearance image data representing the features of (1) is specified, and the target object identification information associated with the appearance image data is specified. As a result, the image forming apparatus 34 photographed by the camera 18 is identified. When the name (for example, product name) or model number of the image forming apparatus 34 is photographed and appearance image data representing the name or model number is generated, the name or model number represented in the appearance image data, etc. The image forming apparatus 34 may be identified based on the. The identifying unit 24 identifies the display information associated with the object identification information by referring to the image AR management information. When the appearance image data is used as the object identification information, that is, when the appearance image data as the object identification information and the display information are associated with each other in the image AR management information, the specifying unit 24 The display information associated with the appearance image data generated by photographing may be specified by referring to the management information for the image AR. The control unit 22 of the terminal device 10 displays the display information on the display unit of the UI unit 16.

  The identifying unit 32 of the server 12 identifies the target object by comparing the appearance image data generated by photographing with each appearance image data included in the appearance image association information, and further corresponds to the target object. The attached display information may be specified. In this case, the display information is transmitted from the server 12 to the terminal device 10 and displayed on the display unit of the UI unit 16.

  When the position AR technology is used, the identifying unit 24 of the terminal device 10 uses GPS, wireless communication, various sensors, or the like to determine the current position (for example, latitude, longitude, or height) of the terminal device 10 (user). Etc.), and by using a geomagnetic sensor, an acceleration sensor, a gyro sensor, etc., the direction in which the camera 18 (user) is facing is specified. Next, the specifying unit 24 refers to the position AR information management information to determine the current position of the terminal device 10 (user), the position associated with each display information, and the orientation of the camera 18. Based on this, the display information displayed on the display unit is specified, and the display position of the display information is specified. For example, when the distance between the terminal device 10 and the image forming apparatus 34 is equal to or less than the threshold value and the camera 18 is aimed at the image forming apparatus 34, the specifying unit 24 indicates that the image forming apparatus includes the position AR management information. The display information associated with the installation position of 34 is specified. The control unit 22 of the terminal device 10 displays the display information on the display unit of the UI unit 16.

  The identifying unit 32 of the server 12 identifies the display information displayed on the display unit based on the current position of the terminal device 10, the position associated with each display information, and the orientation of the camera 18. You may. In this case, the display information is transmitted from the server 12 to the terminal device 10 and displayed on the display unit of the UI unit 16.

  The target may be identified by using a plurality of identification techniques, or the display information may be specified. For example, the target object may be identified by using a plurality of AR techniques selected from the marker type AR technique, the markerless type AR technique, and the position AR technique, or the display information may be specified. Further, when the target object cannot be identified by a certain AR technology or when the display information cannot be specified, another AR technology may be used to identify the target object or specify the display information. For example, when the target information cannot be identified and the display information cannot be specified by the marker type AR technology or the markerless type AR technology, the display information may be specified by using the position AR technology.

  Hereinafter, the operation of the terminal device 10 will be described in detail with a specific example.

(Specific example 1)
Specific Example 1 will be described with reference to FIG. FIG. 5 shows an example of a screen displayed on the terminal device 10. The control unit 22 of the terminal device 10 causes the display unit of the UI unit 16 to display the screen 38.

  Display information specified by the AR technique is displayed on the screen 38. For example, when the image forming apparatus 34 is photographed by the camera 18, the control unit 22 causes the image 38 associated with the image forming apparatus 34 to be displayed on the screen 38.

  When the position AR technology is used and the camera 18 is aimed at the image forming device 34 and the distance between the terminal device 10 and the image forming device 34 is less than or equal to the threshold value, the specifying unit of the terminal device 10 24 identifies the display information 42, 44 associated with the installation position of the image forming apparatus 34 in the position AR management information.

  When the marker 36 provided in the image forming apparatus 34 is photographed by the camera 18 and the marker type AR technology is applied, the identifying unit 24 uses the device identification information (target) of the image forming apparatus 34 in the image AR management information. The display information 42, 44 associated with the item identification information) is specified. The same applies when a part or all of the image forming apparatus 34 is photographed by the camera 18 and the markerless AR technology is applied.

  The control unit 22 displays the display information 42 and 44 on the screen 38. The display information 42, 44 is, for example, information indicating the state of the image forming apparatus 34. Specifically, the display information 42 is information indicating a state in which the toner loaded in the image forming apparatus 34 is insufficient, and the display information 44 is insufficient in the sheets stored in the image forming apparatus 34. It is information indicating a state of Of course, display information indicating the function of the image forming apparatus 34 may be associated with the installation position of the image forming apparatus 34, and the display information indicating the function may be displayed on the screen 38.

  The image 40 may be a photographed image (for example, one frame in a photograph or a moving image) generated by photographing the image forming apparatus 34 with the camera 18, or an image schematically showing the image forming apparatus 34. (For example, an icon) may be used.

  The control unit 22 controls the display form of each display information so that a part or all of each display information is displayed on the screen 38. In the example illustrated in FIG. 5, the control unit 22 displays the display information 42 such that a part or all of the display information 42 is displayed on the screen 38 and a part or all of the display information 44 is displayed on the screen 38. , 44 are controlled. The control unit 22 may display the display information 42, 44 on the screen 38 so that the display information 42 and the display information 44 do not overlap each other at all, or the display information 42 and the display information 44 partially overlap each other. The display information 42, 44 may be displayed on the screen 38 as described above. That is, the control unit 22 of the terminal device 10 controls the display mode of the plurality of display information so that the certain display information is not completely covered by the other display information and invisible to the user.

  For example, the display information 42 and 44 are each displayed in a predetermined display area. That is, one display area is used for one display information, and each display information is displayed in a separate display area. The display mode of each display area is controlled so that a part or all of each display area is displayed on the screen 38.

  The control unit 22 also displays the display information 42 and 44 on the screen 38 in association with the image 40. For example, the control unit 22 partially displays the display information 42 and 44 on the image 40 so as to be displayed. In the example illustrated in FIG. 5, the display information 42 and 44 are displayed in a display area having a balloon-like shape connected to the image 40, and the display areas are partially overlapped and displayed on the image 40. Has been done. Note that the shape is merely an example, and another shape may be used. Further, only the character string indicated by the display information may be displayed without displaying the boundary itself of the display area.

  The control unit 22 adjusts the display position of each of the display information 42, 44, the shape and size of each display area, the display size of the character string displayed in each display area, etc. A part or all of each of 44 is displayed on the screen 38. For example, the control unit 22 causes the display information to be displayed in a larger display area as the number of characters indicated by the display information (for example, the display information 42) increases. At this time, if all of the other display information (for example, the display information 44) is covered by the larger display area, the control unit 22 of the terminal device 10 causes at least the other display information (display information 44). The display size of the character indicated by the display information 42 is reduced so that the display information 42 is displayed in a smaller display area so as to be displayed without being covered. By doing so, at least a part of each display information is displayed on the screen 38.

  In the example shown in FIG. 5, the display information 44 is partially overlapped and displayed on the display information 42. That is, the display information 44 is displayed on the front side of the display information 42. In this case, when the user operates the UI unit 16 to specify the display information 42 (for example, when the display information 42 is pressed on the screen 38 ), the control unit 22 displays the display information 44 on the front side. The information 42 is displayed. By doing so, the display information specified by the user is displayed without being covered by other display information.

  Further, when the display information is designated by the user, the control unit 22 may display more detailed information regarding the display information on the screen 38. For example, when the display information 44 (information indicating that the paper is out) is designated by the user, the control unit 22 displays, on the screen 38, more detailed information such as the size of the paper to be replenished and the tray. Let For example, when the display information displayed on the foreground is designated by the user, the control unit 22 may display more detailed information regarding the display information on the screen 38. As another example, the control unit 22 determines whether or not the display information is displayed by the user for a predetermined time or more (regardless of whether or not the display information is displayed on the foreground (so-called long press is performed. In this case, more detailed information regarding the display information may be displayed on the screen 38. The control unit 22 acquires detailed information from an object (for example, an image forming apparatus), the server 12, and the like.

  Note that a plurality of pieces of display information may be displayed in one display area (for example, a balloon-shaped area). Also in this case, the size and arrangement of characters in the display area are adjusted so that at least a part of each display information is displayed.

  In the example shown in FIG. 5, the display information about one target object (for example, the image forming apparatus 34) is displayed, but when a plurality of target objects are identified, the display information about each of the plurality of target objects is displayed on the screen. 38 is displayed. The plurality of objects may be objects of the same type or objects of different types. For example, when the marker type AR technology is applied and the image forming apparatus 34 and the PC are identified as objects, the control unit 22 causes the display information (for example, the display information 42, 44) associated with the image forming apparatus 34. ) And display information associated with the PC are displayed on the screen 38. In addition, the marker type AR technology identifies the image forming apparatus 34 and specifies the display information associated with the image forming apparatus 34. Further, the position AR technology identifies display information associated with surrounding buildings. When specified, the control unit 22 causes the screen 38 to display the display information associated with the image forming apparatus 34 and the display information associated with the surrounding buildings and the like. Even in such a case, the control unit 22 controls the display mode of each display information so that at least a part of each display information is displayed on the screen 38.

  As described above, according to the first specific example, each display information is displayed without being completely covered by other display information or images, and thus at least a part of each display information is provided to the user.

  The control unit 22 may display the display information on the display unit of the UI unit 16 and output the display information as audio information. For example, a speaker emits a sound based on the sound information.

  Note that display information may be associated with each part of the object. The correspondence is registered in the display information management information. In this case, the identifying unit 24 identifies the portion of the object by the image AR technique and identifies the display information associated with the portion. The control unit 22 causes the display information to be displayed on the screen 38 in association with the image associated with the portion. For example, the image forming apparatus 34 includes a main body, an automatic document feeder, and a post-processing device, the main body is associated with display information indicating a print function, and the automatic document feeder indicates a scan function. It is assumed that the display information is associated, and the post-processing device is associated with the display information indicating the stapling function. When the main body of the image forming apparatus 34 is identified by the image AR technique, the specifying unit 24 specifies the display information (information indicating the print function) associated with the main body, and the control unit 22 determines the display information. It is displayed on the screen 38. As described above, by associating the display information with each part, it is possible to provide the user with different display information for each identified part even when the same object is photographed.

(Specific example 2)
A second specific example will be described with reference to FIG. A screen is shown in FIG. For example, when the display information 46 is completely covered by the display information 42, 44 and all of the display information 46 is not displayed, the control unit 22 of the terminal device 10 displays each of the display information 42, 44, 46. By shifting the position, at least a part of each of the display information 42, 44, 46 is displayed on the screen 38. The display information 46 is information indicating that the image forming apparatus 34 is receiving a facsimile.

  For example, when the image forming apparatus 34 is not receiving a facsimile, the control unit 22 displays the display information 42 and 44 on the screen 38 as shown in FIG. When the image forming apparatus 34 starts the facsimile reception in this state, the identifying unit 24 of the terminal device 10 acquires the display information 46 indicating the facsimile reception as the display information indicating the state of the image forming apparatus 34, and the control unit 22. Causes the display information 46 to be displayed on the screen 38 in addition to the display information 42 and 44. The display information 46 indicating the state of the image forming apparatus 34 is registered in the display information management information, and the identifying unit 24 acquires the display information 46 from the display information management information. The control unit 22 adjusts the display position of the display information 42, 44, 46 so that at least a part of each of the display information 42, 44, 46 is displayed on the screen 38. Of course, the control unit 22 may adjust the display size of the character indicated by each display information so that at least a part of each of the display information 42, 44, 46 is displayed on the screen 38.

  By shifting the display position of each display information as described above, each display information is displayed without being completely covered by other display information or images.

  Also in the second specific example, when the user operates the UI unit 16 to specify the display information, the control unit 22 displays the display information specified by the user on the front side of other display information. For example, although a part of the display information 44 is covered by the display information 46, when the user specifies the display information 44, the display information 44 is displayed on the front side of other display information. That is, the display information 44 is displayed without being covered by other display information.

(Specific example 3)
Specific Example 3 will be described with reference to FIG. 7. A screen is shown in FIG. In the example illustrated in FIG. 7, the control unit 22 of the terminal device 10 displays all of one display information (for example, the display information 42) on the screen 38, and a part of the other display information (for example, the display information 44, 46). Is displayed on the rear side of the display information 42 so that is displayed on the screen 38. That is, the control unit 22 superimposes the display information 42 on the display information 42 and 44, and shifts the display position of each display information so that a part of each of the display information 44 and 46 is displayed on the screen 38. The display information 42, 44, 46 is displayed on the screen 38. More specifically, the display information 44 is superimposed on the display information 46 so that a part of the display information 46 is displayed, and further, the display information 44 is displayed so that a part of the display information 44 is displayed. Display information 42 is overlaid and displayed. In this way, a plurality of pieces of display information are displayed in an overlapping manner while gradually changing the display position.

  For example, when the user operates the UI unit 16 to specify the display information (for example, the display information 42) displayed on the foreground, the control unit 22 causes the display displayed one level below the display information 42. The information 44 is displayed on the foreground, and the display information 42 is displayed below the display information 46. Similarly, when the user specifies the display information 44 displayed on the foreground, the control unit 22 causes the display information 46 displayed immediately below the display information 44 to be displayed on the foreground. By doing so, each display information is provided to the user, and all the specific display information is provided to the user by a simple operation. When the user specifies the display information displayed on the foreground for a predetermined time or longer, the control unit 22 may display detailed information on the display information on the screen 38.

  In Specific Example 1 to Specific Example 3, the control unit 22 may change the display mode according to the priority of each display information. For example, the control unit 22 causes the display information with higher priority to be displayed on the front side. In the example shown in FIG. 6, it is assumed that the display information 42 has the highest priority (1), the display information 44 has the second priority, and the display information 46 has the third priority. In this case, the control unit 22 displays the display information 42 on the foreground, displays a part of the display information 44 below the display information 42 so as to be covered by the display information 42, and displays a part of the display information 46. The display information 46 is displayed below the display information 44 so as to be covered by the information 42 and 44.

  The priority may be set by the user, may be set based on the display history of each display information, or may be set based on the urgency, for example. For example, higher priority may be given to the display information having a larger number of display times within a predetermined period. Further, the display information that has been designated more frequently by the user within a predetermined period may be given a higher priority. In addition, when a predetermined emergency event occurs (for example, when a device breaks down), the priority of the display information indicating that may be the highest.

  As described above, by displaying the high-priority display information by superimposing it on the low-priority display information, the high-priority display information is displayed more prominently than the low-priority display information. Provided to users.

(Specific Example 4)
Specific Example 4 will be described with reference to FIG. A screen is shown in FIG. In the example illustrated in FIG. 8, the control unit 22 of the terminal device 10 sets the display information (for example, the display information 42, 44, 46) so that the display information (for example, the display information 42, 44, 46) does not overlap each other. It is displayed on the screen 38. The control unit 22 adjusts the display position of each display area, the display size of the character indicated by each display information, and the shape and size of the display area so that the display information does not overlap each other. It is displayed on the screen 38. By doing so, the entire display information is provided to the user.

(Specific Example 5)
Specific Example 5 will be described with reference to FIGS. 9 and 10. FIG. 9 schematically shows the image forming apparatus 34 and the person 48. A screen is shown in FIG. When a new image is additionally displayed on the screen 38 while the display information is being displayed, the control unit 22 of the terminal device 10 displays the display information on the front side of the new image. For example, as shown in FIG. 9, it is assumed that a person 48 (an example of an obstacle) exists between the image forming apparatus 34 and the terminal device 10, and the camera 18 photographs the person 48 together with the image forming apparatus 34. .. In this case, when the display information associated with the person 48 is not registered in the display information management information, the control unit 22 sets the front side of the image 50 associated with the person 48 as illustrated in FIG. 10. , The display information 42, 44, 46 associated with the image forming apparatus 34 is displayed. That is, the control unit 22 superimposes and displays the display information 42, 44, and 46 on the image 50.

  As described above, by displaying the display information in front of other images that are not related to the display information, the display information is blocked by the other images even when the other images are displayed. Not provided but provided to the user.

  When the display information associated with the person 48 is registered in the display information management information, the control unit 22 displays the display information on the screen 38. In this case, the control unit 22 controls the display mode of each display information so that at least a part of each of the display information and the display information 42, 44, 46 is displayed on the screen 38.

(Specific Example 6)
Specific Example 6 will be described with reference to FIGS. 11 and 12. Screens are shown in FIGS. 11 and 12. For example, the image forming apparatus and the PC are photographed by the camera 18, and the display information associated with the image forming apparatus (or the installation position of the image forming apparatus) and the PC (or the image forming apparatus) by the image AR technique or the position AR technique are used. The display information associated with the installation position of the PC) is specified. Further, as shown in FIG. 11, an image 40 associated with the image forming apparatus and an image 52 associated with the PC are displayed on the screen 38. The image 52 may be a photographed image (for example, one frame in a photograph or a moving image) generated by photographing the PC with the camera 18, or an image schematically showing the PC (for example, an icon). May be

  In the specific example 6, the display information regarding the specific object among the plurality of display information is displayed on the display unit of the UI unit 16. The specific object may be, for example, a target object designated by the user, or may be predetermined. Here, it is assumed that the image forming apparatus is designated as a specific object. In this case, as shown in FIG. 12, the control unit 22 does not display the display information associated with the PC on the screen 38, but displays the display information 42, 44, 46 associated with the image forming apparatus. It is displayed on the screen 38. By doing this, the display information related to the objects other than the specific object is not displayed, so that the display information related to the specific object is provided to the user in an easier-to-see state than when the display information related to the objects other than the specific object is also displayed. To be done.

  In Specific Example 6 as well, as in Specific Example 1 to Specific Example 5 above, the control unit 22 of the terminal device 10 displays the display information 42, 44, 46 associated with the image forming apparatus as a specific object. The display information 42, 44, 46 is displayed in such a manner that a part or all of the display information 42, 44, 46 is displayed on the screen 38 so as to be partially overlapped and displayed on the image 40 linked to the image forming apparatus. Control the display mode. In the example shown in FIG. 12, all the pieces of display information are displayed on the screen 38 as in the example shown in FIG. Of course, similar to the example shown in FIGS. 5 to 7, a part of each display information may be displayed on the screen 38 by partially overlapping each display information.

  Note that the specific object may be changed by the user. For example, when the display information 42, 44, 46 related to the image forming apparatus is displayed on the screen 38 and the user operates the UI unit 16 to specify the image 52 to be linked to the PC, the control unit 22 The PC is identified as a specific object, and the display information associated with the PC is displayed on the screen 38 without displaying the display information 42, 44, 46.

  Moreover, a plurality of specific objects may be defined. For example, when the mode for selecting a specific object is executed and the user specifies a plurality of target objects, the plurality of target objects are identified as the specific objects. For example, when the image forming apparatus and the PC are designated by the user, the image forming apparatus and the PC are identified as specific objects, and the display information 42, 44, and 46 associated with the image forming apparatus are associated with the PC. The displayed information is displayed on the screen 38.

(Specific Example 7)
In the specific example 7, the control unit 22 of the terminal device 10 causes the display unit of the UI unit 16 to display the display information associated with the position where the camera 18 is in focus (the position where the focus is in focus). For example, in the example shown in FIG. 11, when the image forming apparatus is focused and the PC is not focused, the control unit 22 displays the display information associated with the PC as shown in FIG. The display information 42, 44, 46 associated with the image forming apparatus is displayed on the screen 38 without being displayed on the screen 38. In this case as well, the control unit 22 causes the display information 42, 44, 46 to be partially overlapped and displayed on the image 40 associated with the image forming apparatus, and each of the display information 42, 44, 46 is partially or entirely displayed. The display mode of the display information 42, 44, 46 is controlled so that it is displayed on the screen 38. The display information associated with the image forming apparatus is specified by the position AR technique or the image AR technique. The location where the focus of the camera 18 is fixed is estimated to be the location where the user is interested. Therefore, the user is interested by displaying the display information associated with the point where the focus is set, without displaying the display information associated with the point where the focus is not set. Display information regarding a portion that is estimated to be provided is provided to the user.

(Specific Example 8)
Specific Example 8 will be described with reference to FIGS. 13 to 16. Screens are shown in FIGS. 13 to 16. For example, the image forming apparatus and the PC are photographed by the camera 18, and the display information associated with the image forming apparatus (or the installation position of the image forming apparatus) and the PC (or the image forming apparatus) by the image AR technique or the position AR technique are used. The display information associated with the installation position of the PC) is specified. Further, as shown in FIG. 13, an image 40 associated with the image forming apparatus and an image 52 associated with the PC are displayed on the screen 38.

  In Specific Example 8, when a plurality of objects are displayed on the screen 38, the control unit 22 of the terminal device 10 displays the display mode of each display information according to the size relation of the images associated with each object. Change. For example, the control unit 22 preferentially displays the display information regarding the object associated with the larger image on the screen 38 over the display information regarding the object associated with the smaller image. Specifically, the control unit 22 displays the display information about the object associated with the larger image in front of the display information about the object associated with the smaller image or displays the display information larger than the display information about the object associated with the smaller image. .. The size of the image is determined, for example, based on the area occupied by the image on the screen 38 and the number of pixels. The image forming apparatus corresponds to an example of the first object, and the PC corresponds to an example of the second object. The image 40 corresponds to an example of the first image, and the image 52 corresponds to an example of the second image.

  In the example shown in FIG. 13, since the image 40 is larger than the image 52, the control unit 22 preferentially displays the display information about the image forming apparatus associated with the image 40 with respect to the display information about the PC associated with the image 52. It is displayed on the screen 38. For example, as shown in FIG. 14, the control unit 22 displays the display information 42, 44, 46 associated with the image forming apparatus on the front side more than the display information 54, 56 associated with the PC. Let In the example illustrated in FIG. 14, the display information 54 associated with the image forming apparatus is partially overlapped and displayed on the display information 54 associated with the PC, and a part of the display information 54 is displayed. Are covered by the display information 44. Further, the display information 42 and 46 associated with the image forming apparatus are partially overlapped and displayed on the display information 56 associated with the PC, and a part of the display information 56 is the display information. It is covered by 42 and 46.

  Further, the control unit 22 may display the character string indicated by the display information 42, 44, 46 with a display size larger than the display size of the character string indicated by the display information 54, 56. , 44, 46 may be larger in size than the display areas of the display information 54, 56.

  Also in the specific example 8, the control unit 22 controls the display form of each display information so that at least a part of each display information is displayed. In the example shown in FIG. 14, all of the display information 42, 44, and 46 associated with the image forming apparatus are displayed on the screen 38, and the display information 54 and 56 associated with the PC are displayed as the display information 42. , 44, 46 are partially covered and displayed. Of course, the display information 42, 44, 46 may be displayed partially overlapping each other.

  Another example is shown in FIG. In the example shown in FIG. 15, since the image 52 is larger than the image 40, the control unit 22 preferentially displays the display information about the PC associated with the image 52, compared to the display information about the image forming apparatus associated with the image 40. It is displayed on the screen 38. For example, as shown in FIG. 16, the control unit 22 displays the display information 54, 56 associated with the PC on the front side more than the display information 42, 44, 46 associated with the image forming apparatus. Let In the example shown in FIG. 16, the display information 54 associated with the PC is partially overlapped and displayed on the display information 44 associated with the image forming apparatus, and a part of the display information 44 is displayed. Are covered by the display information 54. Further, the display information 56 associated with the PC is partially overlapped and displayed on the display information 42, 46 associated with the image forming apparatus, and each of the display information 42, 46 is displayed. The part is covered by the display information 56.

  Further, the control unit 22 may display the character string indicated by the display information 54, 56 by making the display size larger than the display size of the character string indicated by the display information 42, 44, 46. , 56 may be larger than the display areas of the display information 42, 44, 46.

  In the example shown in FIG. 16, all of the display information 54, 56 associated with the PC is displayed on the screen 38, and the display information 42, 44, 46 associated with the image forming apparatus is displayed. The information 54, 56 is partially covered and displayed. Of course, the display information 54 and 56 may be displayed partially overlapping each other.

  The size of the image displayed on the screen 38 may be changed by the user operating the UI unit 16. In this case, the display mode of the display information is controlled according to the change. For example, in the example illustrated in FIG. 15, when the user operates the UI unit 16 to make the image 40 associated with the image forming apparatus larger than the image 52 associated with the PC, the control unit 22 causes the image forming apparatus to operate. The associated display information is displayed on the screen 38 with priority over the display information associated with the PC. By doing so, the display information regarding the object desired by the user is preferentially displayed by a simple operation.

  The specific example 6 and the specific example 8 may be combined. For example, the control unit 22 identifies the target object associated with the largest image as the specific object, and does not display the display information associated with the target object other than the specific object on the screen 38, but the specific object is displayed. The associated display information may be displayed on the screen 38.

(Specific example 9)
In the specific example 9, the control unit 22 of the terminal device 10 changes the display mode of the display information according to the positional relationship between the user (terminal device 10) and each target object. For example, the control unit 22 preferentially displays the display information associated with the object closer to the user on the screen 38 than the display information associated with the object farther from the user.

  For example, when the distance between the user and the image forming apparatus is shorter than the distance between the user and the PC, as shown in FIG. 14, the control unit 22 causes the display information 54 associated with the PC to be displayed. The display information 42, 44, 46 associated with the image forming apparatus is displayed on the front side rather than 56. On the other hand, when the distance between the user and the PC is shorter than the distance between the user and the image forming apparatus, the control unit 22 causes the display information associated with the image forming apparatus as illustrated in FIG. The display information 54, 56 associated with the PC is displayed on the front side more than 42, 44, 46.

  Further, the control unit 22 sets the display size of the character string indicated by the display information associated with the object closer to the user to the display size of the character string indicated by the display information associated with the object further away from the user. The size of the display area of the display information associated with the object closer to the user may be larger than that of the display area of the display information associated with the object further away from the user. May be larger than size.

  According to the specific example 9, the display information associated with the object closer to the user is provided to the user in a more prominent display mode.

(Specific Example 10)
Specific Example 10 will be described with reference to FIG. A screen is shown in FIG. For example, the display information associated with the store (or the position of the store) is specified by the image AR technique or the position AR technique. Further, as shown in FIG. 17, an image 58 associated with the store is displayed on the screen 38.

  The control unit 22 of the terminal device 10 displays the display information 60 and 62 associated with the store on the screen 38. The display information 60 is information indicating a store name, and the display information 62 is information indicating a menu provided at the store. For example, when the user operates the UI unit 16 to specify the display information 60, the control unit 22 causes the screen 38 to display detailed information such as a contact information and a map. When the user operates the UI unit 16 to specify the display information 62, the control unit 22 displays detailed information of the menu on the screen 38. In the example shown in FIG. 17, all of the display information 60 and 62 are all displayed on the screen 38, as in the example shown in FIG.

  In Specific Example 10, the control unit 22 acquires information (related information) about an object (for example, a store) by one or a plurality of applications (software) and displays the related information on the screen 38 as display information. For example, the control unit 22 may acquire the information about the target object by using the Internet, or may acquire the information about the target object by using the SNS (Social Networking Service).

  In the example illustrated in FIG. 17, the control unit 22 acquires relevant information about the store. For example, the control unit 22 searches for a website on the Internet related to the store by using a web browser, and from the website, related information about the store (for example, comments of other users about the store). (Reputation, so-called word of mouth, etc.) may be acquired. In addition, the control unit 22 may use the SNS application to acquire related information (for example, a comment of another user) related to the store in the SNS.

  The control unit 22 acquires related information for each application and displays the related information on the screen 38 as display information. The display information group 64 is a related information group regarding a store acquired from a website. The display information group 66 is a related information group regarding the store acquired from the SNS (referred to as SNS(A)) by using the SNS application A. The display information group 68 is a related information group regarding a store acquired from another SNS (SNS(B)) by using another SNS application B. The display information groups 64, 66, 68 each include one or a plurality of display information. One piece of display information includes one comment, information indicating a reputation, and the like. The control unit 22 may cause the screen 38 to display information indicating the number of pieces of display information included in the display information group.

  The control unit 22 collectively displays related information for each application on the screen 38. For example, the control unit 22 shifts the display position of each display information included in the display information group 64 and causes the display information to be partially overlapped with each other and displayed on the screen 38. When the user operates the UI unit 16 to specify the display information displayed on the foreground, the control unit 22 displays the display information displayed below the display information on the foreground. The display information originally displayed on the foreground is displayed on the back. The same applies to the display information groups 66 and 68.

  Further, the control unit 22 may display only one display information included in the display information group 64 on the screen 38 as the representative display information. When the user operates the UI unit 16 to specify the representative display information, the control unit 22 uses other display information included in the display information group 64 as the representative display information, and replaces the original representative display information with the screen. No. 38 is displayed.

  Note that the control unit 22 may collectively display related information for each website on the screen 38.

  According to Specific Example 10, the related information other than the information associated with the target object is provided to the user.

(Specific Example 11)
Specific Example 11 will be described with reference to FIGS. 18 and 19. Screens are shown in FIGS. 18 and 19. For example, the display information associated with the store (or the position of the store) is specified by the image AR technique or the position AR technique. Further, as shown in FIGS. 18 and 19, an image 58 associated with the store is displayed on the screen 38.

  In the specific example 11, the control unit 22 of the terminal device 10 changes the display information and displays it on the screen 38 in accordance with the positional relationship between the user (terminal device 10) and the target object. For example, when the distance between the user and the target object is less than the threshold, the control unit 22 displays more detailed information as the display information on the screen 38 as compared with the case where the distance between the user and the target object is greater than or equal to the threshold. To display. The threshold value is a predetermined value and may be changed by a user, an administrator, or the like.

  When the distance between the user and the store as the object is equal to or greater than the threshold value, the control unit 22 causes the screen 38 to display the display information 70, 72, and 74 associated with the store, as illustrated in FIG. 18. .. The display information 70, 72, 74 is general information about the store. For example, the display information 70 is information indicating a store name, the display information 72 is information indicating an outline of reservation of the store, and the display information 74 is information indicating an outline of a menu provided in the store. is there. For example, when the user operates the UI unit 16 to specify the display information 70, the control unit 22 displays the contact information of the store, the map, and the like on the screen 38 as detailed information. When the user specifies the display information 72, the control unit 22 causes the screen 38 to display detailed information about the reservation (for example, information indicating the reservation status of each month). When the user specifies the display information 74, the control unit 22 causes the screen 38 to display detailed menu information (for example, a morning menu, a lunch menu, and a dinner menu). When the user is far from the store, the user may check the availability of seats or confirm the reservation status by telephone. Therefore, the above-mentioned outline information is useful for confirmation. Further, the control unit 22 may display the related information 76 regarding the store acquired by using the Internet or SNS on the screen 38.

  When the distance between the user and the store is less than the threshold value, the control unit 22 displays the display information 74a, 74b, 74c associated with the store on the screen 38, as shown in FIG. The display information 74a, 74b, 74c is detailed information about the store, and is, for example, detailed information about the menu. The display information 74a is information indicating a morning menu, the display information 74b is information indicating a lunch menu, and the display information 74c is information indicating a dinner menu. It is estimated that the closer the user is to the store, the higher the possibility that the user will use the store. Therefore, by displaying more detailed information (for example, detailed information of the menu), more significant information is provided to the user as compared with the case of displaying schematic information.

  According to the eleventh example, when the distance between the user and the object is equal to or greater than the threshold value, schematic information is displayed, and when the distance is less than the threshold value, more detailed information is displayed. By doing so, significant information according to the position of the user is provided to the user.

  When the user gives a switching instruction by operating the UI unit 16, the control unit 22 may switch between the display of the summary information and the display of the detailed information according to the instruction. For example, when the distance between the user and the store is less than the threshold, the control unit 22 displays the detailed information on the screen 38, but when the user gives a switching instruction, the control unit 22 displays the summary information on the screen 38. To display. The same applies when the distance between the user and the store is equal to or greater than the threshold.

  Note that although the specific example 11 has been described by taking a store as an example, similar display control is performed even when the object is a device. For example, when the distance between the user and the image forming apparatus is equal to or greater than the threshold value, the control unit 22 causes the screen 38 to display the display information indicating the state of the image forming apparatus (for example, the state of copying, sleeping, etc.). On the other hand, when the distance between the user and the image forming apparatus is less than the threshold value, the control unit 22 causes the screen 38 to display the display information indicating the function of the image forming apparatus (for example, the copy function or the scan function).

  According to the first embodiment described above, even when a plurality of pieces of display information are associated with a particular position or a particular image and the plurality of pieces of display information are simultaneously displayed on the screen 38, each piece of display information is displayed. Since at least a part of the display information is displayed, generation of display information that is not displayed at all (display information that is not displayed at all) is prevented. For example, by utilizing the AR technology, it is possible to prevent the generation of display information that is not displayed at all even if the amount of display information associated with a specific position or a specific image increases. Further, it is not necessary to limit the amount of information displayed at the same time in order to prevent the generation of display information that is not displayed at all.

<Second Embodiment>
The second embodiment will be described below. In the second embodiment, the control unit 22 of the terminal device 10 controls the display of the display information according to the position of the user (terminal device 10). For example, the control unit 22 may change the content of the display information or the display mode of the display information according to the position of the user.

  Specifically, when the display information associated with a specific position (for example, the position of the target object) is specified by using the position AR technology, the control unit 22 controls the content of the display information according to the position of the user. Or change the display format. For example, the control unit 22 changes the content and display form of the display information according to the positional relationship between the specific position and the position of the user. More specifically, the control unit 22 changes the content of the display information and the display form according to the angular relationship between the specific position and the position of the user, the distance relationship, the relationship regarding the direction of the user with respect to the specific position, and the like. . The control unit 22 detects the position of the user and a specific position (for example, the position of the target object) by using GPS or the like, and based on the detection result, the angle between the position of the user and the specific position. Relationships, distance relationships, direction relationships, etc. are detected.

  When the display information associated with a specific image (for example, an image associated with an object) is identified by using the image AR technique, the control unit 22 determines the content of the display information or the content of the display information according to the position of the user. Change the display format. For example, the control unit 22 changes the content of the display information and the display mode according to the positional relationship between the object and the user. More specifically, the control unit 22 changes the content and display form of the display information according to the angular relationship between the object and the user, the distance relationship, the relationship regarding the direction of the user with respect to the object, and the like. The control unit 22 detects the position of the user and the position of the target object by using GPS or the like, and based on the detection result, determines the angular relationship, distance relationship, direction relationship, etc. between the user and the target object. Detect.

  Hereinafter, the second embodiment will be described in detail with a specific example.

(Specific Example 12)
Specific Example 12 will be described with reference to FIGS. 20 to 22. FIG. 20 shows the positional relationship between the image forming apparatus 78 as an object and the user (terminal device 10). Screens are shown in FIGS. 21 and 22. Similar to the first embodiment, the display information associated with the image forming device 78 (or the installation position of the image forming device 78) is specified by the image AR technique or the position AR technique.

  The first position 80 is closer to the image forming apparatus 78 than the second position 82. Display information for each positional relationship between each position and the position of the image forming apparatus 78 is registered in the display information management information. For example, display information for each distance from the image forming apparatus 78 is registered in the display information management information. The control unit 22 changes the content and display form of the display information according to the distance relationship between the user and the image forming apparatus 78. For example, the control unit 22 displays different display information on the screen 38 when the user is at the first position 80 and when the user is at the second position 82.

  FIG. 21 shows display information displayed when the user is at the first position 80 (position closer to the image forming apparatus 78). As shown in FIG. 21, the control unit 22 displays the image 84 associated with the image forming apparatus 78 on the screen 38, and is further associated with the distance between the first position 80 and the position of the image forming apparatus 78. The displayed display information 86, 88 is displayed on the screen 38. Similar to the first embodiment, the control unit 22 may partially display the display information 86 and 88 on the image 84, or a part or all of the display information 86 and 88 may be displayed on the screen 38. The display form of the display information 86 and 88 may be controlled so as to be displayed on the screen. In the example shown in FIG. 21, all of the display information 86 and 88 are displayed on the screen 38. The display information 86, 88 is information indicating the state of the image forming apparatus 78. Specifically, the display information 86 is information indicating that the toner is running out, and the display information 88 is information indicating that the paper is running out.

  FIG. 22 shows the display information displayed when the user is in the second position 82 (position farther from the image forming apparatus 78). As shown in FIG. 22, the control unit 22 displays the image 84 on the screen 38, and further displays the display information 90 associated with the distance between the second position 82 and the position of the image forming apparatus 78 on the screen. No. 38 is displayed. As in the first embodiment, the control unit 22 may partially overlap the display information 90 on the image 84, or when a plurality of display information is displayed, a part or all of each display information is displayed on the screen. The display form of each display information may be controlled so as to be displayed at 38. The display information 90 is information indicating that the image forming apparatus 78 is printing.

  According to the specific example 12, the display information according to the distance between the user and the target object is provided to the user. For example, more detailed display information may be displayed at the first position 80, and more schematic display information may be displayed at the second position 82. When the object is a device, a menu screen or the like for operating the device may be displayed as display information at the first position 80, and display information indicating the state of the device may be displayed at the second position 82. .

  Further, the control unit 22 may change the display mode of the display information between the first position 80 and the second position 82. Specifically, the control unit 22 enlarges the size of the display area of the display information to display the display information on the screen 38 at the first position 80, and displays the size of the display area of the display information at the second position 82. You may make it small and display information on the screen 38. Further, the control unit 22 increases the display size of the character string indicated by the display information at the first position 80 to display the display information on the screen 38, and at the second position 82, the display size of the character string indicated by the display information. May be reduced to display the display information on the screen 38.

  As another example, the control unit 22 causes the screen 38 to display each display information associated with each distance from the image forming apparatus 78 regardless of the position of the user, and the user and the image forming apparatus 78. The size of the display area of each piece of display information may be changed, or the display size of the character string indicated by each piece of display information may be changed, depending on the distance between and. For example, the control unit 22 corresponds to the display information 86 and 88 associated with the distance between the first position 80 and the image forming apparatus 78 and the distance between the second position 82 and the image forming apparatus 78. The attached display information 90 is displayed on the screen 38. When the user is in the first position 80, the control unit 22 causes the display information 86 and 88 to be displayed larger on the screen 38 than the display information 90. On the other hand, when the user is in the second position 82, the control unit 22 causes the display information 90 to be displayed on the screen 38 in a larger size than the display information 86 and 88.

(Specific Example 13)
Specific Example 13 will be described with reference to FIG. FIG. 23 shows the positional relationship between the image forming apparatus 92 as an object and the user (terminal device 10). Similar to the first embodiment, the display information associated with the image forming apparatus 92 (or the installation position of the image forming apparatus 92) is specified by the image AR technique or the position AR technique.

  The image forming apparatus 92 has, for example, a front surface 92a, a back surface 92b, a right side surface 92c, and a left side surface 92d. The virtual line 94 is a line that passes through the right side surface 92c and the left side surface 92d.

  Similar to the specific example 12, the display information for each positional relationship between each position and the position of the image forming apparatus 92 is registered in the display information management information. For example, display information for each distance from the image forming apparatus 92 is registered in the display information management information. Display information for each angle with the image forming apparatus 92 is registered in the display information management information. The angle of a certain position is, for example, the angle formed by the virtual line 94 and the position. Display information for each direction with respect to the image forming apparatus 92 is registered in the display information management information. The direction is, for example, above or below the image forming apparatus 92. The front surface, the back surface, and the side surface of the image forming apparatus 92 may be defined as the direction. Further, display information for each height with respect to the image forming apparatus 92 may be registered in the display information management information. At least two combinations of each distance, each angle, each direction, and each height may be associated with the display information and registered in the display information management information.

  For example, the distance between the first position 96 and the image forming apparatus 92 is the same as the distance between the third position 100 and the image forming apparatus 92, but the distance between the first position 96 and the image forming apparatus 92 is the same. Is different from the angle between the third position 100 and the image forming apparatus 92. In this case, the control unit 22 causes the screen 38 to display different display information depending on whether the user is at the first position 96 or at the third position 100. By doing so, even if the distance to the image forming apparatus 92 is the same, different display information can be provided to the user for each angle.

  The angle between the third position 100 and the image forming apparatus 92 is the same as the angle between the fifth position 104 and the image forming apparatus, but the angle between the third position 100 and the image forming apparatus 92 is the same. The distance is different from the distance between the fifth position 104 and the image forming apparatus 92. In this case, the control unit 22 causes the screen 38 to display different display information depending on whether the user is in the third position 100 or in the fifth position 104. For example, the control unit 22 displays the detailed display information on the screen 38 at the third position 100 closer to the image forming apparatus 92, and displays the rough display information on the screen 38 at the fifth position 104 farther from the image forming apparatus 92. Display it. By doing so, it is possible to provide the user with different display information for each distance even at the same angle with respect to the image forming apparatus 92.

  The second position 98 is a position above the image forming apparatus 92. For example, when the user looks down at the image forming apparatus 92 at the second position 98 and photographs the image forming apparatus 92 with the camera 18, the control unit 22 controls the positional relationship between the second position 98 and the image forming apparatus 92. Display information associated with (distance, height, angle) is displayed on the screen 38. For example, schematic display information of the image forming apparatus 92 is displayed. By doing so, even if the distance to the image forming apparatus 92 is the same, it is possible to provide the user with different display information for each height.

  The fourth position 102 is a position on the back side of the image forming apparatus 92. When the user is photographing the back surface of the image forming apparatus 92 with the camera 18 from the fourth position 102, the control unit 22 controls the user to a position on the front side of the image forming apparatus 92 (for example, the first position, the second position, The display information different from that at the third position and the fifth position) is displayed on the screen 38. For example, display information regarding the structure, parts, functions, etc. on the back side of the image forming apparatus 92 is displayed on the screen 38.

  Further, the control unit 22 may change the display size of the display information while changing the content of the display information for each distance, each angle, each direction, or each height, or without changing the content of the display information. ..

  Hereinafter, an application example of the specific example 13 will be described. Example 13 may be applied to the manufacturing site. In this case, the control unit 22 changes the display information to be displayed on the screen 38 when the device is photographed from the front side of the device and when the device is photographed from the back side of the device. When the device is photographed from the front side, display information about the structure, parts, functions, etc. on the front of the device is displayed, and when the device is photographed from the back side, the rear face of the device is displayed. Display information about the structure, parts, functions, etc. is displayed. By doing so, the person who works on the front side (for example, the operator who assembles the front side parts) is provided with the display information about the front side, and the person who works on the back side (for example the worker who assembles the back side parts) Is provided with display information about the back surface.

  As another example, Example 13 may be applied to an event venue. For example, when a wide range of an event venue exhibited by a company or the like is photographed from above (a bird's eye view of the event venue), the control unit 22 displays simple information such as the name of each exhibiting company on the screen 38. Let By doing so, the user can easily understand what kind of company is exhibiting at the event. When the exhibit booth of a company is photographed up close, the control unit 22 causes the screen 38 to display detailed information about the company (for example, detailed information about the exhibition content). By doing so, detailed information about the company is provided to the user when the user moves to the booth.

  Further, when the wide area of the event venue is photographed from above, the control unit 22 causes the screen 38 to display information on points that can be used at the event venue. Various information may be displayed on the screen 38.

<Other embodiments>
Hereinafter, other embodiments will be described. Here, a case will be described in which the above-described first and second embodiments are applied to route guidance. FIG. 24 shows the relationship between the user and the destination. The destination corresponds to an example of a specific position, and is the position where the object 106 (object α) is installed. It is assumed that the user 108 is photographing the object 106 with the camera 18. In addition, since the user group 110 exists around the object 106, it is assumed that a crowd is generated around the object 106.

  Similar to the first embodiment, the AR technique is used to identify the object 106, and the information associated with the object 106 is displayed on the UI unit 16 of the terminal device 10.

  25 and 26 show a screen 38 of the terminal device 10. FIG. 25 shows the information displayed on the screen 38 when the user 108 is at the position A. FIG. 26 shows the information displayed on the screen 38 when the user is at the position B. The position B is closer to the object 106 than the position A.

  As shown in FIG. 25, an image 112 associated with the object 106 (corresponding to an example of a specific image) and an image 114 associated with the user group 110 are displayed on the screen 38. The image 112 may be an image generated by shooting the object 106, or may be an image (for example, an icon) schematically representing the object 106.

  The control unit 22 of the terminal device 10 uses various sensors to measure the distance between the user 108 and the object 106, the direction of the object 106 based on the position of the user 108, and the user 108 arrives at the object 106. Then, at least one of the predicted times (scheduled arrival times) is calculated. The distance and the direction are calculated by using GPS or the like. The estimated arrival time is calculated based on the distance between the user 108 and the object 106 and the moving speed of the user 108. The moving speed is, for example, an average moving speed of the user 108 in the past, an average moving speed within a predetermined time period (for example, an average moving speed of 10 minutes) retroactively from the present time, a predetermined moving speed (for example, a user The designated moving speed, the default moving speed), the average moving speed of a person having the same attributes (for example, sex, age, height, weight, etc.) as the user 108, and the like.

  The control unit 22 displays each piece of information obtained as described above on the screen 38 as guidance information. The guidance information is information for guiding the user to the object 106 and can be said to be information associated with the object 106. As shown in FIG. 25, the control unit 22 displays the arrow 116, the mark 118, and the supplementary information 120 on the screen 38 as guide information, for example. The control unit 22 may display at least one of the arrow 116, the mark 118, and the supplementary information 120 on the screen 38 as guidance information. The user may specify the destination, and the control unit 22 may create guide information about the destination specified by the user and display the guide information on the screen 38. For example, when the user designates the object 106 as the destination on the screen 38 or designates the object 106 as the destination by voice, the control unit 22 creates guidance information for guiding the user to the object 106. And display it on the screen 38.

  An arrow 116 is an image showing the direction of the object 106 based on the position A of the user 108. The control unit 22 superimposes the arrow 116 on the image 114 as a background image (an image representing a person or an object other than the object 106) and displays the superimposed image on the screen 38. That is, the control unit 22 displays the arrow 116 on the front side of the image 114. By doing so, even when a person or object other than the object 106 is displayed on the screen 38, the direction of the object 106 is shown to the user.

  The mark 118 is an image serving as a mark of the object 106. The control unit 22 superimposes the mark 118 on the image 112 associated with the object 106 and displays it on the screen 38. When the background image is displayed at the display position of the mark 118, the control unit 22 superimposes the mark 118 on the background image and displays it on the screen 38.

  The supplementary information 120 is information including, for example, the name of the object 106 (for example, object α), the distance to the object 106 (for example, 200 m), the estimated arrival time (for example, 2 minutes), and the like. The control unit 22 may display at least one of the name, the distance, and the estimated arrival time on the screen 38 as the supplementary information 120.

  The control unit 22 changes at least one of the arrow 116, the mark 118, and the supplementary information 120 according to the positional relationship between the user 108 and the object 106.

  For example, when the user moves from the position A to the position B, the display content of the screen 38 is changed from the display content shown in FIG. 25 to the display content shown in FIG. 26 with the movement. ..

  In FIG. 26, as in FIG. 25, the images 112 and 114, the arrow 116, the mark 118, and the supplementary information 120 are displayed on the screen 38. Since the user is at the position B closer to the object 106 than the position A, the images 112 and 114 at the position B are displayed larger than the images 112 and 114 at the position A. In this case also, the control unit 22 superimposes the arrow 116 on the image 114 and the mark 118 on the images 112 and 114. Further, since the distance to the object 106 and the estimated arrival time have changed, the control unit 22 updates the supplementary information 120.

  The control unit 22 controls between each image (for example, the images 112 and 114) displayed on the screen 38 and the guide information (for example, the arrow 116, the mark 118, and the supplementary information 120) displayed on the screen 38. The display mode of the guidance information is changed in accordance with the relationship. For example, the control unit 22 changes at least one of the display position, the display color, and the display size of the guide information as the change of the display mode.

  For example, when the display position of the supplementary information 120 overlaps with the display position of another image (for example, the images 112 and 114), or in relation to the other image (for example, the images 112 and 114), the color of the supplementary information 120 is different. When the contrast is weak (for example, when the contrast value is equal to or less than the threshold value), the control unit 22 changes the display color of the character string so that the character string indicated by the supplementary information 120 is conspicuously displayed. For example, when the display position of the supplementary information 120 does not overlap with the display positions of the images 112 and 114 (see FIG. 25), the control unit 22 sets a predetermined color (for example, light blue) as the display color of the character string. To use. When the display position of the supplementary information 120 overlaps the display position of the image 112 (see FIG. 26 ), the control unit 22 causes a color having a higher contrast in relation to the color of the image 112 on which the supplementary information 120 overlaps. (For example, black) is used as the display color of the character string. By doing so, the situation in which the character string is buried in another image and becomes difficult for the user to see is eliminated. In addition, the control unit 22 may change the character thickness, the character size, or the font type.

  FIG. 27 shows another display example when the user 108 is at the position B. In the example illustrated in FIG. 27, the control unit 22 changes the display position of each piece of information included in the supplementary information 120 so that the pieces of information do not overlap other images (for example, the images 112 and 114). For example, the control unit 22 divides the information 122 (character string) indicating the estimated arrival time and the information 124 (character string) indicating the destination and the distance into the information 122, 114 at a position not overlapping the images 112, 114. 124 is displayed. Note that the control unit 22 may change the display position of each information and change the display color and display size (for example, the thickness and size of characters) of each information.

  Further, the control unit 22 may output the supplementary information 120 as voice information according to a user's instruction or automatically. For example, when the supplementary information 120 cannot be displayed without overlapping the other images (for example, the images 112 and 114) while maintaining the predetermined character size, the control unit 22 displays the supplemental information 120. Alternatively, or without displaying the supplementary information 120, the supplementary information 120 is automatically output as voice information. The control unit 22 may output information indicating the direction of the object 106 as audio information.

  FIG. 28 shows another display example when the user 108 is at the position B. In the example shown in FIG. 28, the control unit 22 displays the supplementary information 126 on the screen 38 instead of the supplementary information 120 shown in FIG. The content itself shown by the supplementary information 126 is the same as the content shown by the supplementary information 120. The supplementary information 126 is surrounded by a frame, whereby the supplementary information 126 is highlighted and displayed. The area in which the supplementary information 126 is displayed (area surrounded by a frame) is filled. By doing so, the background portion on which the supplementary information 126 is superimposed is not displayed, and the supplementary information 126 is not buried in the background portion.

  Each of the terminal device 10 and the server 12 described above is realized by cooperation of hardware and software as an example. Specifically, each of the terminal device 10 and the server 12 includes one or more processors such as a CPU (not shown). The functions of the respective units of the terminal device 10 and the server 12 are realized by the one or more processors reading and executing the program stored in a storage device (not shown). The program is stored in the storage device via a recording medium such as a CD or a DVD, or via a communication path such as a network. As another example, the respective units of the terminal device 10 and the server 12 may be realized by hardware resources such as a processor, an electronic circuit, and an ASIC (Application Specific Integrated Circuit). A device such as a memory may be used in the realization thereof. As still another example, each unit of the terminal device 10 and the server 12 may be realized by a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), or the like.

  10 terminal device, 12 server, 20 storage unit, 22 control unit, 24 specifying unit.

Claims (9)

A target object image associated with the target object and guide information for guiding the target object to the user are displayed on the display means, and the guide information of the guide information is displayed according to the relationship between the target image and the guide information. have a control means for changing the display mode,
The guidance information includes a direction image indicating a direction toward the target object based on the position of the user, a mark image serving as a mark of the target object, and supplementary information for the user to reach the target object. At least one,
If the supplementary information cannot be displayed without overlapping the object image while maintaining a predetermined character size, the control means outputs the supplemental information as voice information, and When it is possible to display the supplementary information without overlapping the object image while maintaining the size of the determined character, a display mode of the supplementary information so that the supplementary information and the object image do not overlap. change,
Information processing device. The control means changes at least one of a display position, a display color, and a display size of the guidance information as the display mode.
The information processing apparatus according to claim 1, wherein: The control means further causes the display means to display each piece of information included in the supplementary information so that the supplementary information and another image do not overlap each other.
The information processing apparatus according to claim 1 or 2 , characterized in that. To display the object image tied to the object of interest on the display means, a direction image pointing directions to the object the position of the user relative to have a control means for displaying on said display means,
When the direction information cannot be displayed without overlapping the object image, the control unit outputs the direction information as voice information and displays the direction information without overlapping the object image. If it is possible, change the display mode of the direction information so that the direction information and the object image do not overlap,
Information processing device. The control means further causes the display means to display a landmark image serving as a landmark of the object,
The information processing apparatus according to claim 4 , wherein: The control means superimposes the landmark image on the object image and displays the superimposed image on the display means.
The information processing apparatus according to claim 5 , wherein: When the background image is displayed at the display position of the mark image, the control unit causes the display unit to display the mark image by superimposing the mark image on the background image,
The information processing apparatus according to claim 6 , characterized in that. Computer,
A target object image associated with the target object and guide information for guiding the target object to the user are displayed on the display means, and the guide information of the guide information is displayed according to the relationship between the target image and the guide information. Control means for changing the display mode,
To function as,
The guidance information includes a direction image indicating a direction toward the target object based on the position of the user, a mark image serving as a mark of the target object, and supplementary information for the user to reach the target object. At least one,
If the supplementary information cannot be displayed without overlapping the object image while maintaining a predetermined character size, the control means outputs the supplemental information as voice information, and When it is possible to display the supplementary information without overlapping the object image while maintaining the size of a predetermined character, a display mode of the supplementary information so that the supplementary information and the object image do not overlap. change,
program. Computer,
Control means for displaying a target image associated with the target object on the display means, and for displaying on the display means a direction image indicating a direction to the target object based on the position of the user;
To function as,
When the direction information cannot be displayed without overlapping the object image, the control unit outputs the direction information as voice information and displays the direction information without overlapping the object image. When it is possible to change the display mode of the direction information so that the direction information and the object image do not overlap,
program.

Images