JP6932907B2 - Information processing equipment, information processing system, information processing method and program - Google Patents

Description

The present invention relates to an information processing apparatus, information processing system, an information processing method, and a program.

In recent years, a technique related to augmented reality (AR) has attracted attention. Some AR applications recognize an object appearing in an image taken by a user with a so-called smartphone or the like, and display information about the recognized object superimposed on the image. In such a technique related to an AR application, there is an application (marker type AR application) that recognizes a two-dimensional marker appearing in an image as an object and displays the corresponding information. For example, Patent Document 1 describes a technique for performing processing so as to continue displaying an image corresponding to the two-dimensional marker even when the two-dimensional marker cannot be detected.

Japanese Unexamined Patent Publication No. 2013-134538

However, with the technique described in Patent Document 1 described above, it is not possible to determine whether the two-dimensional marker cannot be detected by chance or is changed so that the user does not intentionally detect it, and the user does not necessarily have to. It was not very convenient for me.

The present invention has been made in view of such a situation, and an object of the present invention is to perform image display processing that enhances user convenience.

In order to achieve the above object, the information processing device of one aspect of the present invention is
An identification information acquisition means for acquiring identification information from an electronic device provided with an identification information display unit on which an identification information image is displayed, and an identification information acquisition means.
A display control means for controlling the display unit to perform virtual display according to the identification information, and
Wherein when the identification information by the identification information acquiring means is no longer acquired, and a determination unit configured to determine whether to continue the virtual display before Symbol display unit,
The determination means determines that the virtual display is continuously performed when the tilt generation notification indicating that the identification information display unit included in the electronic device is tilted is acquired from the electronic device.
When the determination means determines that the virtual display is continuously performed, the display control means controls the display unit so as to continuously perform the virtual display based on the identification information.
It is characterized by that.

According to the present invention, it is possible to perform an image display process that enhances the convenience of the user.

It is a system configuration diagram which shows the system configuration of the AR display system S which concerns on one Embodiment of the information processing system of this invention. It is a schematic diagram for demonstrating the AR display in this embodiment. It is a block diagram which shows the hardware structure of the mobile terminal 1 and the list terminal 2 which concerns on one Embodiment of this invention. It is a functional block diagram which shows the functional configuration for executing AR display processing and notification processing among the functional configurations of the mobile terminal 1 and the list terminal 2 of FIG. It is a flowchart explaining the flow of AR display processing executed by the mobile terminal 1 of FIG. 3 having the functional configuration of FIG. FIG. 5 is a flowchart illustrating a flow of notification processing executed by the list terminal 2 of FIG. 3 having the functional configuration of FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a system configuration diagram showing a system configuration of an AR display system S according to an embodiment of an information processing system of the present invention.
As shown in FIG. 1, the AR display system S includes a mobile terminal 1 and a list terminal 2.

The mobile terminal 1 has a photographing function for photographing an area including a marker serving as identification information, a communication function for communicating with the list terminal 2, and a display function for displaying the photographed image and the result corresponding to the photographed marker. Have.

The list terminal 2 has a display function for displaying a marker (two-dimensional code in the present embodiment), a communication function for communicating with the mobile terminal 1, and a sensing function for detecting the posture of the terminal.

In the AR display system S configured in this way, the marker displayed on the list terminal 2 is photographed, and the display corresponding to the marker is performed on the mobile terminal 1. In the present embodiment, the image corresponding to the marker is superimposed on the background at the corresponding portion on the live view display screen. On the screen of the mobile terminal 1, the scenery taken by the mobile terminal 1 and the image corresponding to the marker are virtually displayed at the positions corresponding to the markers to be photographed. Hereinafter, such a display is referred to as an "AR display".

The AR display system S of the present embodiment corresponds to the marker when the marker is no longer photographed from the state where the marker was photographed, for example, when the posture of the list terminal 2 is changed as intended by the user. Continue to display the image to be displayed.

In this way, regardless of whether or not the marker is photographed, by displaying the image corresponding to the marker according to the change in the posture of the list terminal 2, for example, it is desired to photograph the marker and maintain the AR display. Contrary to the user's intention, even if the hand on which the wrist terminal 2 is attached temporarily moves or the shooting angle changes temporarily and the marker is temporarily stopped shooting. Since the display corresponding to the marker can be performed, stable AR display can be performed even when the marker cannot be read due to a factor not intended by the user.

FIG. 2 is a schematic diagram for explaining the AR display in the present embodiment.
In the mobile terminal 1, as shown in FIG. 2, when the area including the marker M is photographed by [normal display] in which the marker is displayed as it is without performing the AR display, the display screen of the marker M is displayed.

On the other hand, when the area including the marker M is photographed by [AR display], an image corresponding to the marker M (hereinafter, referred to as “AR image”) is displayed. In the present embodiment, as the AR image, for example, a cylindrical three-dimensional image is displayed.

If, for example, the angle of the list terminal 2 changes from the state in which the marker M is photographed readable, and the display surface displaying the marker M is no longer photographed readable, it is usually the case. The image AR corresponding to the marker is not displayed in response to the fact that the marker M is no longer photographed readable.
However, in the present embodiment, due to the change in the angle of the list terminal 2, for example, the display screen of the list terminal 2 faces in a direction different from the direction in which the mobile terminal 1 can read the marker M, and the marker M When is no longer photographed, it is determined that the marker has disappeared temporarily unintentionally by the user, and the AR image is continuously displayed.

In the present embodiment, the AR image to be continuously displayed is, for example, continuously displayed at the position corresponding to the position where the marker M was last photographed, that is, the position where the AR image was last displayed.

FIG. 3 is a block diagram showing a hardware configuration of a mobile terminal 1 and a list terminal 2 according to an embodiment of the present invention.
The mobile terminal 1 is configured as, for example, a smartphone.

As shown in FIG. 3, the mobile terminal 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, and an image pickup. A unit 16, a sensor unit 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22 are provided.

The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 20 into the RAM 13.

Data and the like necessary for the CPU 11 to execute various processes are also appropriately stored in the RAM 13.

The CPU 11, ROM 12 and RAM 13 are connected to each other via the bus 14. An input / output interface 15 is also connected to the bus 14. An image pickup unit 16, a sensor unit 17, an input unit 18, an output unit 19, a storage unit 20, a communication unit 21, and a drive 22 are connected to the input / output interface 15.

Although not shown, the image pickup unit 16 includes an optical lens unit and an image sensor.

The optical lens unit is composed of a lens that collects light, such as a focus lens or a zoom lens, in order to photograph a subject.
The focus lens is a lens that forms a subject image on the light receiving surface of the image sensor. A zoom lens is a lens that freely changes the focal length within a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, and white balance, if necessary.

The image sensor is composed of a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a CMOS (Complementary Metal Oxide Semiconductor) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element performs photoelectric conversion (imaging) of the subject image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to AFE.
AFE executes various signal processing such as A / D (Analog / Digital) conversion processing on the analog image signal. A digital signal is generated by various signal processing and is output as an output signal of the imaging unit 16.
Such an output signal of the imaging unit 16 is hereinafter referred to as “captured image data”. The captured image data is appropriately supplied to the CPU 11 or an image processing unit (not shown).

The sensor unit 17 is composed of various sensors such as an acceleration sensor and a gyro sensor, and can acquire posture information of the device from the output sensor information.

The input unit 18 is composed of various buttons and the like, and inputs various information according to a user's instruction operation.
The output unit 19 is composed of a display, a speaker, or the like, and outputs an image or sound.
The storage unit 20 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 21 controls communication with other devices (not shown) via a network including the Internet.

A removable media 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 22. The program read from the removable media 31 by the drive 22 is installed in the storage unit 20 as needed. Further, the removable media 31 can also store various data such as image data stored in the storage unit 20 in the same manner as the storage unit 20.

Further, the wrist terminal 2 is configured as, for example, a smart watch.

The list terminal 2 includes a CPU 11, a ROM 12, a RAM 13, a bus 14, an input / output interface 15, a sensor unit 17, an input unit 18, an output unit 19, a storage unit 20, and a communication unit 21. I have.
The list terminal 2 includes a CPU 11, a ROM 12, a RAM 13, a bus 14, an input / output interface 15, a sensor unit 17, an input unit 18, an output unit 19, a storage unit 20, and the mobile terminal 1. The configuration of the communication unit 21 is the same. In the list terminal 2, the description of the same configuration as that of the mobile terminal 1 will be omitted.

In the following, when the description of the hardware configuration of the mobile terminal 1 and the list terminal 2 is separated, different reference numerals are given, and in the mobile terminal 1, "-1" is added to the end of the configuration to make the list terminal. 2 adds "-2" to the end of the configuration. That is, in the mobile terminal 1, it becomes the CPU 11-1 to the drive 22-1, and in the list terminal 2, it becomes the CPU 11-2 to the communication unit 21-2.

FIG. 4 is a functional block diagram showing a functional configuration for executing AR display processing and notification processing among the functional configurations of the mobile terminal 1 and the list terminal 2 of FIG.

The AR display process refers to a series of processes for recognizing a marker displayed on the list terminal 2 and performing AR display. In the AR display process, even if the marker cannot be recognized, the AR display is performed so as to continue the display of the AR image according to the state (posture) of the list terminal 2.

Further, in the notification process, when there is an allowable or greater posture change (in the present embodiment, a predetermined or greater inclination occurs) in the state where the marker is displayed, the posture change is performed with respect to the mobile terminal 1. It refers to a series of processes for notifying the fact that there was a problem (hereinafter, referred to as "inclination occurrence notification" in the present embodiment).

When the AR display process is executed, as shown in FIG. 4, the imaging control unit 51, the image analysis unit 52, the display control unit 53, and the communication control unit 54 function in the CPU 11-1.

Further, an AR image storage unit 71 is set in one area of the storage unit 20.
The AR image storage unit 71 stores AR image data associated with the shape of the marker. The AR image storage unit 71 stores, for example, three-dimensional image data as shown in FIG.

The shooting control unit 51 controls the imaging unit 16-1 so as to execute the shooting process. In the present embodiment, live view shooting is performed in which live view images are sequentially output.
Specifically, the imaging control unit 51 can control the imaging unit 16-1 so as to perform imaging, for example.

The image analysis unit 52 analyzes the captured image, for example, and detects the presence or absence of a marker.
Specifically, the image analysis unit 52 can analyze the captured image, for example, and determine whether or not the marker has been recognized. In addition, the image analysis unit 52 can analyze the image and determine whether or not the marker has disappeared. In addition, the image analysis unit 52 can analyze the image and determine whether or not the marker has returned. In addition, the image analysis unit 52 can determine whether or not a certain period of time has elapsed.

The display control unit 53 controls the output unit 19-1 so as to display an image.
Specifically, the display control unit 53 can control the output unit 19-1 so as to display the AR image corresponding to the marker on the live view image, for example. Further, the display control unit 53 can control the output unit 19-1 so as to erase the AR image from the displayed image. Further, the display control unit 53 can control the output unit 19-1 so as to continue displaying the AR image at the same position.

The communication control unit 54 controls the communication unit 21 so as to communicate with the list terminal 2.
Specifically, the communication control unit 54 can determine whether or not the tilt occurrence notification has been received, for example, via the communication unit 21-1.

When the notification process is executed, as shown in FIG. 4, the display control unit 91, the sensor information analysis unit 92, and the communication control unit 93 function in the CPU 11-2.

The display control unit 91 controls the output unit 19-1 so as to display an image.
Specifically, the display control unit 91 can control the output unit 19-1 so as to display an image of the marker, for example.

The sensor information analysis unit 92 analyzes the sensor information acquired from the sensor unit 17-2 and detects the posture of the device.
Specifically, the sensor information analysis unit 92 can analyze the sensor information acquired from the sensor unit 17-2, for example, and acquire the screen direction. In addition, the sensor information analysis unit 92 can analyze the sensor information and determine whether or not an inclination has occurred.

The communication control unit 93 controls the communication unit 21 so as to communicate with the mobile terminal 1.
Specifically, the communication control unit 93 can control the communication unit 21-2 so as to transmit, for example, a tilt occurrence notification.

FIG. 5 is a flowchart illustrating a flow of AR display processing executed by the mobile terminal 1 of FIG. 3 having the functional configuration of FIG.
The AR display process is started by the user's operation of starting the AR display process on the input unit 18-1.

In step S11, the imaging control unit 51 controls the imaging unit 16-1 so as to perform imaging.

In step S12, the image analysis unit 52 analyzes the captured image and determines whether or not the marker has been recognized.
When the marker is recognized, YES is determined in step S12, and the process proceeds to step S13.
On the other hand, when the marker is not recognized, NO is determined in step S12, and the process returns to step S11.

In step S13, the display control unit 53 controls the output unit 19-1 so as to display the AR image corresponding to the marker on the live view image.

In step S14, the image analysis unit 52 analyzes the image and determines whether or not the marker has disappeared.
If the marker disappears, YES is determined in step S14, and the process proceeds to step S15.
On the other hand, if the marker has not disappeared, NO is determined in step S14, and the process returns to step S13.

In step S15, the communication control unit 54 determines whether or not the inclination generation notification has been received via the communication unit 21-1.
When the tilt occurrence notification is received, it is determined as YES in step S15, and the process proceeds to step S17.
On the other hand, when the inclination generation notification is not received, it is determined as NO in step S15, and the process proceeds to step S16.

In step S16, the display control unit 53 controls the output unit 19-1 so as to erase the AR image from the displayed image. After that, the process returns to step S13.

In step S17, the display control unit 53 controls the output unit 19-1 so as to continue displaying the AR image at the same position.

In step S18, the image analysis unit 52 analyzes the image and determines whether or not the marker has returned.
When the marker returns, it is determined as YES in step S18, and the process proceeds to step S20.
On the other hand, if the marker has not returned, NO is determined in step S18, and the process proceeds to step S19.

In step S19, the image analysis unit 52 determines whether or not a certain period of time has elapsed.
When a certain period of time has elapsed, YES is determined in step S19, and the process returns to step S16.
On the other hand, if a certain time has not elapsed, NO is determined in step S19, and the process returns to step S17.

In step S20, the display control unit 53 controls the output unit 19 so as to erase the AR image from the displayed image. After that, the process returns to step S13.

FIG. 6 is a flowchart illustrating a flow of notification processing executed by the list terminal 2 of FIG. 3 having the functional configuration of FIG.
The notification process is started by the operation of the user to start the notification process to the input unit 18-2.

In step S41, the display control unit 91 controls the output unit 19-2 so as to display the marker.

In step S42, the sensor information analysis unit 92 analyzes the sensor information acquired from the sensor unit 17-2 and acquires the screen direction.

In step S43, the sensor information analysis unit 92 analyzes the sensor information and determines whether or not an inclination has occurred.
When the inclination occurs, it is determined as YES in step S43, and the process proceeds to step S44.
On the other hand, if the inclination does not occur, NO is determined in step S43, and the process returns to step S42.

In step S44, the communication control unit 93 controls the communication unit 21-2 so as to transmit the tilt occurrence notification. After that, the process returns to step S41.

In the conventional AR image, a marker installed mainly on a flat surface is photographed by the mobile terminal 1, and an image associated with the marker is superimposed and displayed on the image. However, when the marker is displayed on the screen of the list terminal 2 and photographed instead of being installed on a flat surface, the marker may not be recognized on the mobile terminal 1 side if the attached arm moves and the screen tilts. There is sex.

A marker is displayed on the screen of the list terminal 2, this is photographed by the mobile terminal 1 and the AR image associated with the marker is superimposed and displayed, and when the list terminal 2 detects a large change in the screen angle, it is carried. The terminal 1 continues to display the AR image at the same place even if the marker cannot be recognized, and ends the display of the AR image when a certain period of time elapses.

As a result, in the AR display system S, even if the list terminal 2 which is difficult to be stationary is used as the display of the AR marker, the AR image can be displayed without interruption regardless of the change in the angle.

Therefore, even if the marker disappears, the mobile terminal 1 continues to display the AR image corresponding to the marker as long as it is due to the inclination of the list terminal 2 that displays the marker. Therefore, even if the marker disappears in an unintended state such as when the list terminal 2 is unexpectedly tilted, the AR image can be continuously displayed. Therefore, the user can stably confirm the AR image even if the marker disappears in an unintended state.

The mobile terminal 1 configured as described above includes an output unit 19-1, an image analysis unit 52, and a display control unit 53.
The image analysis unit 52 acquires identification information from the identification information image.
The display control unit 53 controls the output unit 19-1 to perform virtual display according to the identification information.
When the identification information is no longer acquired, the image analysis unit 52 determines whether or not to continuously perform the virtual display in the output unit 19-1 based on the state of the identification information image.
The display control unit 53 controls the output unit 19-1 so that the virtual display based on the identification information is continuously performed when the image analysis unit 52 determines that the virtual display is continuously performed.
As a result, in the AR display system S, in order to determine whether or not the virtual display is continuously performed based on the state of the output unit 19-1, an image display process that enhances the convenience of the user is performed. Can be done.

The image analysis unit 52 acquires the identification information from the list terminal 2 including the output unit 19-2 on which the identification information image is displayed.
As a result, in the AR display system S, virtual display can be performed, for example, in cooperation with the list terminal 2 worn by the user.

The image analysis unit 52 determines whether or not to continuously perform the virtual display based on the posture change of the output unit 19-2.
As a result, in the AR display system S, the continuation can be easily determined in order to determine the continuation of the display based on the posture change of the list terminal 2 (output unit 19-2).

The attitude change of the output unit 19-2 is detected based on the image taken by the output unit 19-2 or the attitude information of the output unit 19-2.
As a result, in the AR display system S, the posture change of the output unit 19-2 can be easily detected.

The wrist terminal 2 further includes a sensor unit 17-2 and a communication unit 20-2.
The sensor unit 17-2 acquires sensor information, which is the first information related to the attitude change of the output unit 19-2.
The communication unit 20-2 transmits the sensor information, which is the first information acquired by the sensor unit 17-2, to the mobile terminal 1.
The mobile terminal 1 further includes a communication unit 20-1 that receives information transmitted by the communication unit 20-2.
The image analysis unit 52 determines whether or not to continuously perform the virtual display based on the sensor information which is the first information received by the communication unit 20-1.
As a result, in the AR display system S, the posture of the list terminal 2 (output unit 19-2) is based on the sensor information from the sensor unit 17-2 by using the communication function between the mobile terminal 1 and the list terminal 2. Changes can be detected.

The identification information image is displayed on the screen of the output unit 19-2 or on the surface shape.
As a result, in the AR display system S, virtual display can be performed corresponding to the list terminal 2, and it is possible to determine whether to continue the virtual display according to the state of the list terminal 2.

The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the range in which the object of the present invention can be achieved are included in the present invention.

In the above-described embodiment, the list terminal 2 is configured to analyze the sensor information, but the sensor information may be transmitted to the mobile terminal 1 and the sensor information may be analyzed on the mobile terminal 1 side. Further, the sensor information acquired by the list terminal 2 may be analyzed by an external device, and only the analysis result may be acquired by the mobile terminal 1.

Further, in the above-described embodiment, it is determined to continue displaying the AR image according to the posture change on the list terminal 2 side, but even when the marker cannot be captured due to the posture change on the mobile terminal 1 side, the AR It may be configured to continue displaying the image. For example, in the mobile terminal 1 for photographing the marker, when the user changes the angle of the mobile terminal 1 and the marker cannot be captured temporarily, the marker is not captured in a state unintended by the user. Therefore, it may be configured to determine whether to continue displaying the AR image in consideration of the case where the marker can be captured after a lapse of a predetermined time, how the marker is no longer captured, and the like.

Further, in the above-described embodiment, the display of the AR image is continued based on the posture change, but the display of the AR image is continued based on a predetermined operation including the continuous posture change. It may be configured as follows. The predetermined motion includes, for example, a motion of simply bending the arm back, in addition to the motion of achieving a predetermined purpose.

Further, in the above-described embodiment, the AR image may be configured to be displayed in a different orientation according to the orientation of the display surface of the marker. Further, even after the marker cannot be captured, the position and orientation of the AR image may be changed and displayed in accordance with the change in the position of the list terminal 1 and the orientation of the display surface of the marker. That is, when the marker cannot be read, the position and orientation of the virtual display may be changed according to the acceleration information (movement left and right, change in orientation, etc.) of the list.

Further, in the above-described embodiment, the AR image is configured to be continuously displayed at the position of the AR image displayed at the timing when the marker was last taken, but at the position of the list terminal 2 for displaying the marker. Correspondingly, the AR image may be configured to be continuously displayed.

Further, in the above-described embodiment, the marker is configured as an image of the two-dimensional code displayed on the screen of the list terminal 2, but the present invention is not limited to this. The marker may be configured to be electronically displayed as an image displayed on the screen of the list terminal 2, may be attached to the list terminal 2, or may have the shape of the list terminal 2.

Further, in the above-described embodiment, the target on which the marker is displayed may be configured to be recognizable. In the present embodiment, the list terminal 2 itself on which the marker is displayed may be configured to be recognizable. With this configuration, even if the marker cannot be detected, the AR image can be continuously displayed or the AR image can be continuously displayed while the list terminal 2 is recognized. It can be displayed on the list terminal 2 so as to follow it.

Further, in the above-described embodiment, if the marker is not detected for a certain period of time, it can be determined that the marker is not intentionally disappeared, and the continuation of the AR image display can be stopped. It may be configured to determine whether or not to continue the display of the AR image depending on how the marker disappears. The method of disappearing the marker is how it disappeared from the shooting range, for example, the direction of disappearance, the presence or absence of a posture change of the wrist terminal 2 before disappearance, the speed of disappearance, and the like. If the tilt of the screen changes and the marker simply cannot be detected within the shooting range, it is assumed that the marker moves within the shooting range and moves out of the shooting range so that it cannot be detected.

Further, in the above-described embodiment, it is determined whether or not the marker is intentionally hidden from the posture change of the list terminal 2, but the present invention is not limited to this. For example, it may be configured to determine whether or not the marker is hidden from the captured image. In this case, the images before and after the hidden image may be used to determine whether or not the display is intentional, and whether or not to continue the display may be determined.

Further, in the above-described embodiment, when the marker cannot be photographed due to the posture change of the wrist terminal 2, the AR image is continuously displayed, but the side that photographs the marker (the present embodiment). In the above, the AR image may be continuously displayed when the marker cannot be photographed due to the posture change of the mobile terminal 1) having a photographing function.

Further, in the above-described embodiment, the mobile terminal 1 and the wrist terminal 2 to which the present invention is applied have been described by taking a digital camera and a smart watch as an example, but the present invention is not particularly limited thereto.
For example, the present invention can be generally applied to electronic devices having an AR display processing function and a notification processing function. Specifically, for example, the present invention can be applied to notebook personal computers, printers, television receivers, video cameras, digital cameras, portable navigation devices, mobile phones, portable game machines, and the like.

The series of processes described above can be executed by hardware or software.
In other words, the functional configuration of FIG. 4 is merely an example and is not particularly limited. That is, it suffices if the mobile terminal 1 and the list terminal 2 are provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is particularly an example of FIG. Not limited to.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.
The functional configuration in the present embodiment is realized by a processor that executes arithmetic processing, and the processor that can be used in the present embodiment is composed of various processing units such as a single processor, a multiprocessor, and a multicore processor. In addition to the above, the present invention includes a combination of these various processing units and processing circuits such as an ASIC (Application Specific Integrated Circuit) and an FPGA (Field-Programmable Gate Array).

When a series of processes are executed by software, the programs constituting the software are installed on a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

The recording medium including such a program is not only composed of the removable media 31 of FIG. 3 distributed separately from the device main body in order to provide the program to the user, but also is preliminarily incorporated in the device main body. It is composed of a recording medium or the like provided to the above. The removable media 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versaille Disk), a Blu-ray (registered trademark) Disc (Blu-ray Disc), or the like. The magneto-optical disk is composed of an MD (Mini-Disc) or the like. Further, the recording medium provided to the user in a state of being preliminarily incorporated in the main body of the apparatus is composed of, for example, the ROM 12 of FIG. 3 in which the program is recorded, the hard disk included in the storage unit 20 of FIG.

In the present specification, the steps for describing a program recorded on a recording medium are not necessarily processed in chronological order, but also in parallel or individually, even if they are not necessarily processed in chronological order. It also includes the processing to be executed.
Further, in the present specification, the term of the system means an overall device composed of a plurality of devices, a plurality of means, and the like.

Although some embodiments of the present invention have been described above, these embodiments are merely examples and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and various modifications such as omission and substitution can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in the present specification and the like, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

The inventions described in the claims at the time of filing the application of the present application are described below.
[Appendix 1]
Identification information acquisition means for acquiring identification information from the identification information image,
A display control means for controlling the display unit to perform virtual display according to the identification information, and
A determination means for determining whether or not to continue the virtual display on the display unit based on the state of the identification information image when the identification information is no longer acquired by the identification information acquisition means is provided.
When the determination means determines that the virtual display is continuously performed, the display control means controls the display unit so as to continuously perform the virtual display based on the identification information.
An information processing device characterized by this.
[Appendix 2]
The identification information acquisition means acquires identification information from an electronic device including an identification information display unit on which the identification information image is displayed.
The information processing device according to Appendix 1, wherein the information processing device is characterized by the above.
[Appendix 3]
The determination means determines whether or not the virtual display is continuously performed based on the posture change of the identification information display unit.
The information processing device according to Appendix 2, wherein the information processing device is characterized by the above.
[Appendix 4]
The posture change of the identification information display unit is detected based on an image of the identification information display unit or the posture information of the identification information display unit.
The information processing device according to Appendix 3, wherein the information processing device is characterized by the above.
[Appendix 5]
The electronic device includes a sensor unit that acquires first information related to a posture change of the identification information display unit, and a sensor unit.
A transmission unit that transmits the first information acquired by the sensor unit to the information processing device is further provided.
The information processing device further includes a receiving unit that receives information transmitted by the transmitting unit.
The determination means determines whether or not the virtual display is continuously performed based on the first information received by the receiving unit.
The information processing device according to any one of Supplementary note 2 to 4, wherein the information processing device is characterized by the above.
[Appendix 6]
The identification information image is displayed on the screen of the identification information display unit or on the surface shape.
The information processing device according to any one of Supplementary note 2 to 5, wherein the information processing device is characterized by the above.
[Appendix 7]
An electronic device equipped with an identification information display unit that displays identification information, and
An information processing system consisting of an information processing device provided with a virtual display unit that performs virtual display based on identification information.
The information processing device
An identification information acquisition means for acquiring the identification information displayed on the identification information display unit, and
When the identification information is no longer acquired by the identification information acquisition means, a determination means for determining whether or not virtual display is continuously performed based on the state of the identification information display unit, and
When the determination means determines that the virtual display is continuously performed, the display control means that controls the virtual display unit so as to continuously perform the virtual display based on the identification information.
An information processing system characterized by being equipped with.
[Appendix 8]
An electronic device that displays the identification information to an external device that performs virtual display based on the identification information.
An identification information display unit that displays identification information,
A detection unit that detects a change in posture of the identification information display unit, and a detection unit.
While the identification information display unit is displaying the identification information, the information on the posture change of the identification information display unit detected by the detection unit is used as information as a determination criterion for continuing the virtual display. Output means to output to the device and
An electronic device characterized by being equipped with.
[Appendix 9]
It is an information processing method executed by an information processing device.
Identification information acquisition step to acquire identification information from the identification information image,
A display control step that controls the display unit to perform virtual display according to the identification information, and
A determination step for determining whether or not to continue the virtual display on the display unit based on the state of the identification information image when the identification information is no longer acquired by the identification information acquisition step is provided.
The display control step controls the display unit so as to continuously perform the virtual display based on the identification information when it is determined by the determination step that the virtual display is continuously performed.
An information processing method characterized by the fact that.
[Appendix 10]
A computer that controls an information processing device
Identification information acquisition means for acquiring identification information from the identification information image,
A display control means for controlling the display unit to perform virtual display according to the identification information.
When the identification information is no longer acquired by the identification information acquisition means, it functions as a determination means for determining whether or not to continue the virtual display on the display unit based on the state of the identification information image.
When the determination means determines that the virtual display is continuously performed, the display control means controls the display unit so as to continuously perform the virtual display based on the identification information.
A program characterized by that.

1 ... mobile terminal, 2 ... list terminal, 11 ... CPU, 12 ... ROM, 13 ... RAM, 14 ... bus, 15 ... input / output interface, 16 ... Imaging unit, 17 ... sensor unit, 18 ... input unit, 19 ... output unit, 20 ... storage unit, 21 ... communication unit, 22 ... drive, 31 ... removable media , 51 ... Imaging control unit, 52 ... Image analysis unit, 53 ... Display control unit, 54 ... Communication control unit, 71 ... AR image storage unit, 91 ... Display control unit, 91 ... Sensor information analysis unit, 93 ... Communication control unit, S ... AR display system

Claims (5)

An identification information acquisition means for acquiring identification information from an electronic device provided with an identification information display unit on which an identification information image is displayed, and an identification information acquisition means.
A display control means for controlling the display unit to perform virtual display according to the identification information, and
A determination means for determining whether or not to continue the virtual display on the display unit when the identification information is no longer acquired by the identification information acquisition means is provided.
The determination means determines that the virtual display is continuously performed when the tilt generation notification indicating that the identification information display unit included in the electronic device is tilted is acquired from the electronic device.
When the determination means determines that the virtual display is continuously performed, the display control means controls the display unit so as to continuously perform the virtual display based on the identification information.
An information processing device characterized by this. The electronic device includes a sensor unit that determines whether or not the identification information display unit is tilted, and a sensor unit that determines whether or not the identification information display unit is tilted.
The information processing device according to claim 1, further comprising a transmission unit that transmits the tilt occurrence notification to the information processing device when the sensor unit determines that the identification information display unit is tilted. An electronic device equipped with an identification information display unit that displays identification information, and
An information processing system consisting of an information processing device provided with a virtual display unit that performs virtual display based on identification information.
The information processing device
An identification information acquisition means for acquiring the identification information displayed on the identification information display unit, and
When the identification information is no longer acquired by the identification information acquisition means, a determination means for determining whether or not to continuously perform virtual display, and
When the determination means determines that the virtual display is continuously performed, the display control means that controls the virtual display unit so as to continuously perform the virtual display based on the identification information.
With
The determination means determines that the virtual display is continuously performed when the tilt generation notification indicating that the identification information display unit included in the electronic device is tilted is acquired from the electronic device. Processing system. It is an information processing method executed by an information processing device.
An identification information acquisition step of acquiring identification information from an electronic device provided with an identification information display unit on which an identification information image is displayed, and an identification information acquisition step.
A display control step that controls the display unit to perform virtual display according to the identification information, and
A determination step for determining whether or not to continue the virtual display on the display unit when the identification information is no longer acquired by the identification information acquisition step is provided.
The determination step determines that the virtual display is continuously performed when the tilt generation notification indicating that the identification information display unit included in the electronic device is tilted is acquired from the electronic device.
The display control step controls the display unit so as to continuously perform the virtual display based on the identification information when it is determined by the determination step that the virtual display is continuously performed.
An information processing method characterized by the fact that. A computer that controls an information processing device
Identification information acquisition means for acquiring identification information from an electronic device provided with an identification information display unit on which an identification information image is displayed.
A display control means for controlling the display unit to perform virtual display according to the identification information.
When the identification information is no longer acquired by the identification information acquisition means, it functions as a determination means for determining whether or not to continue the virtual display on the display unit.
The determination means determines that the virtual display is continuously performed when the tilt generation notification indicating that the identification information display unit included in the electronic device is tilted is acquired from the electronic device.
When the determination means determines that the virtual display is continuously performed, the display control means controls the display unit so as to continuously perform the virtual display based on the identification information.
A program characterized by that.

Images