JP7614276B2 - PROGRAM, ELECTRONIC DEVICE AND SERVER SYSTEM - Google Patents

Description

The present invention relates to a program for displaying an augmented reality image on the display unit of a portable computer equipped with a position and orientation measurement function for measuring position and orientation.

Augmented reality, which has become increasingly popular in recent years, is realized by synthesizing and displaying augmented reality objects (AR (Augmented Reality) content, such as objects that do not exist in the real world or information display objects) with images of the real world. It has a wide range of uses, including widely known uses in games (see, for example, Patent Document 1 and Patent Document 2), as well as in advertising and route guidance (see, for example, Patent Document 3).

JP 2013-122708 A JP 2017-182579 A JP 2016-053776 A

Hardware that users can use to realize augmented reality includes glasses-type wearable computers known as smart glasses, but these have not yet become widespread. The main hardware used by users in augmented reality, which has become increasingly popular in recent years, is electronic devices with a flat panel display on the front and a camera on the back, such as smartphones. Many smartphones are equipped with positioning sensors (e.g., GPS sensors that obtain latitude and longitude) to obtain location information, orientation sensors to measure direction, three-axis acceleration sensors, and three-axis gyro sensors, allowing them to realize location-aware augmented reality.

To use augmented reality, users must hold the smartphone in one or both hands, pointing it out in front of them, as if they were taking pictures of a landscape with a compact camera. However, this position is not suitable for long-term use as it tires the arms, and in some situations it may be necessary to avoid it from the standpoint of etiquette. There are times when users want to temporarily change the shooting direction.

For example, consider a service that uses location-aware augmented reality to support users who are sightseeing or shopping. The service provider prepares a large number of augmented reality objects that correspond to location information within the target geographical area. The user's smartphone selects each augmented reality object that is set within the shooting range of the camera at that time, and displays it by synthesizing it with the real image being captured by the camera at that time (image being captured: viewfinder image).

The user holds the smartphone in a position similar to taking pictures of scenery with a compact camera, and moves around the city relying on augmented reality objects (in this case, advertisements, event information, road signs, etc.) that are composited with the real image captured by the camera. However, it is tiring to keep the arm raised until the destination is reached. If the arm gets tired and the user puts the smartphone down, the shooting direction changes, and the augmented reality objects that were relied on for movement will no longer be displayed. In addition, the change in shooting direction will also change the real image being captured. If the user rests the arm and points the smartphone back in the original shooting direction, the previous real image will be displayed again, and the augmented reality objects that were relied on for movement should also be displayed again, but the user must search for the augmented reality objects again from the displayed image. Even for users who are confident in their strength, walking in a crowd with their arms constantly extended forward is not recommended from the perspective of etiquette and safety.

For example, suppose there is a game that uses location-aware augmented reality, in which a variety of characters are prepared and associated with location information, and the player must find and capture these characters. The user walks around the city, holding the smartphone in his or her hand in the same position as when taking pictures of the scenery with a compact camera. When the location information measured by the smartphone and the location information of a character come close to each other, an image of the character (augmented reality object) is displayed on the real image captured by the camera. The user enjoys the game by performing a specified operation to capture the displayed character, or by taking and saving a commemorative photo of the character that appears in the city. In this example, the problems of arm fatigue and etiquette are the same as in the previous example. If the camera's shooting direction is not maintained, the same real image cannot be displayed continuously, and the desired augmented reality object cannot be displayed continuously, resulting in the same inconvenience as in the previous example.

The present invention was conceived with the aim of providing a technology that reduces the inconvenience that occurs when temporarily changing the shooting direction when using augmented reality.

The first invention for solving the above-mentioned problems is a program for displaying an augmented reality image on the display of a computer having a photographing unit, a display unit, and a position and orientation measurement function for measuring position and orientation, the program causing the computer to function as a first display mode control means for controlling display of a first display mode in which an image of a given augmented reality object is synthesized with an image currently being photographed by the photographing unit and displayed on the display, and a second display mode control means for controlling display of a second display mode in which an image of the augmented reality object is synthesized with an image currently being photographed in the past and displayed on the display.

According to the first invention, there is a first display mode in which an augmented reality object (in other words, AR content) is composited and displayed on an image currently being shot, which is an image of the real world being shot at that time (in other words, a real-time shot image, a viewfinder image), and a second display mode in which an augmented reality object is composited and displayed on an image currently being shot in the past.

Therefore, for example, even if the shooting direction is temporarily changed while in the first display mode, by switching to the second display mode, the display of the augmented reality object that was displayed before the shooting direction was changed is maintained, and the augmented reality object is compositely displayed on the image captured in the past. This reduces the inconvenience of temporarily changing the shooting direction.

The second invention is a program of the first invention for causing the computer to further function as a mode switching control means for switching between the first display mode by the first display mode control means and the second display mode by the second display mode control means.

According to the second invention, it becomes possible to switch between the first display mode and the second display mode, improving usability when temporarily changing the shooting direction.

The third invention is the program of the second invention, in which the mode switching control means has a short break switching means for switching from the first display mode by the first display mode control means to the second display mode by the second display mode control means, and the second display mode control means controls the display of the second display mode by setting the image being shot that was displayed in the first display mode before switching by the short break switching means to the past image being shot.

According to the third invention, in the second display mode, the image being captured before switching to the second display mode is displayed, so that the user can see as if the display before switching to the second display mode was temporarily held (paused).

A fourth invention is a third invention program, in which the augmented reality object includes at least a character object, the first display mode control means performs display control of the first display mode by changing a display position of the character object in a display image in response to a change in a measurement result of the position and orientation measurement function, and the second display mode control means performs display control of the second display mode by displaying an image of the character object at a predetermined display position.

The fifth invention is the program of the fourth invention, in which the second display mode control means controls the display of the second display mode by displaying an image of the character object at the display position of the character object that was displayed in the first display mode before the switching by the pause switching means.

According to the fourth or fifth invention, when the augmented reality object is a character object, after switching to the second display mode, the same character object is displayed at the same display position as the character object displayed in the first display mode before switching. Therefore, it is convenient for the user when switching to the second display mode, since the display position of the character object does not change.

A sixth invention is a program according to any one of the third to fifth inventions, in which the augmented reality object includes at least a position information object associated with a position, and the second display mode control means controls the display of the second display mode by changing the display position of an image of the position information object displayed in the first display mode and displaying the image on the display unit, the image being shot being displayed in the first display mode before switching by the pause switching means being combined with the image being shot being displayed in the first display mode before switching by the pause switching means.

According to the sixth aspect of the present invention, the display position of the location information object can be made different between the first display mode and the second display mode. This provides an advantage, for example, in that it is easier for the user to recognize that the display mode has changed.

The seventh invention is a program according to any one of the third to sixth inventions, in which the short break switching means switches from the first display mode to the second display mode when a short break switching condition is satisfied, the short break switching condition including at least that the shooting direction calculated based on the measurement result of the position and orientation measurement function satisfies a predetermined shooting direction change condition.

According to the seventh invention, when a change in the shooting direction satisfies a predetermined shooting direction change condition, it becomes possible to automatically switch from the first display mode to the second display mode.

The eighth invention is the seventh invention program, in which the shooting direction change condition is determined based on the angle of change when the shooting direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function changes to approach the direction of gravity.

According to the eighth aspect of the invention, when the elevation/depression angle of the shooting direction changes so as to approach the direction of gravity, it becomes possible to automatically switch to the second display mode.

The ninth invention is the program of the seventh invention, in which the shooting direction change condition is defined as the shooting direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function changing from the angle range defined for the first display mode to the angle range defined for the second display mode.

According to the ninth aspect of the invention, the user can change the elevation/depression angle of the shooting direction to switch to the second display mode without performing a button operation, touch operation, or the like.

A tenth invention is a program according to any one of the seventh to ninth inventions, in which the augmented reality object includes at least a character object, and the pause switching condition further includes that an image of the character object is displayed in the first display mode.

According to the tenth aspect of the invention, when an image of a character object is displayed in the first display mode, it becomes possible to switch to the second display mode.

An eleventh aspect of the present invention is a program according to a tenth aspect of the present invention, in which the character object is associated with characteristic information that is level and/or attribute information, a plurality of types of short break switching conditions are prepared according to the characteristic information of the character object displayed in the first display mode, and the short break switching means switches from the first display mode to the second display mode when the short break switching condition of a type according to the characteristic information of the character object displayed in the first display mode is satisfied.

According to the eleventh invention, the conditions for switching the display mode can be diversified based on the characteristic information of the character object. For example, it is possible to set it so that the short break switching condition is easily satisfied for a low-level character object, but is not so easily satisfied for a high-level character object. For example, if the augmented reality service is a game, it is possible to create a mechanism that fosters a sense of challenge in the game, thereby making the game more interesting.

The twelfth invention is a program according to the tenth or eleventh invention for causing the computer to further function as a game progress control means for controlling the progress of a predetermined game whose progress changes based on the character object, and a game event control means for initiating a given game event when the pause switching means switches to the second display mode.

According to the twelfth invention, a game event can be triggered in response to switching to the second display mode.

The thirteenth invention is the program of the twelfth invention, in which the game event control means, when switching to the second display mode is performed by the pause switching means, changes the content of the game event based on the measurement result of the position and orientation measurement function before switching.

According to the thirteenth aspect of the invention, the content of the game event changes based on the measurement results of the position and orientation measurement function before switching to the second display mode, making the game more interesting.

The fourteenth invention is the program described in any one of claims seven to thirteen for causing the computer to further function as a switching notification control means for notifying the user that a switching to the second display mode has been performed by the pause switching means.

According to the fourteenth aspect of the invention, it is possible to notify that switching to the second display mode has occurred.

The fifteenth invention is a program according to any one of the third to fourteenth inventions, in which the mode switching control means has a recovery switching means that switches from the second display mode to the first display mode when a predetermined recovery switching condition is satisfied after switching to the second display mode by the short pause switching means.

According to the fifteenth aspect of the invention, when a predetermined recovery switching condition is satisfied, the second display mode can be switched to the first display mode.

The 16th invention is the program of the 15th invention, in which the recovery switching condition at least includes satisfaction of a predetermined shooting direction recovery condition indicating that the shooting direction related to the elevation/depression angles calculated based on the measurement results of the position and orientation measurement function has returned to the state before switching to the second display mode by the short break switching means.

According to the sixteenth aspect of the present invention, when the elevation/depression angle of the shooting direction is set to the state before switching to the second display mode, the display mode is switched to the second display mode. This makes it easier for the user to intuitively understand the switching of the display mode.

The seventeenth invention is a program according to any one of the first to sixteenth inventions for causing the computer to further function as an operation icon display control means for controlling the display of a given operation icon at a predetermined position on the display unit, regardless of the mode switching performed by the mode switching control means.

According to the seventeenth aspect of the present invention, the operation icons can be displayed separately from the switching of the display mode. For example, regardless of the display mode, operation icons for changing settings, calling up other functions, taking commemorative photos, etc. can be displayed in a predetermined position on the display unit.

The eighteenth invention is an electronic device that includes a photographing unit, a display unit, and a position and orientation measurement function that measures position and orientation, and displays an augmented reality image on the display unit, and includes a first display mode control means that performs display control of a first display mode in which an image of a given augmented reality object is synthesized with an image currently being photographed by the photographing unit and displayed on the display unit, a second display mode control means that performs display control of a second display mode in which an image of the augmented reality object is synthesized with an image currently being photographed in the past and displayed on the display unit, and the first display mode controlled by the first display mode control means.

According to the eighteenth invention, an electronic device can be realized that can achieve the same effects as the first invention.

The 19th invention is a server system that controls the display of an augmented reality image on the display of a user terminal that has a photographing unit, a display unit, and a position and orientation measurement function that measures position and orientation, and includes a first display mode control means that controls display of a first display mode in which an image of a given augmented reality object is synthesized with an image currently being photographed by the photographing unit and displayed on the display unit, a second display mode control means that controls display of a second display mode in which an image of the augmented reality object is synthesized with an image currently being photographed in the past and displayed on the display unit, and the first display mode controlled by the first display mode control means.

The term "server system" here refers not only to a single computer that functions as a server, but also to a system consisting of multiple computers working together.

According to the 19th aspect, a server system can be realized that provides the same effects as the first aspect.

FIG. 1 is a diagram showing an example of the configuration of a game system. A front view showing an example of the configuration of a user terminal. A conceptual diagram for explaining the game. FIG. 4 is a diagram showing an example of a display image in a first display mode. FIG. 11 is a diagram showing an example of a display image in a second display mode. 11A and 11B are diagrams for explaining examples of shooting direction change conditions. FIG. 4 is a functional block diagram showing an example of a functional configuration of a user terminal. FIG. 4 is a diagram showing examples of programs and data stored in a terminal storage unit. FIG. 4 is a diagram showing an example of the data configuration of augmented reality object definition data. FIG. 13 is a diagram showing an example of a data configuration of short break switching condition definition data. FIG. 4 is a diagram showing an example of the data configuration of notification pattern definition data. FIG. 13 is a diagram showing an example of the data configuration of augmented reality management data. 10 is a flowchart illustrating a process flow in a user terminal. FIG. 11 is a functional block diagram showing an example of the functional configuration of a server system according to a second embodiment. FIG. 11 is a functional block diagram showing an example of a functional configuration of a user terminal according to the second embodiment.

An example of an embodiment of the present invention will be described below, but it goes without saying that the forms to which the present invention can be applied are not limited to the following embodiment.

First Embodiment
1 is a diagram showing an example of the configuration of a game system in the first embodiment. The game system 1000 in this embodiment is a computer system including a server system 1100 and a plurality of user terminals 1500 (1500a, 1500, ...) that are capable of mutual data communication by being connected to a communication line 9.

The communication line 9 refers to a communication path that allows data communication. In other words, the communication line 9 includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) such as Ethernet (registered trademark), a telephone communication network, a cable network, the Internet, and other communication networks, and the communication method can be either wired or wireless.

The server system 1100 has a main unit 1101, a keyboard 1106, a touch panel 1108, and storage 1140, and the main unit 1101 is equipped with a control board 1150.

The control board 1150 is equipped with various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM, and ROM, and a communication device 1153. Note that a part or all of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a SoC (System on a Chip).

The server system 1100 realizes a user management function related to user registration etc., and a function of providing the data necessary for registered users, players 2 (2a, 2b, ...), to play the game on user terminals 1500 (1500a, 1500, ...), by the control board 1150 performing calculations based on predetermined programs and data.

Although the server system 1100 is illustrated as a single device, it may be configured with multiple blade servers each sharing a function, connected to each other via an internal bus for data communication. Alternatively, it may be configured such that multiple independent servers installed in remote locations communicate with each other via communication line 9, functioning as the server system 1100 as a whole.

The user terminal 1500 (1500a, 1500, ...) is a computer system that is used individually by the user player 2 (2a, 2b, ...) to play the game, and is an electronic device that can access the server system 1100 via the communication line 9. Functionally speaking, the user terminal 1500 is a game device. The user terminal 1500 in this embodiment is a device known as a smartphone, but it may also be a portable game device, a tablet computer, a compact digital camera, etc.

FIG. 2 is a front view showing an example of the configuration of a user terminal 1500 according to this embodiment.
The user terminal 1500 includes a directional input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a contact position input device, a speaker 1510, an internal battery 1509, a microphone 1512, an image sensor unit 1520, a control board 1550, and a memory card reader 1542 that can read and write data from a memory card 1540 that is a computer-readable storage medium. In addition, a power button, a volume control button, etc. that are not shown are provided. In addition, an IC card reader that can contactlessly read and write data from an IC card type credit card or prepaid card that can be used to pay for game play may be provided.

The control board 1550 is
1) Various microprocessors such as the CPU 1551, GPU, DSP, etc.
2) Various IC memories 1552 such as VRAM, RAM, ROM, etc.
3) A wireless communication module 1553 for wireless communication with a mobile phone base station or a wireless LAN base station connected to the communication line 9;
4) A direction sensor 1554 for measuring a direction;
5) A positioning module 1555 that acquires positioning information such as the latitude and longitude of the aircraft using a positioning system;
6) 3-axis acceleration sensor 1556,
7) 3-axis gyro 1557,
8) Equipped with an interface circuit 1558, etc.

The positioning module 1555 is a means for acquiring the user's position. In this embodiment, the positioning module 1555 is a means for acquiring position coordinates in real space using an existing positioning system. That is, the positioning module 1555 provides a positioning function in the user terminal 1500 by receiving a signal provided by the positioning system and outputting positioning information at a predetermined period (for example, every second). In this embodiment, the positioning module 1555 uses GPS (Global Positioning System) as the positioning system. Therefore, the positioning module 1555 can use an existing "GPS module" or "GPS receiver". The "positioning information" includes the positioning date and time (for example, UTC: Coordinated Universal Time), position coordinates (latitude, longitude, altitude), and direction. The positioning system to be used is not limited to GPS and can be selected as appropriate. For example, the positioning module 1555 can be replaced with a configuration of a communication device that wirelessly connects to a wireless base station of a mobile phone or a wireless base station of a Wi-Fi network, and a direction sensor. In other words, it is possible to obtain pre-measured location information for the connected wireless base station and use this as the device's position coordinates in real space.

The interface circuit 1558 includes a driver circuit for the touch panel 1506, a circuit for receiving signals from the directional input keys 1502 and button switches 1504, an output amplifier circuit for outputting audio signals to the speaker 1510, an input signal generation circuit for generating audio signals collected by the microphone 1512, a circuit for inputting image data of an image captured by the image sensor unit 1520, and a signal input/output circuit for the memory card reader 1542.

These elements mounted on the control board 1550 are electrically connected via a bus circuit or the like, and are connected so that data can be read and written and signals can be sent and received. Note that part or all of the control board 1550 may be configured using an ASIC, FPGA, or SoC. The control board 1550 stores programs and various data for realizing the functions of the user terminal of the game of this embodiment in IC memory 1552.

In this embodiment, the user terminal 1500 is configured to download the game program and various setting data from the server system 1100, but it may also be configured to read them from a separately obtained storage medium such as a memory card 1540. In the following, the image sensor unit 1520 will be referred to as the "camera" for simplicity.

[Game Description]
FIG. 3 is a conceptual diagram for explaining the game of this embodiment.
The game of this embodiment is a location-aware augmented reality game in which the playfield is a city in the real world, and the objective is to discover and collect phantom characters hidden in the real world. The game of this embodiment can also be considered an augmented reality service or content that uses augmented reality to provide the player with information on tourist spots and shopping spots to actively promote sightseeing, shopping, participation in events, and the like while playing in the city.

In the geographical area that constitutes the playfield, the game provider defines multiple augmented reality objects ARn (AR1, AR2, ...), each of which is associated with location information.

"Location information" includes at least latitude and longitude. It may also include altitude, and if in a building, the number of floors.

Information regarding various settings related to the augmented reality object ARn is defined by the augmented reality object definition data 510. The augmented reality object definition data 510 includes a unique object ID 511, a type 512 that corresponds to the apparent classification of the object, setting position coordinates 513, and object data 514. Of course, other data may also be included as appropriate.

The types of type 512 can be set as appropriate, but in this embodiment, they include "character objects" which are game elements, and "banner objects (location information objects)" which are objects used mainly to provide location information such as advertisements and route guidance. The former corresponds to the augmented reality objects AR8 and AR9 in FIG. 3. The latter corresponds to the augmented reality objects AR1 to AR7 in the figure. The data for displaying each on the screen is set in object data 514.

The set position coordinates 513 are position information associated with the augmented reality object. They may also be called a geotag.

The user terminal 1500 executes the game program to obtain the augmented reality object definition data 510 from the server system 1100. Then, based on the position information, attitude, and orientation measured by the user terminal 1500, it determines the camera's shooting direction Dp (meaning the shooting direction of the image sensor unit 1520), and determines the shooting range based on the camera's optical system specifications.

Then, based on the current position information of the player's device, an augmented reality object ARn whose associated position information is included in the camera's shooting range is selected, and the selected augmented reality object ARn is composited and displayed on the image currently being shot by the camera (also called a viewfinder image). In this embodiment, this type of augmented reality display control is referred to as the "first display mode."

FIG. 4 is a diagram showing an example of a display image in the first display mode.
A registered user, player 2, plays the game by holding the user terminal 1500 in one or both hands and supporting it so that it is extended forward, as if taking pictures of scenery with a compact camera (see FIG. 1). A game screen 20 and operation icons 22 (22a, 22b, ...) are displayed on the touch panel 1506. Of course, other displays such as a communication status display and a built-in battery remaining display can also be included as appropriate.

On the game screen 20, the currently selected augmented reality object ARn (AR2, AR3, AR4, AR5, AR8 in the example of Figure 4) is compositely displayed on the currently captured image 21, which is captured in real time by the camera (image sensor unit 1520).

How many of the augmented reality objects ARn included in the camera's shooting range are displayed and how - in other words, the display position, display orientation, and display size - a separate placement algorithm is prepared and followed. In this embodiment, "five" are selected in order of proximity to the user terminal 1500, and the closer they are to the user terminal 1500, the larger the display size is, and the farther they are, the higher up on the screen they are displayed. However, the display size of the character object (augmented reality object AR8 in Figure 4) is not included in this, and the display size is determined based on the perspective from the size indicated by the character settings.

As with conventional augmented reality, when the player 2 changes the shooting direction, the user terminal 1500 changes its position and orientation, which it detects as a measurement result, and changes the display position of the augmented reality object ARn in the displayed image in accordance with the change. Of course, the display position of the augmented reality object AR8, which is a character object, can also be changed in accordance with the change in the measurement result. Such control over the change in the display position is one of the features of image display using augmented reality.

In this embodiment, the display control of the game screen 20 can be switched from the first display mode to the second display mode, and from the second display mode back to the first display mode.

As a premise, the images 21 being shot are buffered for a certain period of time and automatically deleted in order from the oldest to the newest. In other words, the images 21 being shot are updated and recorded in the ring buffer in chronological order as they are shot.
In the "second display mode", the user terminal 1500 displays an image of the augmented reality object ARn by combining the image 21 of the past image being captured that is recorded in the ring buffer.

FIG. 5 is a diagram showing an example of a display image in the second display mode.
In the second display mode, among the past images 21 being photographed and stored in the ring buffer, the image 21 being photographed and displayed in the first display mode before the switching is set as the past image 21 being photographed to be displayed in the second display mode. Specifically, instead of the image being photographed in real time at that time, a past image recorded in the ring buffer a predetermined time or a predetermined number of drawing frames before the switching is executed is selected and read out, and is controlled to be displayed as the background of the game screen 20. Therefore, in the example of Fig. 5, it is the same as the image being photographed 21 that was used as the background in Fig. 4.

Then, in the second display mode, the user terminal 1500 notifies the player 2 that the display mode has changed.

The first notification is realized by changing the display position of the augmented reality object ARn from the display position in the first display mode.
Specifically, the user terminal 1500 realizes the first notification by changing the display position of the augmented reality object ARn of the "banner" type, which is a location information object, displayed in the image being shot 21 that was displayed in the first display mode before the switch, and synthesizing and displaying it.

More specifically, as shown in FIG. 5, the display position is changed so as to move toward the edge of the game screen 20. It is possible to set which edge of the four periphery of the game screen 20 the object is moved toward as appropriate. The object may be moved toward the upper edge, or toward the edge in the opposite direction to the change in shooting direction associated with the display mode change. For example, if a downward change in the shooting direction is measured when the display mode is changed, the object is moved toward the upper edge. The player understands that the display mode has changed from the appearance of the augmented reality object ARn that was displayed before the display mode was changed being moved toward the edge of the game screen 20.

However, for an augmented reality object ARn of the "character" type (augmented reality object AR8 in the example of FIG. 4), the user terminal 1500 controls the image of the object to be displayed at a predetermined display position. Specifically, the image of the object is displayed at the display position of the object that was displayed in the first display mode before switching to the second display mode.

In addition, even when switching to the second display mode, the operation icon 22 is displayed in the same position as in the first display mode.

The second notification is realized by the occurrence of a game event. That is, when switching to the second display mode, the user terminal 1500 triggers a given game event. The content of the game event can be set appropriately. For example, if the displayed augmented reality object ARn is a character object, a video related to the character may be played, a special motion of the character may be executed, or a mini-game using the character (such as a rock-paper-scissors game) may be executed. The example of FIG. 5 shows a state in which a rock-paper-scissors game event has been triggered, and an event notification 23 is displayed as a pop-up. Note that in this embodiment, the user terminal 1500 changes the content of the game event based on the measurement results of the position and orientation of the user terminal 1500 before switching.

In addition, even if the player 2 changes the shooting direction during the second display mode and the user terminal 1500 detects the change in position and posture, the user terminal 1500 does not change the display position of the augmented reality object ARn in the displayed image in response to the change as in the first display mode, but continues to maintain the display position. In other words, the selection of the augmented reality object ARn at any time in real time and the synthesis and display of the object on the image being shot 21 being shot by the camera are temporarily suspended.

Switching from the first display mode to the second display mode is performed by touching an icon that performs a short break operation among the operation icons 22 (the third operation icon 22c with the word "pause" written on it in the example of FIG. 4), but it can also be performed automatically when a predetermined condition is met.

The condition for automatic execution is called the "break switching condition." In this embodiment, the "break switching condition" is defined by combining several sub-conditions with AND or OR. The following examples can be used as sub-conditions that describe the break switching condition.

The first sub-condition is a condition for the augmented reality object ARn displayed in the first display mode, and can be called a "displayed object condition."
For example, the displayed object condition 522a may be, for example, "a character of level '1' or higher is being displayed" or "the number of displayed objects is five or more".
Also, for example, it may be that an image of a character object (an augmented reality object ARn whose type is "character") is displayed in the first display mode, and the characteristic information of the character object satisfies a given condition.
The "characteristic information" here includes information on the level and/or attributes associated with the character object, such as "Level 1 or higher (m is a natural number)", "All attributes", "Level 5 or higher and fire attribute", etc.

The second sub-condition is a condition regarding the game progress, and can be called a "game progress condition." The game progress condition 522b can be, for example, "the number of acquired characters is between 10 and 20" or "the number of events cleared is 5 or more." Of course, parameters other than these can also be used. Characteristics information can also be used.

The third sub-condition is a condition related to the shooting direction and can be called a "shooting direction condition."
The shooting direction condition 522c can be described as a condition related to the current shooting direction or a change in the shooting direction. For example, it can be set as "facing a 45° range east-west from the north". It can also be set as "the shooting direction calculated based on the measurement result of the position and orientation measurement function satisfies a predetermined shooting direction change condition".

The "imaging direction change condition" referred to here is preferably determined based on the angle of change when the imaging direction related to the elevation/depression angles calculated based on the measurement results of the position and orientation measurement function changes so as to approach the direction of gravity.
For example, as shown in FIG. 6, the case can be defined as follows: (1) the shooting direction Dp falls within a predetermined permissible range W1 (e.g., minus 5° to plus 80° when the horizontal direction is 0°, the depression angle side is negative, and the elevation angle side is positive); (2) the change in the shooting direction Dp falls within a predetermined range W2 (e.g., within 3°) for a predetermined period of time (e.g., 10 seconds) or more; (3) a downward change ΔDp occurs that exceeds a predetermined reference value (e.g., 10°); and (4) an acceleration change ΔGz such that the shooting position descends is detected.

For example, the "imaging direction change condition" can be defined as the imaging direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function changing from an angle range defined for the first display mode (e.g., -5° to +80°, with the horizontal direction being 0°) to an angle range defined for the second display mode (e.g., -15° or less).

When these examples of shooting direction change conditions are adopted, when the player puts down the hand holding the user terminal 1500, the display mode is automatically switched to the second display mode.
For example, when a player finds an augmented reality object ARn, which is a character object, and lowers his/her hand holding the user terminal 1500 to perform a capture operation, or when the player lowers his/her hand holding the user terminal 1500 to photograph the character object in a preferred composition, the display mode will automatically switch to the second display mode.
Of course, the display mode will also be automatically switched if the user puts down the user terminal 1500 simply because the hand holding it becomes tired, or if the user suddenly puts down the hand holding it to avoid bumping into another person.

The fourth sub-condition is a condition related to the position or change in position of the user terminal 1500, and can be called a "position condition." The position condition 522d can be set, for example, as "movement speed is equal to or greater than a reference value (e.g., 10 km/h)," "altitude changes by 20 m or more," "descent is detected after remaining stationary or nearly stationary for 5 seconds or more," etc.

The fifth sub-condition can be called the "attitude condition," which is a sub-condition related to the attitude or attitude change of the user terminal 1500. The attitude condition 522e can be set, for example, as "pitch angle -5° to +80°," "pitch angle change of -10° or more within 2 seconds," or "attitude change of 2G or more detected."

Because this embodiment is a game, it is preferable to configure the short break switching condition with the first sub-condition "an image of a character object is displayed in the first display mode," or to have a short break switching condition that includes at least the first sub-condition. Note that any of the first to fifth sub-conditions can be set to "no condition," which essentially means that there is no condition.

Switching from the second display mode to the first display mode is performed by touching an icon related to a short break operation among the operation icons 22 (the third operation icon 22c in the example of FIG. 5), and after switching to the second display mode, if a predetermined recovery switching condition is met, switching to the first display mode is automatically performed.

The "recovery switching condition" referred to here can be defined as a pair of the "break switching condition." For example, it is preferable to define the recovery switching condition to at least satisfy "a predetermined shooting direction recovery condition indicating that the shooting direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function has returned to the state before switching to the second display mode is satisfied." When the recovery switching condition is set with multiple OR conditions, it is preferable to set one of the multiple OR conditions as "a predetermined shooting direction recovery condition indicating that the shooting direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function has returned to the state before switching to the second display mode is satisfied." Also, if the user terminal 1500 is configured to time the elapsed time after switching to the second display mode, "a predetermined time has elapsed since switching" may be set as one of the OR conditions.

[Explanation of Functional Configuration]
7 is a functional block diagram showing an example of a functional configuration of a user terminal 1500 according to this embodiment. The user terminal 1500 according to this embodiment includes an operation input unit 100, an image capture unit 120, a device processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, and a device storage unit 500.

The operation input unit 100 outputs operation input signals corresponding to various operation inputs made by the player 2 to the device processing unit 200. For example, this can be realized by a push switch, a joystick, a touch pad, a trackball, etc. Examples of this include the directional input keys 1502, button switches 1504, and touch panel 1506 in FIG. 2.

In addition, in this embodiment, movement of the user terminal 1500 and changes in posture (changes in orientation) are also considered to be part of user operations. Therefore, the operation input unit 100 has a positioning unit 102, a direction measurement unit 104, an acceleration measurement unit 106, and a posture measurement unit 108.

The positioning unit 102 acquires the location information of the user terminal 1500 and outputs it to the terminal processing unit 200. This corresponds to the positioning module 1555 in Figure 2.

The direction measurement unit 104 measures the direction in which the user terminal 1500 is facing and outputs the direction to the device processing unit 200. This corresponds to the direction sensor 1554 in Figure 2.

The acceleration measurement unit 106 measures the acceleration occurring in the user terminal 1500 and outputs it to the device processing unit 200. This corresponds to the three-axis acceleration sensor 1556 in Figure 2.

The posture measurement unit 108 measures the posture change of the user terminal 1500 and outputs it to the device processing unit 200. This corresponds to the three-axis gyro 1557 in Figure 2.

The photographing unit 120 photographs the outside world, generates image data, and outputs the image data to the device processing unit 200. This corresponds to the image sensor unit 1520 in Figure 2.

The device processing unit 200 is realized by electronic components such as a microprocessor such as a CPU or GPU, and an IC memory, and controls the input and output of data between each functional unit including the operation input unit 100, the photographing unit 120, and the device storage unit 500. It controls the operation of the user terminal 1500 by executing various arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100, image data of images photographed by the photographing unit 120, and various data received from the server system 1100. This corresponds to the control board 1550 in Figure 2.

In this embodiment, the device processing unit 200 has a game management unit 210, a timing unit 280, a sound generation unit 290, an image generation unit 292, and a communication control unit 294.

The game management unit 210 executes various processes for managing the game. The game management unit 210 has an image-in-capture storage control unit 211, a game progress control unit 212, a first display mode control unit 214, a second display mode control unit 216, a game event control unit 218, a mode switching control unit 220, a switching notification control unit 230, and an operation icon display control unit 232.

The image-in-shooting storage control unit 211 controls the storage of images captured by the image capture unit 120 (images being captured at each time) in the device storage unit 500 for a predetermined period of time in the past. This is called ring buffer control. The stored images become "past images being captured."

The game progress control unit 212 controls the progress of a specific game whose progress changes based on character objects.

The first display mode control unit 214 performs display control of the first display mode in which an image of a given augmented reality object ARn is synthesized with an image being captured by the image capture unit 120 and displayed on the image display unit 392. At this time, the display control of the first display mode is performed by changing the display position of the augmented reality object ARn (including a character object) in the displayed image in response to changes in the measurement results of the position and orientation measurement function. In other words, the control unit 214 repeatedly executes control to select the augmented reality object ARn to be displayed and update its display position and display size one by one in response to changes in the measurement results such as the position information obtained by the positioning unit 102, the orientation measured by the orientation measurement unit 104, the movement measured by the acceleration measurement unit 106, and the orientation measured by the orientation measurement unit 108.

The second display mode control unit 216 performs display control of the second display mode in which an image of the augmented reality object ARn is synthesized with a past image being captured and displayed on the image display unit 392 .
Specifically, the second display mode control unit 216 performs display control of the second display mode by adopting the image during shooting displayed in the first display mode before the display mode is switched as the past image during shooting. Also, the second display mode control unit 216 performs display control so that the image of the augmented reality object ARn, which is a character object, is displayed at the display position of the character object displayed in the first display mode before the switching. For the image of the augmented reality object ARn, which is a position information object, the second display mode control unit 216 changes the display position and composites the image of the position information object displayed in the first display mode with the image during shooting displayed in the first display mode before the switching, and displays the composite image on the image display unit 392.

The game event control unit 218 triggers a given game event when switching to the second display mode. At that time, the game event control unit 218 can change the content of the game event based on the measurement results of the position and orientation measurement function before switching.

The mode switching control unit 220 controls switching of the display mode between the first display mode and the second display mode.
Specifically, the mode switching control unit 220 has a short break switching unit 222 .
The short break switching unit 222 switches from the first display mode to the second display mode when a short break switching condition is satisfied. In this embodiment, a plurality of types of short break switching conditions corresponding to characteristic information that may be set for the character object displayed in the first display mode are prepared, and the short break switching unit 222 performs control to switch from the first display mode to the second display mode when a short break switching condition of a type corresponding to the characteristic information of the character object displayed in the first display mode is satisfied.

In addition, the mode switching control unit 220 includes a recovery switching unit 224 .
The recovery switching unit 224 performs control to switch from the second display mode to the first display mode when a predetermined recovery switching condition is satisfied after switching to the second display mode.

When switching to the second display mode has been performed, the switching notification control unit 230 notifies the user that switching has been performed. In this embodiment, the first notification is provided by changing the display position of the augmented reality object ARn that was displayed in the first display mode before switching in the second display mode from the display position in the first display mode. Thus, part of the function of the switching notification control unit 230 is realized by the function of the second display mode control unit 216.

The operation icon display control unit 232 controls the display of a given operation icon at a specified position on the image display unit 392, regardless of whether the display mode is switched.

The timing unit 280 uses the system clock to measure the current date and time, time limits, etc.

The sound generation unit 290 is realized, for example, by a digital signal processor (DSP), a processor such as a voice synthesis IC, or an audio codec capable of playing audio files, and generates sound signals for game-related sound effects, background music, and various operation sounds, and outputs them to the sound output unit 390.
The sound output unit 390 is realized by a device that outputs sound such as sound effects and background music based on the sound signal input from the sound generation unit 290. This corresponds to the speaker 1510 in FIG.

The image generation unit 292 can generate images such as game images, etc. A signal for displaying the image is output to the image display unit 392.
The image display unit 392 displays various game images based on image signals input from the game screen display control unit 262. For example, this can be realized by an image display device such as a flat panel display or a head mounted display. In this embodiment, this corresponds to the touch panel 1506 in FIG. 2.

The communication control unit 294 executes data processing related to data communication, and realizes data exchange with external devices via the communication unit 394 .
The communication unit 394 realizes communication by connecting to the communication line 9. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc., and corresponds to the wireless communication module 1553 in FIG.

The terminal storage unit 500 stores programs and various data for implementing various functions for the terminal processing unit 200 to comprehensively control the user terminal 1500. It is also used as a working area for the terminal processing unit 200, and temporarily stores the results of calculations executed by the terminal processing unit 200 according to various programs and input data input from the operation input unit 100. These functions are implemented, for example, by IC memory such as RAM or ROM, or magnetic disks such as hard disks. This corresponds to the IC memory 1552 and memory card 1540 mounted on the control board 1550 in Figure 2. A configuration using online storage is also possible.

Figure 8 is a diagram showing examples of programs and data stored in the device storage unit 500. The device storage unit 500 includes a game program 502, augmented reality object definition data 510, pause switching condition definition data 520, notification pattern definition data 530, first display mode definition data 540, second display mode definition data 542, play data 700, and current date and time 800. Of course, other programs and data may also be included as appropriate.

The game program 502 is a program that is read and executed by the device processing unit 200 to realize the functions of the game management unit 210.

The augmented reality object definition data 510 is prepared for each augmented reality object ARn and stores various setting data related to the object. The original data is managed by the server system 1100, and the user terminal 1500 periodically downloads and updates the augmented reality object definition data 510 from the server system 1100.

As shown in FIG. 9, for example, one piece of augmented reality object definition data 510 includes a unique object ID 511, a type 512, set position coordinates 513, object data 514, unique notification control data 515, and characteristic information 516. Of course, other data may also be included as appropriate.

The unique notification control data 515 is data for notifying the player that the display mode has changed from the first display mode to the second display mode, and is data for realizing a notification unique to the object. For example, if the object is a position information object, data for realizing a change pattern in the display form or a motion such as jumping can be set. If the object is a character object, data for realizing a so-called cut-in image that displays a two-dimensional image of the character, data for realizing a special motion for the character, etc. can be set.

Characteristic information 516 stores information related to the object. For example, if the object is a location information object, such as an object indicating a store location, data such as the URL of the store's website, the store's opening hours and contact text, and an image of the store's exterior can be set. If the object is a character object, data such as the character's type, level, attributes, etc. can be set.

When the user terminal 1500 detects that the augmented reality object ARn of the position information object is displayed and that the player has touched it, if the characteristic information 516 includes a URL, it can launch a web browser, and if it includes text, it can display it as a pop-up.

Returning to FIG. 8, the short break switching condition definition data 520 is prepared for each type of short break switching condition, and stores various data regarding the corresponding condition.
10, one piece of short break switching condition definition data 520 includes a unique switching condition ID 521 and a short break switching condition 522. Of course, other data than these may also be included as appropriate.

The short break switching condition 522 defines the contents of the short break switching condition. The short break switching condition 522 is, for example,
1) A displayed object condition 522a which is a sub-condition for the augmented reality object ARn displayed in the first display mode; and
2) A game progress condition 522b which is a sub-condition regarding the game progress status;
3) an imaging direction condition 522c which is a sub-condition related to the imaging direction;
4) A location condition 522d that is a sub-condition related to the location or change in location of the user terminal 1500; and
5) A posture condition 522e, which is a sub-condition related to the posture or posture change of the user terminal 1500; and
Of course, other sub-conditions may be used as appropriate. The short break switching condition 522 also includes information on whether each sub-condition is an OR condition or an AND condition.

Returning to FIG. 8, notification pattern definition data 530 is prepared for each notification pattern that notifies the player that the display mode has been switched, and stores various settings for realizing that notification pattern. As shown in FIG. 11, for example, one notification pattern definition data 530 includes a unique pattern ID 531, application requirements 532 that must be satisfied for that notification pattern to be applied, and pattern content 533 that indicates the content of the applied notification pattern. Of course, other data than these may also be included as appropriate.

The application requirement 532 is described as a single sub-condition, or an AND or OR of multiple sub-conditions, similar to the short break switching condition 522 of the short break switching condition definition data 520.

Pattern content 533 stores data necessary to execute a notification. For example, for notification pattern definition data 530 in which application requirement 532 is that a particular character is displayed in the first display mode, the use of unique notification control data 515 of that character can be set as pattern content 533. For example, if a motion is defined in unique notification control data 515, notification is realized by controlling the display so that the character performs that motion. Also, when a certain game event is to be generated, data for executing that game event is set in pattern content 533, and notification is realized by executing that game event.

Returning to FIG. 8, the first display mode definition data 540 stores various data for determining at what position and at what size the augmented reality object ARn is to be displayed in the first display mode. For example, it includes a first display position calculation function, which is a function for determining the display position (first display position), and a first display size determination function, which is a function for determining the display size (first display size). Of course, data other than these may also be included as appropriate.

The second display mode definition data 542 stores various data for determining at what position and at what size the augmented reality object ARn is to be displayed in the second display mode. For example, it includes a second display position calculation function, which is a function for determining the display position (second display position), and a second display size determination function, which is a function for determining the display size (second display size). Of course, data other than these may also be included as appropriate.

The play data 700 stores various data describing the progress of the game.
For example, it includes a play start date and time 701, a captured character object ID list 703, and augmented reality management data 710. Of course, data other than these may also be included as appropriate.

The augmented reality management data 710 stores various data for managing the display of the augmented reality object ARn. For example, as shown in FIG. 12, the augmented reality management data 710 includes the latest position coordinates 711, the latest orientation 712, the latest attitude 713, the latest shooting direction 714, the latest shooting range 715, a ring buffer 720 for storing images being shot for a certain period of time, a list of object IDs to be displayed 721, object management data 730, a current display mode 740 indicating whether the first display mode or the second display mode is set, data for determining a short break switching condition 741, a switching date and time 742, and data for determining a recovery switching condition 743. Of course, other data than these may also be included as appropriate.

The display target object ID list 721 is a list of object IDs of augmented reality objects ARn defined in the augmented reality object definition data 510 that have been selected as display targets to be displayed in the image being captured based on the latest shooting direction 714 and latest shooting range 715 at that time.

Object management data 730 is prepared for each augmented reality object ARn to be displayed, and stores various data related to its display control. One object management data 730 includes a unique object ID 731, a display position 732, a display size 733, and a display operation 734. Of course, other data may also be included as appropriate.

The data for determining the short break switching condition 741 stores data required to determine whether or not the short break switching condition is satisfied. In this embodiment, the short break switching condition is set to include the time during which the shooting direction is stable, so for example, a counter value that counts the stable time during which the change in the shooting direction is kept within a range that satisfies a predetermined error condition is stored.

The switching date and time 742 stores the current date and time 800 when switching from the first display mode to the second display mode is performed.

The recovery switching condition determination data 743 stores data for determining the recovery switching condition. In this embodiment, at least a recovery switching condition is set such that the shooting direction related to the elevation/depression angles calculated based on the measurement results of the position and orientation measurement function has satisfied a predetermined shooting direction recovery condition indicating that the shooting direction has returned to the state before switching to the second display mode, so the recovery switching condition determination data 743 stores the shooting direction before switching to the second display mode.

[Operation Description]
Next, a description will be given of the flow of processing in the user terminal 1500. The flow of processing described here is realized by the terminal processing unit 200 executing the game program 502 in this embodiment.

Although omitted in the flowchart, after the game has started, each time a touch operation on an operation icon 22 is detected, the user terminal 1500 executes an operation corresponding process that executes processing corresponding to the operation icon 22 that has been touched.
For example, if a character object is included in the augmented reality object ARn to be displayed and a predetermined catching operation input is detected, the user terminal 1500 sets the character object as having been acquired. In other words, the game progresses.

For example, when the augmented reality object ARn to be displayed includes a position information object and it is detected that the player has touched the image, the user terminal 1500 executes a process of providing the player with characteristic information of the augmented reality object ARn (for example, displaying a website with a URL indicated by the characteristic information, displaying a characteristic image, etc.). It is assumed that the process associated with each of the other operation icons 22 is also executed.

13 is a flowchart for explaining the flow of processing in the user terminal 1500. The user terminal 1500 first starts periodically and continuously acquiring basic information for realizing a location-aware augmented reality (step S10).
Specifically, the user terminal 1500 starts positioning using the positioning module 1555, starts measuring the orientation of the shooting direction using the orientation sensor 1554, starts measuring the attitude of the user terminal 1500 using the three-axis gyro 1557, and starts measuring acceleration using the three-axis acceleration sensor 1556.

Next, the user terminal 1500 starts capturing images with the camera, and starts controlling the saving of the images being captured in the ring buffer 720 (step S12). The user terminal 1500 also starts sequentially updating the latest capturing direction 714 and the latest capturing range 715 (step S14), and starts controlling the progress of the game (step S16).

Next, when the game is started, the user terminal 1500 selects an augmented reality object ARn to be displayed, and stores the selection result in the display target object ID list 721 (step S20).
Then, the user terminal 1500 determines the display position and display size of each of the augmented reality objects ARn to be displayed based on the first display mode definition data 540 (step S22). Accordingly, for any of the augmented reality objects ARn to be displayed that does not have corresponding object management data 730, new object management data 730 is created.

Then, the user terminal 1500 displays the game screen (step S24). Specifically, the image of the augmented reality object ARn to be displayed is synthesized with the image being captured 21, the operation icon 22 is arranged, and the resulting image is displayed as the game screen.

The user terminal 1500 then determines whether input of a short break operation has been detected or whether a short break switching condition has been satisfied (step S60). Specifically, the user terminal 1500 makes a positive determination when it detects a touch operation on the third operation icon 22c corresponding to the short break switching operation. It also makes a positive determination when any of the short break switching conditions 522 in the short break switching condition definition data 520 (see FIG. 10) is satisfied.

If the answer is yes (YES in step S60), the user terminal 1500 temporarily suspends the selection process of the augmented reality object ARn to be displayed (step S62), sets the current display mode 740 to "2", sets the switching date and time 742 to the current date and time 800, and sets the recovery switching condition determination data 743 (step S64).

Next, the user terminal 1500 displays a game screen using the previous captured image instead of the latest captured image (step S65), and updates the display position and display size of the augmented reality object ARn to be displayed based on the second display mode definition data 542 (step S66).

Next, the user terminal 1500 selects a notification pattern and displays the notification (step S68). Specifically, the notification pattern definition data 530 (see FIG. 11) is searched for definition data for which the application requirement 532 is satisfied, and the pattern content 533 of the searched definition data is referenced, and the second notification is executed according to the definition. If a game event is set in the pattern content 533, the game event is triggered.

Next, the user terminal 1500 determines whether a predetermined game end condition is satisfied.
If the game end condition is not satisfied (NO in step S80) and the current display mode is the first display mode (first in step S82), the user terminal 1500 returns to step S20 and continues to control the game progress in the first display mode.

If the game end condition is not satisfied (NO in step S80) and the current display mode is the second display mode (step S82, second), the user terminal 1500 determines whether the recovery switching condition is satisfied (step S84). If the recovery switching condition is satisfied (YES in step S84), the process returns to step S20. If the recovery switching condition is not satisfied (NO in step S84), the process returns to step S65.

If the game end condition is met (YES in step S80), the user terminal 1500 ends the series of processes.

As described above, according to this embodiment, it is possible to switch between a first display mode in which an augmented reality object is compositely displayed on an image currently being captured, which is an image of the real world being captured at that time, and a second display mode in which an augmented reality object is compositely displayed on a past image currently being captured. For example, even in a situation in which the shooting direction is temporarily changed while in the first display mode, by switching to the second display mode, the display of the augmented reality object that was displayed before the shooting direction was changed is maintained, and the augmented reality object is compositely displayed on the past image currently being captured. This reduces the inconvenience of temporarily changing the shooting direction.

In addition, in the second display mode, the image being captured before switching to the second display mode is displayed, making it appear to the user as if the display before switching to the second display mode was temporarily held (paused).

In addition, if the augmented reality object is a character object, after switching to the second display mode, the same character object will be displayed in the same display position as the character object that was displayed in the first display mode before the switch. Therefore, it is convenient for the user when switching to the second display mode, since the display position of the character object does not change.

The display position of the location information object can be made different between the first display mode and the second display mode. This has the effect of making it easier for the user to recognize that the display mode has changed, for example.

In addition, if a change in the shooting direction satisfies a specified shooting direction change condition, it will be possible to automatically switch from the first display mode to the second display mode.

In addition, if the elevation/depression angle of the shooting direction changes so that it approaches the direction of gravity, the display mode can be automatically switched to the second display mode. Therefore, for example, if the shooting direction changes because the arm becomes tired and the arm is involuntarily lowered, the display mode can be automatically switched to the second display mode.

In addition, the user can change the elevation/depression angle of the shooting direction to switch to the second display mode without performing button or touch operations.

In addition, the conditions for switching the display mode can be diversified based on the characteristic information of the character object. For example, it is possible to set it so that the short break switching condition is easily met for a low-level character object, but not so easily met for a high-level character object, which can create a mechanism that fosters a sense of challenge in the game and improves the interest of the game.

Furthermore, by triggering a game event when the display mode is switched to the second display mode, the user can be informed in an easily understandable manner that the display mode has been switched to the second display mode. In addition, the switch to the second display mode can be used as a turning point in the game progress presentation to add excitement to the game.

In addition, the content of the game events can be changed based on the measurement results of the position and orientation measurement function before switching to the second display mode, which increases the variety of game events and makes the game more interesting.

In addition, the recovery from the second display mode to the first display mode includes at least satisfying a predetermined shooting direction recovery condition indicating that the shooting direction related to the elevation and depression angles calculated based on the measurement results of the position and orientation measurement function has returned to the state before switching to the second display mode. Therefore, when the player returns the orientation and posture of the user terminal 1500 to the state before switching to the second display mode, the first display mode can be restored. This makes it easier for the user to intuitively understand the switching of display modes.

In addition, a given operation icon can be displayed at a predetermined position regardless of whether the display mode is switched, so that operation icons such as changing settings, calling up other functions, taking commemorative photos, etc. can be displayed at a predetermined position on the display unit regardless of the display mode.

Second Embodiment
Next, a second embodiment to which the present invention is applied will be described. This embodiment is basically realized in the same way as the first embodiment, but differs in that the processing entity related to game management is the server system 1100. In other words, the game in this embodiment is realized as a kind of client-server type online game. Note that hereinafter, differences from the first embodiment will be mainly described, and components similar to those in the first embodiment will be given the same reference numerals and explanations will be omitted.

FIG. 14 is a functional block diagram showing an example of the functional configuration of a server system 1100 in this embodiment. The server system 1100 in this embodiment includes an operation input unit 100s, a server processing unit 200s, a sound output unit 390s, an image display unit 392s, a communication unit 394s, and a server storage unit 500s.

The operation input unit 100s is a means for inputting various operations for managing the server. This corresponds to the keyboard 1106 in FIG. 1.

The server processing unit 200s is realized by electronic components such as a microprocessor such as a CPU or GPU, an ASIC, and an IC memory, and controls the input and output of data between each functional unit including the operation input unit 100s and the server storage unit 500s. It performs various types of arithmetic processing based on predetermined programs and data, operation input signals from the operation input unit 100s, data received from the user terminal 1500, etc., and comprehensively controls the operation of the server system 1100.

The server processing unit 200s of this embodiment includes a user management unit 202, a game management unit 210, a timing unit 288s, a sound generation unit 290s, an image generation unit 292s, and a communication control unit 294s. Of course, other functional units may also be included as appropriate.

The user management unit 202 performs processing related to user registration procedures and manages the data of each user linked to a user account. In this embodiment, the user management unit 202 has the following functions: (1) assigning a unique user account to a registered user; (2) registration information management for registering and managing personal information for each user account; (3) ledger management for electronic payment media consumed for payment of billing elements; (4) play history management for managing login and logout history for playing games, and (5) management of game save data. Of course, management functions for other data linked to accounts other than these may also be included as appropriate.

The game of this embodiment is a client-server type online game, so the game management unit 210 of this embodiment can control the game progress and provide the data necessary for game play while communicating with the user terminal 1500.

The timing unit 288s uses the system clock to measure the current date and time, time limit, etc.

The sound generation unit 290s is realized by an IC or the like that generates and decodes audio data, and generates or decodes audio data such as operation sounds and background music related to system management and game play of the server system 1100. Then, audio signals related to system management are output to the sound output unit 390s.

The sound output unit 390s emits an audio signal. In the example of FIG. 1, this corresponds to a speaker (not shown) provided in the main device 1101 or the touch panel 1108.

The image generation unit 292s can generate images related to system management of the server system 1100, game images (or data for displaying game images on the user terminal 1500), etc. Then, the images related to system management can be output to the image display unit 392s.

The image display unit 392s displays various images for system management based on the image signal input from the image generation unit 292s. For example, this can be realized by an image display device such as a flat panel display, a projector, or a head-mounted display. In the example of FIG. 1, this corresponds to the touch panel 1108.

The communication control unit 294s executes data processing related to data communication and realizes data exchange with external devices via the communication unit 394s.

The communication unit 394s connects to the communication line 9 to realize communication. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc. In the example of Figure 1, it corresponds to the communication device 1153.

The server storage unit 500s stores programs and various data for implementing various functions that allow the server processing unit 200s to comprehensively control the server system 1100. It is also used as a working area for the server processing unit 200s, and temporarily stores the results of calculations that the server processing unit 200s executes according to various programs. This function is implemented, for example, by IC memory such as RAM or ROM, magnetic disks such as hard disks, optical disks such as CD-ROMs or DVDs, online storage, and the like. In the example of FIG. 1, this corresponds to storage media such as the IC memory 1152 and hard disk mounted on the main unit 1101, and the storage 1140.

The server storage unit 500s stores a server program 501, a distribution game client program 503, augmented reality object definition data 510, pause switching condition definition data 520, notification pattern definition data 530, first display mode definition data 540, second display mode definition data 542, user management data 600 prepared for each user and storing registration information, game save data, etc., play data 700, and current date and time 800. Of course, other data than these may also be included as appropriate.

Figure 15 is a functional block diagram showing an example of the functional configuration of a user terminal 1500 in this embodiment. Compared to the functional configuration of the user terminal 1500 in the first embodiment (see Figures 7 and 8), the game management unit 210 is omitted in this embodiment. Instead, a user terminal calculation unit 260 is provided.

The user terminal calculation unit 260 includes an operation signal transmission control unit 261 and a game screen display control unit 262.

The operation signal transmission control unit 261 executes processing to transmit various data and requests to the server system 1100 in response to operations performed on the operation input unit 100.

The game screen display control unit 262 performs control to display the game screen based on various data received from the server system 1100. The game screen also includes screens and images that accompany the execution of content. In this embodiment, the image data of the game screen is generated by the server system 1100, but it is also possible to generate the image data by the image generation unit 292.

The process flow executed by the server system 1100 in this embodiment is basically the same as that executed by the user terminal 1500 in the first embodiment (see FIG. 13), and in accordance with the game management unit 210 being executed by the server system 1100, the entity executing each step may be appropriately replaced with the server system 1100.

This embodiment provides the same effects as the first embodiment.

[Modifications]
Although the embodiment to which the present invention is applied has been described above, the forms to which the present invention can be applied are not limited to the above-described forms, and constituent elements can be added, omitted, or modified as appropriate.

[Modification 1]
For example, in the above-mentioned second embodiment, an example was given of implementing an online game in a client-server type computer system, but it may be implemented in a computer system in which a plurality of user terminals 1500 (or user terminals 1500), which may be considered game devices, are connected in a peer-to-peer manner. In this case, one of the user terminals 1500 is made to function as the server system 1100 in the second embodiment. Alternatively, it may be configured such that the functions of the game management unit 210 are shared among the plurality of user terminals 1500.

[Modification 2]
Furthermore, in the above embodiment, a game using location-aware augmented reality is exemplified, but the game element may be omitted and the application program may be implemented as a tourist information service or route guidance service.

[Modification 3]
In the above embodiment, an example was shown in which a past image being photographed was used as the background of the game screen in the second display mode, but a configuration in which the most recent image being photographed is used as the background, as in the first display mode, is also possible. Alternatively, a configuration in which the most recent image being photographed is used as the background, as in the first display mode, is also possible only when an operation input for taking a commemorative photograph is detected.

2...player 20...game screen 21...image currently being captured 22...operation icon 102...positioning unit 106...acceleration measurement unit 108...posture measurement unit 120...capturing unit 200...device processing unit 210...game management unit 211...image currently being captured storage control unit 212...game progress control unit 214...first display mode control unit 216...second display mode control unit 218...game event control unit 220...mode switching control unit 222...break switching unit 224...recovery switching unit 230...switching notification control unit 232...operation icon display control unit 500...device storage unit 502...game program 510...augmented reality object definition data 516...characteristic information 520...break switching condition definition data 522...break switching condition 522a...displayed object condition 522b...game progress condition 522c...shooting direction condition 522d...position condition 522e... Posture condition 530... Notification pattern definition data 540... First display mode definition data 542... Second display mode definition data 700... Play data 710... Augmented reality management data 711... Latest position coordinates 714... Latest shooting direction 720... Ring buffer 721... Display target object ID list 730... Object management data 741... Data for determining break switching condition 743... Data for determining recovery switching condition 1000... Game system 1100... Server system 1500... User terminal 1520... Image sensor unit 1550... Control board 1555... Positioning module 1556... 3-axis acceleration sensor 1557... 3-axis gyro ARn... Augmented reality object Dp... Shooting direction

Claims (7)

A program for displaying an augmented reality image on a display unit of a computer having an imaging unit, a display unit, and a position and orientation measurement function for measuring a position and orientation, the program comprising:
a first display mode control means for performing display control in a first display mode in which an image of a given augmented reality object is synthesized with an image being captured by the image capturing unit and displayed on the display unit;
a second display mode control means for performing display control in a second display mode switched from the first display mode, the second display mode control means performing display control of the second display mode by synthesizing an image of the augmented reality object with the image being shot that was displayed in the first display mode immediately before the switching and displaying the synthesized image on the display unit;
a mode switching control means for performing temporary switching control, which is a control for switching from the first display mode to the second display mode when an imaging direction calculated based on a measurement result of the position and orientation measurement function satisfies a predetermined imaging direction change condition in the first display mode by the first display mode control means, and switching from the second display mode to the first display mode when a recovery switching condition indicating that the imaging direction has returned to the state before switching to the second display mode is satisfied thereafter;
and causing the computer to function as
The augmented reality object includes at least a character object;
the mode switching control means switches from the first display mode to the second display mode when the shooting direction change condition is satisfied while the image of the character object is being displayed in the first display mode,
a game progress control means for controlling the progress of a predetermined game whose progress changes based on the character object;
a game event control means for triggering a given game event when the mode switching control means switches to the second display mode;
A program for causing the computer to further function as a the first display mode control means performs display control of the first display mode by changing a display position of the character object in a display image in response to a change in a measurement result of the position and orientation measurement function;
the second display mode control means performs display control of the second display mode by displaying an image of the character object at a predetermined display position.
The program according to claim 1 . the second display mode control means performs display control of the second display mode by displaying an image of the character object at a display position, in the display unit, of the character object that was displayed in the first display mode immediately before the switching.
The program according to claim 2 . the game event control means, when the mode switching control means switches to the second display mode, changes content of the game event based on a measurement result of the position and orientation measurement function before the mode switching.
The program according to any one of claims 1 to 3 . a switching notification control means for, when the mode switching control means switches to the second display mode, notifying the user that the mode switching has been performed;
The program according to any one of claims 1 to 4 , for causing the computer to further function as: 1. An electronic device comprising: a photographing unit; a display unit; and a position and orientation measurement function for measuring a position and orientation, the electronic device displaying an augmented reality image on the display unit,
a first display mode control means for performing display control in a first display mode in which an image of a given augmented reality object is synthesized with an image being captured by the image capturing unit and the synthesized image is displayed on the display unit;
a second display mode control means for performing display control in a second display mode switched from the first display mode, the second display mode control means performing display control of the second display mode by synthesizing an image of the augmented reality object with the image being shot that was displayed in the first display mode immediately before the switching and displaying the image on the display unit;
a mode switching control means for performing temporary switching control, which is a control for switching from the first display mode to the second display mode when an imaging direction calculated based on a measurement result of the position and orientation measurement function satisfies a predetermined imaging direction change condition in the first display mode by the first display mode control means, and switching from the second display mode to the first display mode when a recovery switching condition indicating that the imaging direction has returned to a state before switching to the second display mode is satisfied thereafter;
Equipped with
The augmented reality object includes at least a character object;
the mode switching control means switches from the first display mode to the second display mode when the shooting direction change condition is satisfied while the image of the character object is being displayed in the first display mode,
a game progress control means for controlling the progress of a predetermined game whose progress changes based on the character object;
a game event control means for triggering a given game event when the mode switching control means switches to the second display mode;
The electronic device further comprises : A server system that controls display of an augmented reality image on a display unit of a user terminal that includes a photographing unit, a display unit, and a position and orientation measurement function that measures a position and orientation,
a first display mode control means for performing display control in a first display mode in which an image of a given augmented reality object is synthesized with an image being captured by the image capturing unit and the synthesized image is displayed on the display unit;
a second display mode control means for performing display control in a second display mode switched from the first display mode, the second display mode control means performing display control of the second display mode by synthesizing an image of the augmented reality object with the image being shot that was displayed in the first display mode immediately before the switching and displaying the image on the display unit;
a mode switching control means for performing temporary switching control, which is a control for switching from the first display mode to the second display mode when an imaging direction calculated based on a measurement result of the position and orientation measurement function satisfies a predetermined imaging direction change condition in the first display mode by the first display mode control means, and switching from the second display mode to the first display mode when a recovery switching condition indicating that the imaging direction has returned to a state before switching to the second display mode is satisfied thereafter;
Equipped with
The augmented reality object includes at least a character object;
the mode switching control means switches from the first display mode to the second display mode when the shooting direction change condition is satisfied while the image of the character object is being displayed in the first display mode,
a game progress control means for controlling the progress of a predetermined game whose progress changes based on the character object;
a game event control means for triggering a given game event when the mode switching control means switches to the second display mode;
The server system further comprises :

Images