JP6147966B2 - Information processing program, information processing apparatus, information processing system, and information processing method - Google Patents

Description

  The present invention relates to an information processing program, an information processing apparatus, an information processing system, and an information processing method, and more particularly to an information processing program, an information processing apparatus, an information processing system, and an information processing method for displaying a panoramic video, for example.

  Conventionally, there is an image processing apparatus that displays a partial range of a panoramic image or an omnidirectional image on a display (see, for example, Patent Document 1).

JP 2004-187298 A

  However, the image processing apparatus described in Patent Document 1 only displays a panoramic video captured at a certain time on the display, and there are cases where the change of the displayed panoramic video is poor.

  Therefore, an object of the present invention is to provide an information processing program, an information processing apparatus, an information processing system, and an information processing method capable of displaying a panoramic video rich in change.

  In order to achieve the above object, the present invention may employ the following configuration, for example. When interpreting the description of the claims, it is understood that the scope should be interpreted only by the description of the claims, and the description of the claims and the description in this column are contradictory. In that case, priority is given to the claims.

  One configuration example of the information processing program of the present invention is executed by a computer included in an information processing apparatus that displays an image on at least one display device. The information processing program causes the computer to function as panoramic video acquisition means, display range setting means, display control means, and display target changing means. The panoramic video acquisition unit reads out and sequentially acquires the panoramic video as a display target from the panoramic video storage unit that stores a plurality of panoramic videos associated through predetermined parameters. The display range setting means sets the viewing direction for the acquired panoramic video, and sets the display range to be displayed on the display device based on the viewing direction. A display control means performs control which displays sequentially the panoramic video within a display range on a display apparatus among the acquired panoramic videos. The display target changing unit changes the panoramic video to be displayed when a predetermined switching condition is satisfied. When the panoramic video to be displayed is changed, the panoramic video acquisition means reads and sequentially acquires the panoramic video after the change. The display range setting means sets the line-of-sight direction set for the panoramic video acquired immediately before the change for the panoramic video acquired immediately after the change without being changed by the change.

  The “information processing device” may be configured by a device different from the display device, or may be configured by a display device when the display device has an information processing function. In the former case, each process of the present invention may be executed by “another device”, and the display device may only display the image generated by “another device”. In the case of having it, the information processing function of the display device and the information processing function of the “other device” may be realized in cooperation. Further, the “other device” may be a device that performs distributed processing by a plurality of information processing devices. The “information processing apparatus” may be a versatile information processing apparatus such as a general personal computer in addition to the game apparatus exemplified in the embodiments described later.

  The display device is, for example, a portable display device. Moreover, the said display apparatus is a stationary display apparatus as another example. In these cases, the display device is a display device that is wirelessly or wired connected to the information processing device, such as the terminal device 2 and the monitor 4 in the embodiments described later, and displays a display image generated by the information processing device. Anything is possible as long as it is possible. For example, the display device may be configured integrally with the information processing device (in one housing).

  The “panoramic video” is preferably a panoramic video having an angle of view of 180 degrees or more in one of the vertical direction and the horizontal direction. More preferably, the panoramic video is 360 degrees in one direction. Further, the other direction is preferably equal to or greater than the angle of view of the moving image displayed on the display device. Preferably there is.

  The “panoramic moving image storage means” may store a moving image captured by the panoramic moving image capturing function of the information processing apparatus, or a moving image captured by another device having the moving image capturing function. May be stored via a predetermined storage medium or network.

  The “display range setting means” typically moves the range to be displayed on the display device in the same direction with respect to at least the panoramic video according to the change in the attitude of the display device. Preferably, the “display range setting means” increases the amount of movement of the range as the amount of change in the attitude of the display device increases. More preferably, the “display range setting means” makes the change amount of the posture of the display device the same as the change amount of the posture of the virtual camera for generating the moving image in the above range. The “display range setting means” may control the change of the current posture of the virtual camera from the reference posture in accordance with the change of the current posture of the display device from the reference posture. In accordance with the change from the previous posture of the posture, the change from the previous posture of the current posture of the virtual camera may be controlled. The “posture” may be set in two dimensions, or may be set in three dimensions.

  According to the above, when a predetermined switching condition is satisfied, a different panoramic video can be displayed because it is switched and displayed in the same line-of-sight direction in another associated panoramic video.

  The plurality of panoramic videos may be contents captured at substantially the same place at different times.

  According to the above, when a predetermined switching condition is satisfied, it is possible to display a panoramic video as if the imaging time has changed instantaneously.

  The plurality of panoramic videos may be contents captured from substantially the same viewpoint.

  According to the above, when a predetermined switching condition is satisfied, it is possible to display a panoramic video as if only the imaging time has changed instantaneously.

  Further, the plurality of panoramic videos may be contents captured while moving the viewpoint along the same course in the real world.

  According to the above, when a predetermined switching condition is satisfied, it is possible to display a panoramic video in which the imaging time is instantaneously changed while moving on the same course.

  The plurality of panoramic videos may be contents captured in different time zones.

  According to the above, when a predetermined switching condition is satisfied, it is possible to display a panoramic video as if the imaging time zone has changed instantaneously.

  Further, the plurality of panoramic videos may be contents captured at a time when at least one of year, month, and day is different.

  According to the above, when a predetermined switching condition is satisfied, it is possible to display a panoramic video as if at least one of the imaging year, the imaging month, and the imaging date has changed instantaneously.

  Further, the panoramic video acquisition means, when the panoramic video to be displayed is changed, the frame in the panoramic video to be read after the change based on a predetermined parameter of the frame of the panoramic video read before the change. After the setting, the panoramic video after the frame may be read and sequentially acquired.

  According to the above, when switching the display of the associated panoramic video, playback of the panoramic video after switching can be started from the frame associated with the frame displayed before the switching.

  The panoramic video storage means may store a plurality of panoramic videos captured at different times while moving the viewpoint along the same course in the real world in association with each imaging location as a predetermined parameter. . In this case, when the panoramic video to be displayed is changed, the panoramic video acquisition means captures a panorama imaged at an imaging location substantially the same as the imaging location where the frame of the panoramic video read before the change is imaged. A frame of a moving image may be set as a frame to be read after the change, and after the change, a panoramic video after the frame may be read and sequentially acquired.

  According to the above, when switching the display of a plurality of panoramic videos captured at different times while moving the viewpoint along the same course in the real world, it is substantially the same as the panoramic video imaging location displayed before the switching. It is possible to start playback of the panoramic video after switching from the panoramic video captured at the same imaging location.

  The panoramic video storage means may store a plurality of panoramic videos captured at a time when at least one of the year, the month, and the day is different in association with each imaging time as a predetermined parameter. In this case, when the panoramic video to be displayed is changed, the panoramic video acquisition unit captures the panorama captured at substantially the same imaging time as the imaging time when the panoramic video frame read before the change was captured. A frame of a moving image may be set as a frame to be read after the change, and after the change, a panoramic video after the frame may be read and sequentially acquired.

  According to the above, when switching the display of a plurality of panoramic videos captured at a time when at least one of the year, the month, and the day is switched, the imaging is substantially the same as the panoramic video that was displayed before the switching. Playback of the panoramic video after switching can be started from the panoramic video captured at the time.

  Further, the panoramic video storage means may store a plurality of panoramic videos captured at substantially the same place at different times in association with frame numbers indicating the playback order of the respective frames as parameters. In this case, when the panoramic video to be displayed is changed, the panoramic video acquisition unit uses the same frame number as the panoramic video frame number read before the change as a panoramic video frame to be read after the change. After the setting, the panoramic video after the frame may be read and sequentially acquired.

  According to the above, when switching the display of a plurality of panoramic videos captured at substantially the same place at different times, from the frame with the same frame number as the frame number of the frame displayed before the switching, Panorama video playback can be started.

  The information processing program may further cause the computer to function as panoramic video arrangement means. The panoramic video arrangement means arranges the acquired panoramic video in the virtual space. In this case, the display range setting means sets a virtual camera for the panoramic video arranged in the virtual space, and sets the display range for the panoramic video according to the visual field range based on the viewing direction of the virtual camera. May be.

  Based on the above, it is possible to easily set a display range for a panoramic video using a virtual camera.

  The display object changing means may determine that the switching condition is satisfied when a predetermined switching operation is performed.

  According to the above, it is possible to switch the panoramic video to be displayed according to the user operation.

  Further, the panoramic video acquisition means reads out panoramic images as each frame of the panoramic video to be displayed from the panoramic video storage means at every predetermined time to be reproduced on the display device and sequentially acquires them to be displayed. When the panoramic video is changed, the panoramic image in the changed panoramic video may be read and sequentially acquired. In this case, the display range setting means sets the display range by setting the line-of-sight direction with respect to the acquired panoramic image. When the panoramic video to be displayed is changed, the panorama acquired immediately before the change is displayed. The display range may be set for the panoramic image acquired immediately after the change without changing the line-of-sight direction set for the image by the change.

  In addition, the present invention may be implemented in the form of an information processing apparatus and an information processing system including the above-described units and an information processing method including operations performed by the above units.

  According to the present invention, when a predetermined switching condition is satisfied, another panoramic video that is associated with the video is switched and displayed in the same line-of-sight direction, so that a panoramic video rich in change can be displayed. .

Block diagram showing an example of the information processing system 1 The figure which shows an example of the panoramic video (image) displayed in the information processing system 1 The figure which shows an example of the method of determining a display range according to the attitude | position of the terminal device 2 The figure which shows an example which switches to the image imaged in the place matched when switching several panoramic videos. The figure which shows an example of the panoramic video file data in which the several panoramic video used by the example of FIG. 4 was memorize | stored The figure which shows an example of the reproduction | regeneration operation | movement of the panoramic video switched according to switching operation in the example of FIG. The figure which shows an example which switches to the image imaged at the time matched when switching several panoramic videos. The figure which shows an example of the panoramic video file data in which the several panoramic video used in the example of FIG. 7 was memorize | stored The figure which shows an example of the reproduction | regeneration operation | movement of the panoramic video switched according to switching operation in the example of FIG. A figure showing an example of main data and a program memorized by memory 6 of information processor 3 The figure which shows the flowchart which shows an example of the first half of the process performed in the information processing apparatus 3 The figure which shows the flowchart which shows an example of the second half of the process performed in the information processing apparatus 3

[Configuration of information processing system]
With reference to FIG. 1, an information processing apparatus that executes an information processing program according to an example of the present embodiment and an information processing system including the information processing apparatus will be described. FIG. 1 is a block diagram illustrating an example of an information processing system 1 including the information processing apparatus 3. As an example, the information processing device 3 is configured by a stationary game device, and the information processing system 1 is configured by a game system including the game device.

  In FIG. 1, the information processing system 1 includes a terminal device 2, an information processing device 3, and a monitor 4. The information processing system 1 in the present embodiment generates an image (panoramic image) and displays it on a display device (terminal device 2 and / or monitor 4).

  In the information processing system 1, the information processing device 3 executes information processing in response to an input to the terminal device 2, and an image obtained as a result of the execution is displayed on the terminal device 2 and / or the monitor 4. Thus, in the present embodiment, the information processing system 1 has a configuration in which the input function, the information processing function, and the display function are realized by a plurality of devices. In other embodiments, the information processing system 1 may be configured by a single information processing apparatus (for example, a portable or portable information processing apparatus) having these functions.

  The terminal device 2 is a (portable) input device that can be held by a user. The terminal device 2 can communicate with the information processing device 3. The terminal device 2 transmits operation data representing an operation on the terminal device 2 to the information processing device 3. In the present embodiment, the terminal device 2 includes a display unit (LCD 11), and the terminal device 2 is also a display device. When an image is transmitted from the information processing device 3, the terminal device 2 displays the image on the LCD 11.

  The terminal device 2 includes a touch panel 12 as an input unit. The touch panel 12 is an example of a position detection unit that detects a position input to a predetermined input surface (screen of the display unit) provided in the housing. Furthermore, the terminal device 2 includes an operation unit 13 such as a slide pad, a cross key, and a button as an input unit.

  Moreover, the terminal device 2 includes an acceleration sensor 14 as an input unit. The acceleration sensor 14 detects an acceleration related to a predetermined axial direction of the terminal device 2 (in this embodiment, it is a triaxial direction, but it may be one or more axes). The terminal device 2 includes a gyro sensor 15 as an input unit. The gyro sensor 15 detects an angular velocity related to rotation about a predetermined axis direction of the terminal device 2 (in this embodiment, it is a triaxial direction but may be one or more axes). The acceleration sensor 14 and the gyro sensor 15 are sensors that detect information for calculating the attitude of the terminal device 2 (information that can calculate or estimate the attitude). In other embodiments, the attitude of the terminal device 2 may be calculated by any method, and the attitude of the terminal device 2 using a sensor other than the sensor or a camera capable of imaging the terminal device 2 is used. May be calculated.

  The information processing apparatus 3 executes various types of information processing executed in the information processing system 1 such as processing for generating an image. In the present embodiment, the information processing apparatus 3 includes a CPU (control unit) 5 and a memory 6, and various functions in the information processing apparatus 3 are executed by the CPU 5 using the memory 6 to execute a predetermined information processing program. Is realized. The information processing device 3 may have any configuration as long as the information processing can be executed. In the present embodiment, an image (panoramic image) is generated by the information processing device 3, and the generated image is output to the terminal device 2 and / or the monitor 4 that are display devices.

  The monitor 4 is an example of a display device that displays a generated image. The monitor 4 can receive data transmitted from the information processing device 3. When an image generated in the information processing apparatus 3 is transmitted to the monitor 4, the monitor 4 displays the image.

[Play panoramic video]
Next, processing for reproducing a panoramic video, which is executed in the information processing system 1, will be described. FIG. 2 is a diagram illustrating an example of a panoramic video (image) displayed in the information processing system 1. As shown in FIG. 2, in this embodiment, a panoramic image is displayed on the terminal device 2 and the monitor 4. In other embodiments, the panoramic image may be displayed on at least one display device.

  Here, the panoramic image is an image having a wider viewing angle than the viewing angle of the image displayed on the display device. In other words, the panoramic image basically has an image of a partial area displayed on the display device (however, when a plurality of display devices are used as in the present embodiment, the panoramic image is displayed on some display devices. The entire image may be displayed.) It can be said that the panoramic image is an image in which the direction of the line of sight changes as the display range changes. In the present embodiment, a panoramic image having viewing angles in all directions (360 °) with respect to the vertical and horizontal directions is used (see FIG. 3). However, the panoramic image may have a blind spot. For example, the viewing angle may be about 180 °. In this embodiment, the panoramic image is a real image (actual image) captured by a camera, but a part or all of the panoramic image may be a virtual image (CG image).

  In the present embodiment, the panoramic image that is each frame of the stored panoramic video is read out on the display device (the terminal device 2 and the monitor 4) every predetermined time as the panoramic video is played. The panoramic video is reproduced by sequentially displaying the panoramic images on the display device. Specifically, a plurality of panoramic images constituting a panoramic video are assigned frame numbers, and the panoramic images are displayed in the order of frame numbers. As an example, when displaying a panoramic video captured while moving the viewpoint in a predetermined space (for example, the real world) as time passes, a panoramic video in which the viewpoint moves as time passes for reproducing the panoramic video is played back. The For example, as shown in FIG. 2, a panoramic image of the surroundings displayed on the road is displayed, and a panoramic video moving along the road (the viewpoint moves along the road) is reproduced. As another example, when displaying a panoramic video imaged without moving in a predetermined space (that is, fixed-point imaging), the state around the viewpoint changes as time passes for reproducing the panoramic video. A panoramic video is played.

  As shown in FIG. 2, a partial range of the panoramic image is displayed on each display device (terminal device 2 and monitor 4). Hereinafter, a range displayed on the display device in the panoramic image is referred to as a “display range”. In FIG. 2, the monitor 4 has a display range in which the line-of-sight direction for displaying a panoramic image is the traveling direction of the viewpoint when the panoramic image is captured (the reference direction (front direction) when capturing a panoramic video). A panoramic image is displayed. In the present embodiment, the display range on the monitor 4 is basically fixed to a predetermined range, but the position and size of the display range with respect to the panoramic image may be changed according to a user operation. .

  On the other hand, the display range in the terminal device 2 is changed according to the attitude of the terminal device 2. For example, in FIG. 2, the terminal device 2 is oriented to the right with respect to the front direction (the direction from the user (terminal device 2) to the monitor 4). On the other hand, a panoramic image of the display range when the line of sight is directed to the right is displayed. Thus, the information processing system 1 determines a display range to be displayed on the display device among the panoramic images based on the input to the input device (terminal device 2). According to this, the user can view the panoramic video by freely changing the line-of-sight direction (display range) by the operation on the terminal device 2 (operation for changing the posture). In other embodiments, the display range may be determined by any method as long as it is determined so as to change according to the user's operation.

  FIG. 3 is a diagram illustrating an example of a method for determining a display range according to the attitude of the terminal device 2. In the present embodiment, as shown in FIG. 3, a stereo model is arranged in a virtual space, and a virtual camera is placed at a predetermined position inside the stereo model (in this embodiment, a position that is substantially the center of the stereo model). Place C1. Then, when generating an image of the three-dimensional model (the inner surface of the three-dimensional model) viewed from the virtual camera C1, a panoramic image is generated by drawing the panoramic image as a texture on the inner surface of the three-dimensional model. Specifically, the panoramic image is pasted on the inner surface of the stereoscopic model as a texture so that the reference direction when the panoramic video is captured is the reference direction of the stereoscopic model. At this time, in the panoramic image, the visual field range (shaded area A1 shown in FIG. 3) of the virtual camera C1 is the display range. In the present embodiment, the shape of the three-dimensional model is a cube, but the shape of the three-dimensional model is arbitrary, and may be, for example, a sphere or a columnar shape (for example, a cylinder).

  In FIG. 3, the attitude of the virtual camera C <b> 1 for generating an image to be displayed on the terminal device 2 is controlled according to the attitude of the terminal device 2. Thereby, the display range can be changed according to the attitude of the terminal device 2. Hereinafter, an example in which the attitude of the virtual camera C1 is controlled according to the attitude of the terminal device 2 will be described.

  First, as an initial setting, the reference posture of the terminal device 2 is set. For example, the reference posture of the terminal device 2 is set according to the posture of the terminal device 2 at a predetermined timing before or after the start of panoramic video reproduction. Specifically, the posture of the terminal device 2 at the start of panoramic video playback may be set as the reference posture, or the posture of the terminal device 2 when the user performs a predetermined operation before the start of panoramic video playback. It may be set as a reference posture, a predetermined fixed terminal device 2 posture may be set as a reference posture, or a user selects a predetermined fixed plurality of terminal device 2 postures. Also good. In the present embodiment, since the attitude of the terminal device 2 is calculated based on the output value of the gyro sensor 15, the setting of the reference attitude is a reset of the attitude value calculated by the gyro sensor 15. However, the setting of the reference posture of the terminal device 2 may be performed appropriately depending on the type of sensor.

  Further, as the initial setting, the initial posture of the virtual camera C1 is set. For example, in the initial posture of the virtual camera C1, the line-of-sight direction (z-axis positive direction) matches the Z-axis positive direction in the virtual space (the reference direction of the stereo model and the reference direction (front direction) in panoramic video imaging). The left direction (x-axis positive direction) coincides with the X-axis positive direction of the virtual space, and the upward direction (y-axis positive direction) coincides with the Y-axis positive direction of the virtual space.

  Then, the posture change of the terminal device 2 with respect to the reference posture (the xt axis that is the left and right direction of the terminal device 2 in the reference posture, the yt axis that is the vertical direction of the terminal device 2, and the zt axis that is the front and rear direction of the terminal device 2 respectively. The orientation of the virtual camera C1 is changed from the initial orientation (typically, the x-axis, y-axis, z-axis of the virtual camera C1 in the initial orientation, that is, the X-axis, Y-axis). The axis and the Z axis are changed with the same rotation direction and the same rotation amount). In addition to the position of the display range, the information processing system 1 may change the size of the display range (zoom-in or zoom-out) according to an operation on the terminal device 2.

  On the other hand, a virtual camera (not shown) for generating an image to be displayed on the monitor 4 is basically a reference direction (here, the above-mentioned traveling direction (front direction) is a panoramic video image. And is set to face the reference direction (Z-axis positive direction) of the three-dimensional model. That is, the display range of the monitor 4 (shaded area A2 shown in FIG. 3) is set at a position in the front direction from the virtual camera. The position of this virtual camera is also fixed at a predetermined position (in this embodiment, the center position of the stereo model).

  In the present embodiment, the relationship between the line-of-sight direction of the panoramic image displayed on the monitor 4 and the line-of-sight direction of the panoramic image displayed on the terminal device 2 is the depth direction of the display screen on the monitor 4 and the terminal device 2. The line-of-sight direction of the virtual camera C <b> 1 is controlled so as to substantially match the relationship with the depth direction of the LCD 11. Specifically, when the reference orientation is set in a state where the depth direction of the LCD 11 of the terminal device 2 faces the monitor 4, the line-of-sight direction of the virtual camera C <b> 1 is the virtual for the monitor 4 in this state. Initially set in the same direction as the viewing direction of the camera. When the attitude of the terminal device 2 changes from the reference attitude, the attitude of the virtual camera C1 changes in an amount corresponding to the change amount in a direction corresponding to the change direction. According to this, the user views a landscape in a desired gaze direction (a direction other than the front direction) by changing the direction of the terminal device 2 while viewing the landscape in a certain gaze direction (front direction) on the monitor 4. Can do. Further, the user can easily display the panorama image in the front direction on the LCD 11 by directing the depth direction of the LCD 11 of the terminal device 2 toward the monitor 4, and display the panorama image on the position of the monitor 4. Can be handled as a reference (standard for displaying the front direction of the panoramic image). Furthermore, by matching the change in the posture of the terminal device 2 and the change in the posture of the virtual camera C1, the orientation of the terminal device 2 in the real space matches the line-of-sight direction of the virtual camera C1 in the virtual space. Can feel the space represented by the panoramic image more realistically.

[Switch panorama image]
Next, a process for switching to another panoramic video in a state in which the panoramic video is being reproduced will be described with reference to FIGS. FIG. 4 is a diagram illustrating an example of switching to an image captured at a location associated with switching of a plurality of panoramic videos. FIG. 5 is a diagram illustrating an example of panoramic video file data in which a plurality of panoramic videos used in the example of FIG. 4 is stored. FIG. 6 is a diagram illustrating an example of a panoramic video playback operation that is switched according to the switching operation in the example of FIG. 4. FIG. 7 is a diagram illustrating an example of switching to an image captured at a time associated with switching a plurality of panoramic videos. FIG. 8 is a diagram illustrating an example of panoramic video file data in which a plurality of panoramic videos used in the example of FIG. 7 is stored. FIG. 9 is a diagram illustrating an example of a panoramic video playback operation that is switched according to the switching operation in the example of FIG. 7.

  As described above, on the terminal device 2, as the panoramic video playback progresses, panoramic images that are stored in each frame of the panoramic video are read at predetermined time intervals and sequentially displayed. Here, when a predetermined switching operation is performed using the terminal device 2 during panoramic video playback, the panoramic video being played back is switched to another panoramic video associated with the panoramic video. For example, as shown in FIG. 4, when a panoramic image is displayed from the road during the daytime and a switching operation is performed during playback of a panoramic video traveling on the road, The night panorama image that has been viewed is displayed and switched to a panoramic video traveling on the road. When switching the panoramic video, as shown in FIG. 4, the panoramic image is switched to a panoramic image captured at substantially the same place in different time zones (for example, daytime and nighttime), and the line-of-sight direction (display range) with respect to the panoramic image is displayed. ) Can be switched without changing. As an example, two panoramic videos taken in the daytime and at night while moving the same course defined in the real world (that is, taken in different time zones while moving the viewpoint along the same course) are related to each other. It is prepared. When a switching operation is performed during the playback of one panoramic video, among the panoramic images constituting the other panoramic video, substantially the same imaging location as the imaging location where the panoramic image reproduced on the other side is captured A panoramic image captured at a place is a display target. Then, the panoramic image newly displayed is pasted as a texture on the stereoscopic model, and the view range of the virtual camera is changed to the terminal device 2 without changing the attitude of the virtual camera for display on the terminal device 2 before and after switching. By displaying, the panoramic video is switched as described above. As a result, while changing the time zone of the real world expressed by the panoramic image before and after switching, the display range (display target) for the real world expressed by the panoramic image becomes the same before and after switching.

  Specifically, as shown in FIG. 5, a panoramic video A captured in the daytime along a predetermined real-world course is captured with the panoramic image IAn together with the panoramic image IAn data of each frame. Data indicating the imaging location Pn (for example, GPS information) is stored. In addition, the panoramic video B captured at night along the same real-world course stores the data of the panoramic image IBn serving as each frame and data indicating the imaging location Pn where the panoramic image IBn is captured. ing. Here, the panorama image IAn and the panorama image IBn each have a reference direction for capturing a panoramic video. For example, the traveling direction when moving and capturing a panoramic video is set as the reference direction. Yes. Then, the panorama image IAn and the panorama image IBn are pasted on the three-dimensional model so that the reference direction (Z-axis positive direction) of the three-dimensional model coincides with the reference direction when capturing a panoramic video. It is preferable that the moving speed in the real world when capturing the panoramic video A and the moving speed in the real world when capturing the panoramic video B are the same, but there is a difference in the moving speed depending on the imaging environment. There can be. In this case, for example, as described in frame number 5 in FIG. 5, different imaging locations Pn (imaging locations P5 and P4) are stored for panoramic images IAn and IBn having the same frame number.

  Then, as shown in FIG. 6, it is assumed that in the reproduction of the panoramic video A, a switching operation is performed at the time when the panoramic image captured at the imaging location Px is displayed. In this case, among the panoramic images constituting the panoramic video B, a panoramic image captured at the imaging location Px is extracted as a display target, and playback of the panoramic video B is started using the panoramic images after the panoramic image. That is, when the panoramic videos A and B are generated by capturing the same course indicated by the imaging locations P1 to Pe, respectively, the terminal device 2 has the daytime panoramic video A captured at the imaging locations P1 to Px. Is displayed and switched to the night panoramic video B imaged at the imaging locations Px to Pe by the switching operation. When switching to the panoramic video B, if the panoramic video B has a plurality of panoramic images captured at the imaging location Px, any one of the panoramic images (for example, a panoramic image captured at the imaging location Px) Among them, the panorama image captured first or the panorama image captured last) may be the display target at the time of switching.

  In this way, by performing a switching operation while the panoramic video is being played back on the terminal device 2, a paired panoramic video (imaging location (imaging target) is associated with the panoramic video while maintaining the same line-of-sight direction. (Same panorama video with different imaging time zone). Then, while changing the time zone of the real world represented by the panoramic image before and after switching, the display range (display target) for the real world represented by the panoramic image is the same before and after switching. The terminal device 2 can display an image as if the time zone has changed instantaneously. In the above description, panoramic videos having different imaging time zones are used. However, panoramic videos in which at least one of the imaging year, the imaging month, and the imaging date are captured at different imaging timings may be used.

  As another example, a plurality of panoramic videos captured at a fixed point without moving in a predetermined space may be used. For example, as shown in FIG. 7, when a switching operation is performed during playback of a panoramic video imaged at a fixed point in the spring real world, the panoramic video is switched from the same fixed point to an autumn panoramic video that looks at the real world. When the panoramic video is switched, as shown in FIG. 7, the same fixed point and different seasons (for example, spring and autumn) and different dates (for example, recent and several years ago, at least year, month, and day) Panorama images captured at the same time (if one is different), and can be switched without changing the viewing direction (display range) with respect to the panorama image. As an example, two panoramic videos captured at a fixed point in the real world in the same time zone in different seasons are prepared in association with each other. When a switching operation is performed during playback of one panoramic video, among the panoramic images constituting the other panoramic video, the image is captured at the same imaging time as the imaging time when the panoramic image reproduced on the other side is captured. The panoramic image that has been displayed is the display target. Then, the panoramic image newly displayed is pasted as a texture on the stereoscopic model, and the view range of the virtual camera is changed to the terminal device 2 without changing the attitude of the virtual camera for display on the terminal device 2 before and after switching. By displaying, the panoramic video is switched as described above. As a result, the real-world time represented by the panoramic image and the display range (display target) for the real world represented by the panoramic image are the same before and after the switching while changing the real-world season represented by the panoramic image before and after the switching. Become.

  Specifically, as shown in FIG. 8, a panoramic video C imaged at a fixed point in the spring of the real world includes an image capturing time Tn when each panoramic image ICn is imaged together with data of the panoramic image ICn serving as each frame. Each of the data shown is stored. Further, the panoramic video D captured at the same fixed point in the autumn of the real world stores data indicating the imaging time Tn at which each panoramic image IDn was captured, together with the data of the panoramic image IDn that is each frame. . Here, the panorama image ICn and the panorama image IDn each have a reference direction for panoramic video imaging. For example, when a panoramic video is captured at a fixed point, the imaging direction is fixedly set in the real world. (For example, the horizontal direction having a predetermined direction in the real world) is set as the reference direction. Then, the panorama image ICn and the panorama image IDn are pasted on the three-dimensional model so that the reference direction (Z-axis positive direction) of the three-dimensional model coincides with the reference direction when capturing a panoramic video.

  Then, as shown in FIG. 9, it is assumed that in the reproduction of the panoramic video C, the switching operation is performed at the time when the panoramic image captured at the imaging time Tx is displayed. In this case, out of the panoramic images constituting the panoramic video D, the panoramic image captured at the imaging time Tx is extracted as a display target, and the playback of the panoramic video D is started using the panoramic images after the panoramic image. That is, when the panoramic videos C and D are generated by imaging at the same fixed point from the imaging time T1 to the imaging time Te, respectively, the terminal device 2 has the spring panoramic video C captured at the imaging times T1 to Tx. After being displayed, it is switched to the autumn panoramic video D captured at the imaging times Tx to Te by the switching operation and displayed.

  In this way, by performing a switching operation while playing a panoramic video imaged at a fixed point in the terminal device 2, a pair of panoramic video images (for example, imaging) associated with the panoramic video image while maintaining the same line-of-sight direction. It is switched to playback of a panoramic video having the same location (imaging target) and imaging time but different imaging seasons and imaging dates. Then, while changing the season and date of the real world represented by the panoramic image before and after switching, the display range (display target) and imaging time for the real world represented by the panoramic image are the same before and after switching. Therefore, it is possible to display an image on the terminal device 2 as if the season or date has changed instantaneously according to the switching operation.

  Note that the panoramic video to be switched may be a video that is paired by another association as long as it is a video captured from substantially the same place (viewpoint) at different times. For example, in the above-described example, a panoramic video that is switched depending on an imaging location and an imaging time is associated. However, a movie that is captured from substantially the same location (viewpoint) at different times may be associated with a frame number. Specifically, in the case of the example described with reference to FIGS. 4 to 6, when imaging is performed with the same moving speed and moving course when imaging in the real world, the imaging location corresponding to each frame number is Although it is the same, in reality, it is difficult to make the imaging locations corresponding to the frame numbers the same. However, the panorama video is corrected in advance so that the imaging locations corresponding to the frame numbers are substantially the same by deleting a part of the frame of one panoramic video or adding a frame. By making the (reproduction time) the same, it is possible to switch to panoramic images at substantially the same imaging location even if they are associated by frame numbers. In the case of the example described with reference to FIGS. 7 to 9, the panoramic video having the same imaging time is cut out from the panoramic video captured in the real world with different seasons and dates, thereby corresponding to the frame number. By correcting in advance so that the imaging times are substantially the same, and by making the panoramic video length (reproduction time) the same, even if they are associated by frame numbers, the panorama images are switched to substantially the same imaging time. It becomes possible. Note that when a plurality of panoramic videos captured at a fixed point in the real world are associated by frame numbers, the time zones when each panoramic video is captured may be different. In addition, as described above, by making the length (playback time) of a plurality of panoramic videos to be switched the same, it is substantially possible to associate with the time (playback elapsed time) from the start of playback to the present time. It is possible to switch to a panoramic image at the same imaging location or a panoramic image at substantially the same imaging time.

  In the above description, when the panoramic video to be displayed is switched, the panoramic image (frame) associated with the panoramic image (frame) displayed immediately before the switching is extracted from the panoramic video after switching, An example of reproducing a panoramic video after the panoramic image (frame) after the switching is used. However, the panoramic video after switching may be played back from the frame associated at the time of switching, or may be played back from the frame next to the frame associated at the time of switching. For example, in the reproduction of the panoramic video A, when a switching operation is performed at the time when a panoramic image captured at the imaging location Px is displayed, the next of the imaging location Px among the panoramic images constituting the panoramic video B is displayed. The panoramic video B may be reproduced using a panoramic image after the panoramic image captured at the imaging location (imaging location Px + 1). Further, in the reproduction of the panoramic video C, when a switching operation is performed at the time when the panoramic image captured at the imaging time Tx is displayed, the next of the imaging time Tx among the panoramic images constituting the panoramic video D. The panoramic video D may be reproduced using the panorama images after the panorama image captured at the imaging time (imaging time Tx + 1). Further, when the panoramic video is associated with the frame number, the panoramic images after the frame number X (frame number X + 1) after the associated frame number X are used among the panoramic images constituting the panoramic video after switching. The panoramic video after switching may be played back.

  In the above description, an example is used in which switching is performed between associated panoramic videos, that is, between two associated panoramic videos. However, switching is performed between three or more associated panoramic videos. It can be done. The panoramic videos to be prepared in association with each other are videos taken from substantially the same place (viewpoint), having different imaging time zones, different imaging seasons, different imaging years, different imaging months, and Any moving image having different imaging conditions selected from at least one of imaging dates (imaging days) may be used. In addition, by preparing and switching panoramic videos taken before and after the subject to be imaged (subject) changes (for example, subject change due to weather, subject change due to renovation or reconstruction, subject change due to growth), further images can be obtained. A variety of panoramic videos can be switched.

  In the above description, an example in which a panoramic video is switched according to a user's switching operation is used. However, a panoramic video may be automatically switched when a predetermined switching condition is satisfied. As an example, when a predetermined object is displayed within the display range of the terminal device 2, it may be automatically switched to another associated panoramic video. As another example, the panoramic video may be automatically selected according to the current time or date when the user is watching the panoramic video. For example, when the information processing apparatus 3 includes a device that acquires the current time such as an RTC (real time clock), the image is captured in the night time zone when the current time indicated by the device indicates the night time zone. The panoramic video may be initially selected and displayed, or when the current date indicated by the device indicates the spring season, the panoramic video captured in spring may be initially selected and displayed.

[Specific processing example of this embodiment]
Next, details of processing performed in the information processing apparatus 3 will be described. In the description of the processing to be described later, an example (see FIGS. 4 to 6) of switching panoramic videos associated with the imaging location is used. First, main data used in the processing will be described with reference to FIG. FIG. 10 is a diagram illustrating an example of main data and programs stored in the memory 6 of the information processing apparatus 3.

  As shown in FIG. 10, panoramic image file data Da, terminal operation data Db, posture data Dc, virtual camera data Dd, virtual space image data De, and the like are stored in the data storage area of the memory 6. In addition to the data shown in FIG. 10, the memory 6 may store data necessary for processing, such as data used in an application to be executed. The program storage area of the memory 6 stores various program groups Pa constituting the information processing program.

  As described with reference to FIGS. 5 and 8, the panorama image file data Da is data indicating panorama image data and imaging information of each frame in each associated panoramic video. The panoramic image data includes data indicating a panoramic image to be pasted as a texture on the inner surface of the above-described three-dimensional model for each frame number N (1, 2, 3,...). For example, panoramic image data is obtained by decoding a panoramic video encoded and stored in a predetermined storage medium by the above method. The data indicating the imaging information of the panoramic image file data Da is data indicating the imaging location and imaging time of the panoramic image for each frame number N.

  The terminal operation data Db is data indicating the operation content for the terminal device 2, and includes operation input data Db1, angular velocity data Db2, and the like. The operation input data Db1 is data indicating the operation content for the touch panel 12 and the operation unit 13. The angular velocity data Db2 is data indicating the angular velocity generated in the terminal device 2, and is data indicating the angular velocity output from the gyro sensor 15.

  The attitude data Dc is data indicating the attitude of the terminal device 2 in real space, for example, data indicating the amount of rotation of the terminal apparatus 2 from the reference attitude.

  The virtual camera data Dd is data related to a virtual camera arranged at the center of the three-dimensional model. For example, the virtual camera data Dd is data relating to the position, posture, field of view (view angle), and the like of the virtual camera in the virtual space.

  The virtual space image data De is data indicating a virtual space image when the inner surface of the stereoscopic model is viewed from the virtual camera.

  Next, details of processing performed in the information processing apparatus 3 will be described with reference to FIGS. 11 and 12. FIG. 11 is a flowchart illustrating an example of the first half of processing executed in the information processing apparatus 3. FIG. 12 is a flowchart illustrating an example of the second half of the process executed in the information processing apparatus 3. Here, in the flowcharts shown in FIGS. 11 and 12, the process of switching the panoramic video according to the switching operation and displaying it on the terminal device 2 among the processes in the information processing apparatus 3 will be mainly described. Detailed descriptions of other processes not directly related are omitted.

  The CPU 5 initializes the memory 6 and the like, and reads the information processing program stored in the nonvolatile memory or the optical disk in the information processing apparatus 3 into the memory 6. Then, the CPU 5 starts executing the information processing program. The flowcharts shown in FIGS. 11 and 12 are flowcharts showing processing performed after the above processing is completed.

  Note that the processing of each step in the flowcharts shown in FIGS. 11 and 12 is merely an example, and if the same result is obtained, the processing order of each step may be changed. In addition and / or instead, another process may be performed. In the present embodiment, the processing of each step in the flowchart is described as being executed by the CPU 5, but the processing of a part or all of the steps in the flowchart is executed by a processor or a dedicated circuit other than the CPU. It may be.

  In FIG. 11, the CPU 5 acquires a panoramic image file (step 81). For example, the CPU 5 acquires a panorama image file from another device via a nonvolatile memory in the information processing device 3, a storage medium attached to the information processing device 3, or a network, and stores it in the panorama image file data Da. To do.

  Next, the CPU 5 arranges the three-dimensional model for pasting the panoramic video in the virtual space so that the center is located at the origin (step 82), and proceeds to the next step. For example, when the shape of the three-dimensional model is a cube, the CPU 5 intersects the XYZ axis set in the virtual space with the front of the three-dimensional model perpendicular to the Z-axis on the positive side of the Z-axis, and the back is the Z-axis It intersects perpendicularly with the Z axis on the negative direction side, the left side intersects perpendicularly with the X axis on the positive X side, the right side intersects perpendicularly with the X axis on the negative X side, and the top surface is on the positive Y axis side The solid model is arranged so that it intersects perpendicularly to the Y axis and the bottom surface intersects perpendicularly to the Y axis on the Y axis negative direction side.

  Next, the CPU 5 places the virtual camera at the reference position in the initial posture (step 83), and proceeds to the next step. For example, the CPU 5 sets the reference position of the virtual camera as the origin of the virtual space (that is, the center of the three-dimensional model), and the xyz axis of the virtual camera (the x-axis positive direction is the left direction of the virtual camera and the y-axis positive direction is the virtual camera). An orientation in which the upward direction and the z-axis positive direction are the direction of the line of sight of the virtual camera) coincides with the XYZ axes of the virtual space is defined as the initial orientation. Then, the CPU 5 updates data related to the position and orientation of the virtual camera in the virtual camera data Dd using the reference position and initial orientation of the virtual camera.

  Next, the CPU 5 prompts the user to adjust the attitude of the terminal device 2 (step 84), waits for the attitude adjustment to be performed (step 85), and repeats the steps 84 and 85 until the attitude adjustment is performed. . Then, when the posture adjustment is performed, the CPU 5 advances the process to step 86.

  In step 86, the CPU 5 sets the current attitude of the terminal device 2 to the reference attitude, and proceeds to the next step. For example, the CPU 5 initializes the attitude (the rotation amount from the reference attitude) of the terminal device 2 indicated by the attitude data Dc (the rotation amount around each axis is 0), and sets the reference attitude of the terminal device 2.

  Note that, as described above, in the processing of Step 84 to Step 86, the time when the processing of Step 84 is performed or the posture of the terminal device 2 after a predetermined time has elapsed from the time may be set as the reference posture. Then, the attitude of the terminal device 2 when the user performs a predetermined operation may be set as a reference attitude, or a predetermined fixed attitude of the terminal device 2 may be set as a reference attitude, or predetermined The user may select among the fixed postures of the terminal devices 2. As an example, when a panoramic video is reproduced and displayed on the monitor 4, the CPU 5 adjusts the orientation of the terminal device 2 so that the orientation of the LCD 11 of the terminal device 2 and the orientation of the display screen of the monitor 4 are the same. Then, a display prompting the user to perform a predetermined operation of the terminal device 2 when the posture is reached is displayed on the terminal device 2 and / or the monitor 4. When the CPU 5 acquires operation data indicating that a predetermined operation has been performed on the operation unit 13 of the terminal device 2, the CPU 5 determines that the posture adjustment has been performed, and the operation has been performed. The posture of the terminal device 2 at the time is set as the reference posture. In this case, the orientation of the terminal device 2 whose orientation has been adjusted so that the orientation of the LCD 11 of the terminal device 2 and the orientation of the display screen of the monitor 4 are the same by the processing of step 84 to step 86 is as follows. The reference posture is set.

  Next, the CPU 5 sets the number of frames n to 1 (step 87), and proceeds to the next step.

  Next, the CPU 5 sets a display target to be displayed on the terminal device 2 by default from among the panoramic videos associated in the panoramic image file data Da (step 88), and performs processing in the next step 91 (see FIG. 12). To proceed. For example, the CPU 5 may set a predetermined panoramic image as a display target, or may set a panoramic image selected by the user as a display target.

  In FIG. 12, the CPU 5 determines whether or not a switching operation has been performed (step 91). For example, the CPU 5 refers to the operation input data Da1 and determines whether or not a switching operation has been performed. In addition, from the terminal device 2, data indicating an operation input to the touch panel 12 or the operation unit 13 is transmitted to the information processing device 3 at a constant cycle and stored in the operation input data Db1. Then, when the switching operation is performed, the CPU 5 advances the process to step 92. On the other hand, if the switching operation is not performed, the CPU 5 advances the process to step 95.

  In step 92, the CPU 5 switches the panoramic video to be displayed to be displayed on the terminal device 2, and proceeds to the next step. For example, the CPU 5 newly selects another panoramic video associated with the panoramic video that is currently displayed as a display target.

  Next, the CPU 5 extracts the frame number X captured at the same imaging location as that of the currently displayed panoramic image using the imaging location data set in the panoramic video that is newly displayed. (Step 93), the process proceeds to the next step. For example, the CPU 5 refers to the panoramic image file data Da regarding the panoramic video that is newly displayed, and extracts the frame number X in which the same imaging location as that of the currently displayed panoramic image is set. .

  Next, the CPU 5 updates the number of frames n using the frame number X extracted in step 93 (step 94), and proceeds to step 95.

  In step 95, the CPU 5 obtains a panorama image corresponding to the frame n from the panorama image data of the panorama image file data Da related to the panoramic video that is currently displayed, and proceeds to the next step.

  Next, the CPU 5 pastes the panoramic image acquired in step 95 as a texture on the inner surface of the three-dimensional model (step 96), and proceeds to the next step. For example, as described above, the panoramic image is pasted as a texture on each inner surface of the stereoscopic model so that the reference direction set for the panoramic image matches the reference direction of the stereoscopic model.

  Next, the CPU 5 acquires data indicating the output value of the gyro sensor 15 of the terminal device 2 (step 97), and proceeds to the next step. Note that the data indicating the output value of the gyro sensor 15 is transmitted from the terminal device 2 to the information processing device 3 at a constant cycle and stored in the angular velocity data Db2.

  Next, the CPU 5 calculates the rotation direction and the rotation amount from the reference posture of the terminal device 2 (the rotation direction and the rotation amount after initialization in Step 86) using the data acquired in Step 97 (Step S). 98) The process proceeds to the next step. For example, in step 98, the rotation direction and the rotation amount about the predetermined axis direction of the terminal device 2 in the reference posture (for example, the xt axis, the yt axis, and the zt axis direction in the reference posture) are respectively calculated and the posture data is calculated. Dc is updated. Since the rotation direction can be expressed by the sign of the rotation amount, only the data indicating the rotation amount may be stored in the posture data Dc. For example, in step 98, the CPU 5 calculates a new rotation amount by adding the rotation amount based on the angular velocity data acquired in step 98 to the rotation amount calculated in step 98 in the previous process.

  Next, the CPU 5 rotates the posture of the virtual camera in the virtual space by the rotation amount calculated in Step 98 from the initial posture (Step 99), and proceeds to the next step. For example, the CPU 5 changes the orientation of the virtual camera from the initial orientation to the X-axis direction of the virtual space (the virtual camera C1 in the initial orientation) as much as the rotation amount about the xt-axis direction of the terminal device 2 calculated in step 98. , And the amount of rotation about the yt-axis direction of the terminal device 2 calculated in step 98 is the same as the Y-axis direction of the virtual space (y of the virtual camera C1 in the initial posture). (Axis direction) and the Z axis direction of the virtual space as much as the rotation amount about the zt axis direction of the terminal device 2 calculated in step 98 (the z axis direction of the virtual camera C1 in the initial posture) ) As an axis to update the data related to the attitude of the virtual camera in the virtual camera data Dd.

  Next, the CPU 5 generates an image (virtual space image) of the inner surface of the three-dimensional model viewed from the virtual camera (step 100), and proceeds to the next step. For example, the CPU 5 updates the virtual space image data De using data indicating the generated virtual space image.

  Next, the CPU 5 transmits a virtual space image based on the virtual space image data De to the terminal device 2 (step 101), and proceeds to the next step. For example, a virtual space image based on the virtual space image data De is received by the terminal device 2 and output to the LCD 11 to display the virtual space image. A predetermined compression process may be performed when the virtual space image is transmitted from the information processing device 3 to the terminal device 2. In this case, the data of the virtual space image that has been subjected to the compression processing is sent to the terminal device 2, and after the predetermined expansion processing is performed by the terminal device 2, the virtual space image is displayed.

  Next, the CPU 5 increments the frame number n (step 102), and determines whether or not to end the process (step 103). The conditions for ending the process include, for example, the end of the playback of the last frame image in the panoramic video being played back, and the user performing an operation to end the process. The CPU 5 returns to step 91 when the process is not ended, repeats the process, and ends the process according to the flowchart when the process is ended. Thereafter, the series of processing from step 91 to step 103 is repeatedly executed until it is determined in step 103 that the processing is to be ended.

[Modification]
The above-described embodiment is an example for carrying out the present invention. In other embodiments, the present invention can be implemented with, for example, the configuration described below.

  In the above processing example, an example in which a part of the panoramic image is displayed on the terminal device 2 is used. However, the information processing system 1 further displays the panoramic image on the monitor 4 and reproduces the panoramic video on substantially the same time axis. The terminal device 2 and the monitor 4 may be displayed. In this case, the panoramic video displayed on the monitor 4 may be switched to the same panoramic video in response to switching of the panoramic video displayed on the terminal device 2, or a predetermined panoramic video may be displayed fixedly. Good. Examples of the display range of the panoramic image displayed on the monitor 4 are as follows.

  As a first example, another virtual camera (second virtual camera) for displaying a panoramic image on the monitor 4 is provided, and the position and posture of the second virtual camera are fixed to the reference position and the initial posture. If an image of the inner surface of the stereoscopic model viewed from the second virtual camera is displayed on the monitor 4, a panoramic image in the reference direction (front direction) can always be displayed on the monitor 4. Note that the posture of the second virtual camera may be changed according to a user operation. In this case, the range of the panoramic image displayed on the monitor 4 changes according to the user operation.

  As a second example, the monitor 4 always displays an image showing the entire panoramic image (an omnidirectional image). For example, the omnidirectional image may be generated by synthesizing an image obtained from an imaging camera that captured a panoramic video, or may be generated by synthesizing a panoramic image that is pasted on each inner surface of the three-dimensional model. However, since the method for generating an omnidirectional image is already well known, detailed description thereof is omitted here.

  As a third example, the monitor 4 displays a display range corresponding to the display range of the panoramic image displayed on the terminal device 2. When the same panoramic video as the terminal device 2 is displayed, the panoramic image displayed on the terminal device 2 may be displayed on the monitor 4 in real time.

  In addition, on the terminal device 2 and / or the monitor 4, a panoramic video that is not a display target among the associated panoramic videos may be displayed as a thumbnail. In this case, a panoramic image having the same frame number as the frame number displayed in the panoramic video that is the display target may be synchronized to display the panoramic video that is not the display target as a thumbnail. The display range of the panoramic image displayed as a thumbnail may be set based on the same line-of-sight direction as that of the panoramic image displayed on the terminal device 2, or may be a predetermined display range, A display range based on a history displayed in the past on another display device may be used. As described above, when a panoramic video that is not a display target is displayed as a thumbnail, a user operation for selecting the thumbnail display (for example, an operation of touching the touch panel 12 at a position overlapping with the thumbnail display displayed on the LCD 11, a monitor) (The operation of selecting the thumbnail display displayed in 4 using a predetermined input device) may be the switching operation.

  Moreover, in the said embodiment, although the information processing system 1 was the structure which has only one terminal device 2, the information processing system 1 may be the structure which has the several terminal device 2. FIG. That is, the information processing device 3 is capable of wireless communication with each of the plurality of terminal devices 2, transmits image data to each terminal device 2, and receives data from the gyro sensor 15 from each terminal device 2. Also good. Then, the virtual camera of each terminal device 2 is arranged in the virtual space, the posture of each virtual camera is controlled according to the posture of each terminal device 2, and the image of the virtual space viewed from each virtual camera is displayed on each terminal. What is necessary is just to make it transmit to the apparatus 2. Note that the information processing device 3 performs wireless communication with each of the plurality of terminal devices 2, but at this time, the information processing device 3 may perform wireless communication with each terminal device 2 in a time-sharing manner or a frequency. The band may be divided.

  Thus, when displaying a panoramic video using a plurality of terminal devices 2, in step 83, the virtual cameras are arranged at the same reference position in the same initial posture. And in step 84-step 86, each attitude | position is adjusted so that the direction of each terminal device 2 may become the same, and when it becomes the said attitude | position, predetermined operation is performed using at least one terminal device 2. Thus, the attitude of each terminal device 2 is set to the reference attitude. Based on such a setting, the posture of each virtual camera is changed according to the rotation amount from the reference posture of each terminal device 2, and an image of the inner surface of the stereoscopic model viewed from each virtual camera is displayed on each terminal device 2. indicate. And according to the switching operation in each terminal device 2, the panoramic video displayed on the terminal device 2 is switched. When switching panoramic videos, panoramic videos may be switched only for the terminal device 2 for which the switching operation has been performed, or panoramic videos for all the terminal devices 2 that are in wireless communication with the information processing device 3 at the same time. You can switch between.

  Further, the terminal device 2 described above functions as a so-called thin client terminal that does not execute the series of processing described with reference to FIGS. 11 and 12 or the information processing performed by the information processing device 3. However, the terminal device 2 may be a device having a function of executing predetermined information processing (game processing) by a predetermined program (game program), such as a portable game machine. In this case, at least a part of the series of processes executed by the information processing device 3 in the above embodiment may be executed by the terminal device 2. As an example, when a panoramic video is displayed on each of a plurality of terminal devices using at least one terminal device that can execute all of the above-described series of processing, one of the executable terminal devices executes the series of processing. By transmitting a panoramic video according to the attitude of the other terminal device from the main processing execution device to the other terminal device, the same panoramic image is displayed and output on each terminal device. Can do.

  In the above-described embodiment, an example in which a panoramic video composed of panoramic images captured while moving in the real world is used. However, a panoramic video composed of panoramic images captured while moving in the virtual world is used. Alternatively, a panoramic video composed of an image obtained by combining a virtual world image with a panoramic image obtained by capturing the real world may be displayed.

  In another embodiment, in an information processing system having a plurality of information processing apparatuses that can communicate with each other, the plurality of information processing apparatuses may share and execute information processing. In addition, when information processing is executed in a plurality of information processing devices, it is necessary to synchronize the processing executed in each information processing device, and the processing becomes complicated. On the other hand, when information processing is executed by one information processing device 3 and the terminal device 2 receives and displays an image as in the above embodiment (that is, when the terminal device 2 is a thin client terminal). ), It is not necessary to synchronize the processing among a plurality of information processing apparatuses, and the processing can be simplified.

  Further, in the above embodiment, the information processing system 1 including the information processing apparatus 3 capable of executing information processing has been described as an example. However, each processing operation described in the above embodiment includes the game system illustrated in FIG. It can be executed not only by the game device but also by any information processing system and information processing device. The information processing system only needs to include an information processing device and at least one display device (for example, the terminal device 2), and the information processing device can output and display an image on the display device. If it is. Further, the above-described processing can be executed by cooperation between one processor or a plurality of processors included in an information processing system including at least one information processing apparatus.

  The information processing program may be supplied to the information processing apparatus 3 through an external storage medium, or may be supplied to the information processing apparatus 3 through a wired or wireless communication line. The program may be recorded in advance in a nonvolatile storage device inside the information processing apparatus 3. The information storage medium for storing the program may be a CD-ROM, a DVD, or an optical disk storage medium similar to them, a nonvolatile memory, a flexible disk, a hard disk, a magneto-optical disk, a magnetic tape, or the like. Further, the information storage medium for storing the program may be a volatile memory for storing the program. Such a storage medium can be said to be a storage medium readable by a computer or the like. For example, the various functions described above can be provided by causing a computer or the like to read and execute the programs of these storage media.

  Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit the scope thereof. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. It is understood that the scope of the present invention should be construed only by the claims. Moreover, it is understood that those skilled in the art can implement an equivalent range from the description of the specific embodiments of the present invention based on the description of the present invention and the common general technical knowledge. In addition, it is to be understood that the terms used in the present specification are used in the meaning normally used in the art unless otherwise specified. Thus, unless defined otherwise, all technical and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In case of conflict, the present specification, including definitions, will control.

  As described above, the present invention is useful as, for example, an information processing program, an information processing apparatus, an information processing system, an information processing method, and the like for the purpose of displaying a variety of panoramic videos.

DESCRIPTION OF SYMBOLS 1 ... Information processing system 2 ... Terminal apparatus 3 ... Information processing apparatus 4 ... Monitor 11 ... LCD
12 ... Touch panel 13 ... Operation unit 14 ... Speed sensor 15 ... Gyro sensor

Claims (16)

An information processing program executed by a computer included in an information processing device that displays an image on at least one display device,
The computer,
Panorama video acquisition means for reading out and sequentially acquiring the panoramic video as a display target from the panoramic video storage means for storing a plurality of panoramic videos associated with each of the contents obtained by imaging the real world in advance through predetermined parameters. When,
A display range setting unit that sets a line-of-sight direction for the acquired panoramic video and sets a display range to be displayed on the display device based on the line-of-sight direction viewed from a fixed viewpoint with respect to the panoramic video ;
Display control means for performing control to sequentially display the panoramic video within the display range on the display device among the acquired panoramic video;
When a predetermined switching condition is satisfied, it functions as a display object changing means for changing the panoramic video to be displayed,
When the panoramic video to be displayed is changed, the panoramic video acquisition means reads and sequentially acquires the panoramic video after the change,
The display range setting means sets the line-of-sight direction set for the panoramic video acquired immediately before the change for the panoramic video acquired immediately after the change without changing by the change ,
The information processing program, wherein the display range setting means changes the line-of-sight direction according to a user operation and fixes and sets a fixed viewpoint for the panoramic video without changing according to the user operation .   The information processing program according to claim 1, wherein the plurality of panoramic videos are contents captured at substantially the same place at different times.   The information processing program according to claim 2, wherein the plurality of panoramic videos are contents respectively captured from substantially the same viewpoint.   The information processing program according to claim 3, wherein the plurality of panoramic videos are contents captured while moving the viewpoint along the same course in the real world.   The information processing program according to any one of claims 1 to 4, wherein the plurality of panoramic videos are contents respectively captured in different time zones.   The information processing program according to any one of claims 1 to 4, wherein the plurality of panoramic videos are contents imaged at different times of at least one of year, month, and day.   When the panoramic video to be displayed is changed, the panoramic video acquisition means selects a frame in the panoramic video to be read after the change based on the predetermined parameter of the frame of the panoramic video read before the change. The information processing program according to any one of claims 1 to 3, wherein the information processing program is set and, after the change, reads and sequentially acquires panoramic videos after the frame. The panoramic video storage means stores a plurality of panoramic videos captured at different times while moving the viewpoint along the same course in the real world, in association with each imaging location as the predetermined parameter,
When the panoramic video to be displayed is changed, the panoramic video acquisition unit is configured to display a panoramic video captured at a substantially same imaging location as the imaging location where the frame of the panoramic video read before the change is captured. The information processing program according to claim 7, wherein a frame is set as a frame to be read after the change, and after the change, a panoramic video after the frame is read and sequentially acquired. The panoramic video storage means stores a plurality of panoramic videos captured at a time when at least one of a year, a month, and a day is different and stores each imaging time as the predetermined parameter,
When the panoramic video to be displayed is changed, the panoramic video acquisition unit is configured to display a panoramic video captured at substantially the same imaging time as the imaging time at which the frame of the panoramic video read before the change was captured. The information processing program according to claim 7, wherein a frame is set as a frame to be read after the change, and after the change, a panoramic video after the frame is read and sequentially acquired. The panoramic video storage means stores a plurality of panoramic videos captured at substantially the same place at different times in association with the frame number indicating the playback order of each frame as the parameter,
When the panoramic video to be displayed is changed, the panoramic video acquisition means sets a frame having the same frame number as the panoramic video frame number read before the change as a panoramic video frame to be read after the change. The information processing program according to claim 7, wherein after the change, a panoramic video after the frame is read and sequentially acquired. As the panoramic video placement means for placing the acquired panoramic video in a virtual space, the computer is further functioned,
The display range setting means sets a virtual camera for a panoramic video arranged in a virtual space, and sets the display range for the panoramic video according to a visual field range based on a viewing direction of the virtual camera. The information processing program according to any one of claims 1 to 10.   The information processing program according to any one of claims 1 to 11, wherein the display target changing unit determines that the switching condition is satisfied when a predetermined switching operation is performed. The panoramic video acquisition means reads from the panoramic video storage means a panoramic image that is each frame of a panoramic video that is a display target, and sequentially acquires the panoramic images that are displayed on the display device at predetermined time intervals. When the panoramic video to be changed is changed, the panoramic image in the changed panoramic video is read and sequentially acquired, and the display range setting unit sets the line-of-sight direction with respect to the acquired panoramic image to display the display When the range is set and the panoramic video to be displayed is changed, it is acquired immediately after the change without changing the line-of-sight direction set for the panoramic image acquired immediately before the change. The information processing program according to any one of claims 1 to 12, wherein the display range is set for a panoramic image. Grams. An information processing device that displays an image on at least one display device,
Panoramic video storage means for storing a plurality of panoramic videos related to each other in advance through predetermined parameters, each of which is obtained by imaging the real world in advance;
Panoramic video acquisition means for reading out the panoramic video to be displayed from the panoramic video storage means and sequentially acquiring the panoramic video;
A display range setting unit that sets a line-of-sight direction for the acquired panoramic video and sets a display range to be displayed on the display device based on the line-of-sight direction viewed from a fixed viewpoint with respect to the panoramic video ;
Display control means for performing control to sequentially display the panoramic video within the display range on the display device among the acquired panoramic video;
A display object changing means for changing the panoramic video to be displayed when a predetermined switching condition is satisfied,
When the panoramic video to be displayed is changed, the panoramic video acquisition means reads and sequentially acquires the panoramic video after the change,
The display range setting means sets the line-of-sight direction set for the panoramic video acquired immediately before the change for the panoramic video acquired immediately after the change without changing by the change ,
The information processing apparatus, wherein the display range setting unit changes the line-of-sight direction according to a user operation and fixes and sets a fixed viewpoint for the panoramic video without changing according to the user operation . An information processing system configured to be able to communicate with a plurality of devices and displaying an image on at least one display device,
Panoramic video storage means for storing a plurality of panoramic videos related to each other in advance through predetermined parameters, each of which is obtained by imaging the real world in advance;
Panoramic video acquisition means for reading out the panoramic video to be displayed from the panoramic video storage means and sequentially acquiring the panoramic video;
A display range setting unit that sets a line-of-sight direction for the acquired panoramic video and sets a display range to be displayed on the display device based on the line-of-sight direction viewed from a fixed viewpoint with respect to the panoramic video ;
Display control means for performing control to sequentially display the panoramic video within the display range on the display device among the acquired panoramic video;
A display object changing means for changing the panoramic video to be displayed when a predetermined switching condition is satisfied,
When the panoramic video to be displayed is changed, the panoramic video acquisition means reads and sequentially acquires the panoramic video after the change,
The display range setting means sets the line-of-sight direction set for the panoramic video acquired immediately before the change for the panoramic video acquired immediately after the change without changing by the change ,
The information processing system, wherein the display range setting unit changes the line-of-sight direction according to a user operation, and fixes and sets a fixed viewpoint for the panoramic video without changing according to the user operation . An information processing method executed by cooperation between one processor or a plurality of processors included in an information processing system configured by an information processing apparatus that displays an image on at least one display device,
A panoramic video acquisition step of reading out and sequentially acquiring the panoramic video as a display target from the panoramic video storage means that stores a plurality of panoramic videos associated with each of the contents obtained by imaging the real world in advance through predetermined parameters. When,
A display range setting step for setting a line-of-sight direction for the acquired panoramic video and setting a display range to be displayed on the display device based on the line-of-sight direction viewed from a fixed viewpoint with respect to the panoramic video ;
A display control step for performing control to sequentially display a panoramic video within the display range on the display device among the acquired panoramic videos;
A display object changing step of changing a panoramic video to be displayed when the predetermined switching condition is satisfied,
In the panoramic video acquisition step, when the panoramic video to be displayed is changed, the panoramic video after the change is read and sequentially acquired.
In the display range setting step, the line-of-sight direction set for the panoramic video acquired immediately before the change is set for the panoramic video acquired immediately after the change without being changed by the change ,
In the display range setting step, the line-of-sight direction is set in accordance with a user operation, and a fixed viewpoint for the panoramic video is fixedly set without being changed in accordance with the user operation. Method.