JP5845819B2 - Movie presentation device - Google Patents

Description

  The present invention relates to a moving image presentation device, and more particularly to a device that presents a field of view viewed from a viewpoint moving along a predetermined route as a moving image.

  When a camera equipped with a fisheye lens or an omnidirectional mirror is used, an omnidirectional image having a 360 ° field of view can be taken. If such an omnidirectional image is prepared in advance, it is possible to provide a service for presenting an image obtained by observing an arbitrary direction from an arbitrary viewpoint position desired by the viewer.

  For example, in Patent Document 1 below, an omnidirectional image is prepared in the information providing center, and an image obtained by observing an arbitrary direction from an arbitrary viewpoint is displayed on the screen of the terminal device in response to a viewer's request. A technique for transmitting the image data is disclosed. Google Inc. (Headquarters: Google Inc. in California, USA) is called “Street View” and provides a service that displays the scenery from any point of view on the road using the Internet. ing. On the other hand, Patent Document 2 discloses an apparatus that removes obstacles such as pedestrians, telephone poles, and roadside trees from a video image of an urban area taken by an in-vehicle camera, and presents an urban area image that does not include foreground objects such as moving objects. Has been.

JP 2001-008232 A JP 2011-118495 A

  By using the system disclosed in the above-mentioned Patent Document 1 and the “Street View” service provided by Google, the user designates an arbitrary viewpoint position and an arbitrary gaze direction. The presentation of the image according to can be received. However, in these conventional techniques, when a video is presented as a moving image, a moving object such as a person or a vehicle is handled without distinction from a background image, and thus a field of view as viewed from a viewpoint moving along a predetermined route is presented as a moving image. In some cases, there is a problem that the motion of the moving object becomes unnatural.

  In the system disclosed in Patent Document 1, a visual field image corresponding to a user-specified viewpoint position and line-of-sight direction is taken out each time, and if these are displayed in order, video presentation according to the user's specification is performed. It is possible. However, in such a presentation method, if a moving object such as a person or a car is included in the video, the movement of the moving object may appear unnatural. For example, if you prepare an omnidirectional image taken while moving along a certain route, you can turn the direction of the line of sight to any direction of 360 °, and you can move forward in either direction Can be displayed. However, if the presentation order of the image frames does not follow the shooting order, the moving object is reproduced with its movement reversed.

  On the other hand, if the technique disclosed in the above-mentioned Patent Document 2 is used, moving objects such as pedestrians and cars can be deleted from the captured video and only the background image can be presented. It can avoid becoming unnatural. However, the video including only the background image is very unsavory, and it is impossible to give the viewer a sense of reality that they are actually in the video screen.

  SUMMARY OF THE INVENTION An object of the present invention is to provide a moving picture presentation apparatus capable of presenting a moving picture so that the movement of the moving object can be seen naturally while giving the viewer a high sense of reality by including the moving object as a subject.

(1) A first aspect of the present invention is a video presentation device that presents a field of view viewed from a moving viewpoint as a video,
A video storage unit that stores a video shot while moving along a path connecting the first node and the second node;
An instruction input unit for inputting user instructions;
Based on the user's instruction, either the forward direction from the first node to the second node or the return direction from the second node to the first node is set as the movement direction on the route. A moving direction setting section to be
A video playback unit that plays back the video stored in the video storage unit and displays it on the screen of the display device;
Provided,
The moving image storage unit stores a moving image including a moving object on the subject taken while moving in the forward direction on the route, and a returning moving image including a moving object on the subject taken while moving on the route in the returning direction. And
When the forward direction is set in the moving direction setting unit, the video playback unit performs forward playback on the forward direction video as a playback target, and when the backward direction is set in the moving direction setting unit, The progressive playback is performed as a playback target.

(2) According to a second aspect of the present invention, in the video presentation device according to the first aspect described above,
The instruction input unit has a function of inputting a reversal instruction for reversing the moving direction,
The movement direction setting unit performs a process of switching the movement direction from the forward direction to the backward direction or from the backward direction to the forward direction based on the input of the reverse instruction.
The video playback unit sets a predetermined period after the inversion instruction is input as the transition period, and during this transition period, the pre-switching image constituting the pre-switching video that is the playback target according to the moving direction before switching, and the after switching A composite image obtained by blending a post-switching image that is a playback target corresponding to the moving direction of the image and that is captured at the same position as the pre-switching image is displayed.

(3) According to a third aspect of the present invention, in the video presentation device according to the second aspect described above,
The moving image reproduction unit creates a composite image by combining both images with the ratio of the image after switching being α and the ratio of the image before switching being (1−α), and the value α is changed from 0 to 1 as the transition period elapses. It is intended to gradually increase toward

(4) According to a fourth aspect of the present invention, in the video presentation device according to the third aspect described above,
A total of N frame images from the first frame shot at the first node position to the Nth frame shot at the second node position of the forward moving image stored in the moving image storage unit Consists of
A total of N frame images from the first frame captured at the second node position to the Nth frame captured at the first node position for the return path video stored in the video storage unit. Consists of
The i-th frame F (i) of the forward moving image and the (N−i + 1) th frame G (N−i + 1) of the backward moving image are images taken at the same point on the route, and Configure the frame corresponding to
The moving image reproduction unit creates a composite image by combining frames corresponding to each other during the transition period, and displays the composite image.

(5) According to a fifth aspect of the present invention, in the video presentation device according to the fourth aspect described above,
The video playback section
During the transition period, a fixed position image consisting of a composite frame obtained by combining the frame immediately before the transition period of the pre-switching video and the corresponding frame of the post-switching video corresponding thereto is displayed.
After the transition period, forward playback is started from the corresponding frame of the moving image after switching.

(6) According to a sixth aspect of the present invention, in the video presentation device according to the fourth aspect described above,
The video playback section
In the first half of the transition period, the pre-switching video is continuously played forward, and in the latter half of the transition period, the pre-switching video is played backward from the last frame of the first half,
During the transition period, display a video image composed of synthesized frames obtained by synthesizing each frame of the video before switching and each frame of the video after switching corresponding to this,
After the transition period ends, the post-switching video is continuously played forward.

(7) According to a seventh aspect of the present invention, in the video presentation device according to the sixth aspect described above,
The video playback unit gradually reduces the playback speed from the speed immediately before the transition period in the first half of the transition period, and gradually increases the playback speed in the second half of the transition period. .

(8) According to an eighth aspect of the present invention, in the video presentation device according to the fourth aspect described above,
The video playback section
During the transition period, the pre-switching video is played back in reverse from the frame immediately before the transition period, and a video image composed of the combined frames of the pre-switching video frames and the corresponding post-switching video frames is displayed. And
After the transition period ends, the post-switching video is continuously played forward.

(9) According to a ninth aspect of the present invention, in the video presentation device according to the first aspect described above,
The instruction input unit has a function of inputting a reversal instruction for reversing the moving direction,
The movement direction setting unit performs a process of switching the movement direction based on the input of the reverse instruction,
In the moving image storage unit, in addition to the forward moving image and the returning moving image, a background moving image that does not include moving objects in the subject and is photographed while moving in the forward direction or the returning direction on the route is further stored.
The video playback unit sets a predetermined period after the inversion instruction is input as the transition period, and during this transition period, the pre-switching image that constitutes the pre-switching video that is the playback target according to the moving direction before switching, and the background video The first synthesized image blended with the background image captured at the same position as the pre-switching image, the background image constituting the background video, and the post-switching that is the playback target according to the moving direction after switching A second composite image obtained by blending a post-switching image that is composed of a moving image and shot at the same position as the background image is displayed in the order of the first composite image and the second composite image. .

(10) According to a tenth aspect of the present invention, in the video presentation device according to the ninth aspect described above,
The video playback section
Within the transition period, set a first transition period and a second transition period following this,
The ratio of the background image is α, the ratio of the pre-switching image is (1−α), a first composite image is created by combining both images, and this is displayed within the first transition period, and The value α is gradually increased from 0 to 1 with the passage of the first transition period,
The ratio of the image after switching is β, the ratio of the background image is (1-β), and a second composite image is created by combining both images, and this is displayed within the second transition period, and As the second transition period elapses, the value β is gradually increased from 0 to 1.

(11) According to an eleventh aspect of the present invention, in the video presentation device according to the tenth aspect described above,
A total of N frame images from the first frame shot at the first node position to the Nth frame shot at the second node position of the forward moving image stored in the moving image storage unit Consists of
A total of N frame images from the first frame captured at the second node position to the Nth frame captured at the first node position for the return path video stored in the video storage unit. Consists of
A total of N frame images from the first frame shot at the first node position to the Nth frame shot at the second node position of the background moving image stored in the moving image storage unit Consists of
The i-th frame F (i) of the forward moving image , the (N−i + 1) th frame G (N−i + 1) of the backward moving image, and the i-th frame H (i) of the background image It is an image taken at the same point on the top, constituting a mutually corresponding frame,
The moving image reproduction unit creates a composite image by combining frames corresponding to each other during the transition period, and displays the composite image.

(12) According to a twelfth aspect of the present invention, in the video presentation device according to the eleventh aspect described above,
The video playback section
During the first transition period, a fixed position image composed of a composite frame obtained by combining the frame immediately before the transition period of the pre-switching video and the corresponding background frame of the background video corresponding thereto is displayed as the first composite image,
During the second transition period, a fixed position image composed of a combined frame obtained by combining the corresponding background frame of the background moving image and the corresponding frame corresponding to the corresponding background frame of the switched moving image is displayed as the second combined image. ,
After the transition period, forward playback is started from the corresponding frame of the moving image after switching.

(13) According to a thirteenth aspect of the present invention, in the video presentation device according to the twelfth aspect described above,
The moving image reproduction unit provides an intermediate transition period between the first transition period and the second transition period, and displays only the corresponding background frame during the intermediate transition period.

(14) In a fourteenth aspect of the present invention, in the video presentation device according to the eleventh aspect described above,
The video playback section
During the first transition period, the pre-switching video is continuously played forward and a video image composed of a composite frame obtained by synthesizing each frame of the pre-switching video and the corresponding background frame of the background video is used as the first composite image. Display
During the second transition period, the post-switching video is sequentially played back from the frame corresponding to the final corresponding background frame of the first transition period, and each frame of the post-switching video and the corresponding background frame of the background video are synthesized. Display a moving image composed of composite frames as a second composite image,
After the second transition period, the moving image after switching is continuously played back in sequence.

(15) According to a fifteenth aspect of the present invention, in the video presentation device according to the fourteenth aspect described above,
The video playback unit gradually decreases the playback speed from the speed immediately before the transition period during the first transition period, and gradually increases the playback speed during the second transition period. .

(16) According to a sixteenth aspect of the present invention, in the video presentation device according to the eleventh aspect described above,
The video playback section
During the first transition period, the moving image before switching is played back in reverse from the frame immediately before the transition period, and the moving image is composed of synthesized frames obtained by combining each frame of the moving image before switching and each frame of the background moving image corresponding thereto. As the first composite image,
During the second transition period, the post-switching video is sequentially played back from the frame corresponding to the final corresponding background frame of the first transition period, and each frame of the post-switching video and the corresponding background frame of the background video are synthesized. Display a moving image composed of composite frames as a second composite image,
After the transition period ends, the post-switching video is continuously played forward.

(17) According to a seventeenth aspect of the present invention, in the video presentation device according to the first to sixteenth aspects described above,
Each moving image stored in the moving image storage unit is composed of an image that is recorded in units of frames, which is an omnidirectional image having a 360 ° field of view, which is taken using an omnidirectional camera while moving along a route. ,
The instruction input unit has a function of inputting an instruction indicating a specific gaze direction,
The moving image reproducing unit cuts out a field-of-view image corresponding to the line-of-sight direction from each frame constituting the moving image and displays it on the display screen.

(18) According to an eighteenth aspect of the present invention, in the video presentation device according to the seventeenth aspect described above,
The instruction input unit includes a controller having a forward button, a backward button, a left button, and a right button,
The movement direction setting unit sets the movement direction to the first direction based on the forward button operation instruction, sets the movement direction to the second direction based on the backward button operation instruction,
The moving image reproduction unit changes the line-of-sight direction to the left based on an operation instruction for the left button, and changes the line-of-sight direction to the right based on an operation instruction for the right button.

(19) According to a nineteenth aspect of the present invention, in the video presentation device according to the eighteenth aspect described above,
The moving direction setting unit switches the moving direction when the angle of the line-of-sight direction with respect to the moving direction is equal to or greater than a predetermined reference based on the operation instruction of the left button or the right button. is there.

(20) According to a twentieth aspect of the present invention, in the video presentation device according to the seventeenth to nineteenth aspects described above,
From the distorted circular image obtained by photographing the hemispherical field located above the predetermined horizontal plane using the omnidirectional camera equipped with the fisheye lens or the omnidirectional mirror, the moving image storage unit has an elevation angle of a predetermined reference value or less. A moving image having a rectangular panoramic image obtained by cutting out an area and correcting the distortion is stored as an omnidirectional image.

(21) According to a twenty-first aspect of the present invention, in the video presentation device according to the seventeenth to nineteenth aspects described above,
The omnidirectional image storage unit stores a moving image in which a rectangular panoramic image created based on a three-dimensional CG image indicating a virtual route is an omnidirectional image.

(22) According to a twenty-second aspect of the present invention, in the video presentation device according to the first to twenty-first aspects described above,
The video storage unit stores videos for a plurality of paths connecting three or more nodes, respectively.
A route information storage unit for storing route information indicating a connection relation of each route through each node;
The instruction input unit has a function of inputting a user's route transfer instruction to move from the first route to the second route via the node;
The video playback unit has a function of performing video playback for the second route continuously with video playback for the first route by referring to the route information when a route transfer instruction is input. It is a thing.

  (23) In a twenty-third aspect of the present invention, the video presentation device according to the first to twenty-second aspects described above is configured by incorporating a dedicated program into a computer.

  According to the moving image presentation apparatus of the present invention, two types of moving images, that is, the forward moving image and the returning moving image, are shot on the same route, and either one of the moving images is reproduced according to the moving direction desired by the user. Therefore, it is possible to present a moving image so that the motion of the moving object can be seen naturally while presenting an image including the moving object to the viewer to give a high sense of realism.

  In addition, if the transition period is set when switching between the presentation of the outbound video and the presentation of the return path video, an embodiment in which a composite image in which the pre-switching video and the post-switching video are blended during this transition period is presented is adopted. Therefore, the moving object can be switched smoothly, and the unnaturalness of the movement of the moving object at the time of switching can be reduced.

  Furthermore, in the embodiment in which a background video that does not include a moving object is prepared as a subject, it is possible to perform blending in which a background video is inserted between the pre-switching video and the post-switching video. It is possible to smoothen and further reduce the unnatural movement of the moving body at the time of switching.

It is a top view of the art museum used as the presentation subject by the animation presentation device concerning the present invention (hatching shows a wall surface). It is a top view which shows each node and each path | route which comprise the visit path of the museum shown in FIG. 4 is a moving image frame chart showing each frame constituting a moving image obtained by photographing along one route R. It is a side view which shows an example of the omnidirectional camera utilized in order to image | photograph the moving image shown in FIG. It is a top view of the distortion circular image image | photographed using the omnidirectional camera shown in FIG. FIG. 6 is a plan view showing an example of a rectangular panoramic image (omnidirectional image) created based on the distorted circular image shown in FIG. 5. 6 is a plan view showing a definition example of a viewing direction (azimuth angle) φ (i) at a point P (i) on a route R. FIG. It is a top view which shows the operation | work which cuts out the visual field image Q (P (i), (phi) (i)) corresponding to the visual line direction of azimuth | direction angle (phi) = 90 degrees from the panoramic image (omnidirectional image) shown in FIG. It is a top view of the controller which inputs a user's instruction | indication. It is a block diagram which shows the structure of the moving image presentation apparatus which concerns on fundamental embodiment of this invention. FIG. 11 is a moving picture frame chart showing a relationship between a forward moving picture Ma and a return moving picture Mb stored in the moving picture storage unit 110 shown in FIG. 10. It is a figure which shows the reproduction | regeneration screen of the outward moving image Ma shown in FIG. It is a figure which shows the reproduction | regeneration screen of the inbound trip moving image Mb shown in FIG. 11 is a time chart showing a basic switching mode of a moving image by the moving image reproducing unit 130 when a reverse instruction is input in the moving image presentation device shown in FIG. 10. 11 is a time chart showing a switching mode through a moving image transition period by the moving image reproduction unit 130 when a reverse instruction is input in the moving image presentation device shown in FIG. 10. It is a figure which shows the principle of the image blend process in the transition period T shown in FIG. 11 is a time chart illustrating another switching mode through a moving image transition period by the moving image reproducing unit 130 when a reverse instruction is input in the moving image presentation apparatus illustrated in FIG. 10. FIG. 11 is a time chart showing still another switching mode through a moving image transition period by the moving image reproducing unit 130 when a reverse instruction is input in the moving image presentation apparatus shown in FIG. 10. It is a block diagram which shows the structure of the moving image presentation apparatus which concerns on embodiment using the background moving image of this invention. 20 is a moving image frame chart showing the relationship between the forward moving image Ma, the return moving image Mb, and the background moving image Mc stored in the moving image storage unit 110 ′ illustrated in FIG. It is a figure which shows the reproduction | regeneration screen of the background moving image Mc shown in FIG. FIG. 20 is a time chart showing a first switching mode of a moving image by a moving image reproducing unit 130 ′ when a reverse instruction is input in the moving image presentation device shown in FIG. 19. 24 is an equation showing the principle of image blending processing in the first transition period T1 and the second transition period T2 shown in FIG. FIG. 20 is a time chart showing a second moving image switching mode by the moving image reproducing unit 130 ′ when a reverse instruction is input in the moving image presenting apparatus shown in FIG. 19. FIG. FIG. 20 is a time chart showing a third moving image switching mode by the moving image playback unit 130 ′ when a reverse instruction is input in the moving image presentation device shown in FIG. 19; FIG. FIG. 20 is a time chart showing a fourth switching mode of moving images by the moving image reproducing unit 130 ′ when a reverse instruction is input in the moving image presentation apparatus shown in FIG.

  Hereinafter, the present invention will be described based on the illustrated embodiments.

<<< §1. Basic method of video presentation >>>
Here, first, a basic method of video presentation employed by the video presentation device according to the present invention will be described. The video presentation device according to the present invention is a device that presents a field of view viewed from a viewpoint moving along a predetermined route as a video, and the route to be displayed may be outdoors or indoors. Absent. Therefore, this device, for example, presents various historical sites and scenery of sightseeing spots on the display screen, and allows the user (viewer) to feel as if they are walking freely around the historical sites and sightseeing spots. It is possible to make it. In the following description, for the sake of convenience, a simple example will be described in which a moving image is presented to the user in a state where the user is walking in a museum with a floor plan as shown in FIG.

  Here, the museum shown in Fig. 1 is a real facility for exhibiting paintings and sculptures. As shown in the figure, the visitors who entered the museum can freely walk around the corridor and enjoy the exhibits. Suppose you leave the exit. The hatched portion in FIG. 1 shows the wall surface of the corridor, and the picture is presented on this wall surface. In addition, sculptures are appropriately placed on the floor of the corridor. The video presentation device according to the present invention is basically a device for presenting the user with an image of the inside of this real art museum, but the user can walk around the hall freely It has a function of simulating a state observed in an arbitrary direction and presenting it as a moving image.

  In this case, such a museum visit path is expressed by a plurality of nodes and paths connecting these nodes, and the entire visit path is expressed by route information having information on each node and each path. To do. FIG. 2 is a plan view showing the nodes and paths constituting the visit path of the museum shown in FIG. 1. Each black dot indicates a node, each straight line indicates a path, and each broken line indicates an outline of the actual visit path. Yes. As illustrated, in this example, six nodes N1 to N6 and six routes R1 to R6 are defined. Each path is defined as a straight line connecting these end points with a pair of nodes as end points.

  It is left up to the device designer to define the nodes on the visit path and the paths between the nodes, but in general, the branch point, turn corner, start point, and end point of the visit path Nodes can be defined, and routes can be defined between necessary nodes according to the actual visiting route. In the case of the illustrated example, each of the paths R1 to R6 is a straight line. However, when the actual path includes a curve, a path composed of a curve may be defined using a Bezier curve or the like. However, if each path is a straight line, the path is also specified only by specifying the nodes at both end points. Therefore, in practice, it is preferable that all the paths are constituted by straight lines. When the actual visiting route includes a curve, the curve may be approximated by a broken line formed by a plurality of nodes and a straight line connecting them.

  The route information indicating the entire visiting route includes information indicating the position of each node (for example, coordinate values on the XY coordinate system) and information indicating a route between a pair of nodes (for example, “nodes N1, N2: route R1”). And the like).

  In order to present a moving image for such a visit route, it is only necessary to prepare moving images taken while moving along the individual routes R1 to R6. FIG. 3 is a moving image frame chart showing a frame group constituting a moving image obtained by photographing along one route R. The illustrated example shows a frame structure of a moving image obtained by shooting while moving from the node A to the node B along the route R connecting the pair of nodes A and B.

  As shown in the figure, this moving image is obtained by discretely defining points P (1), P (2),..., P (i),. The frame images F (1), F (2),..., F (i),. The Here, the point P (1) coincides with the node A, the point P (N) coincides with the node B, and unique frame images are taken for the total N points. Each frame image is a still image, but a moving image can be presented by reproducing the frame image in order. The interval between the points is determined by the number of frames per second of the photographing apparatus and the moving speed of the photographing apparatus. For example, if shooting is performed while moving a 30-frame / second imaging device at a speed of 150 mm / second, the interval between points is 5 mm.

  In the case of the example shown here, a photographing device as shown in FIG. 4 is used as a device for photographing a moving image. As shown in the figure, this photographing apparatus is constituted by a fisheye lens 10, a video camera 20, a data processing unit 30, and a carriage 40, and functions as an omnidirectional camera capable of photographing an image having a 360 ° field of view. As shown in FIG. 4, the fisheye lens 10 is disposed on the uppermost part of the apparatus, and forms an image of a hemispherical field located above a predetermined horizontal plane on the imaging surface of the video camera 20. If such a configuration is adopted, a distorted circular image as shown in FIG. 5 is formed on the imaging surface of the video camera 20 (both the hatched donut-shaped portion and the white circular portion inside thereof are formed). And a distorted circular image taken with the fisheye lens 10). The celestial vertex O at the center of the distorted circular image is a point (corresponding to the ceiling of the corridor) at a position vertically above the imaging device shown in FIG. 4, and the outer periphery of the distorted circular image is shown in FIG. This corresponds to the lower end position of the field of view taken by the apparatus (field of view around 360 °). ).

  In the case of the example shown here, the museum's hall displaying paintings and sculptures is to be displayed, so the image of the white circular part near the top of the top O (corresponding to the image of the ceiling part of the corridor) is used. Instead, only the donut-shaped area hatched in the figure is used. FIG. 6 shows a rectangular panoramic image obtained by cutting out a region having an elevation angle equal to or smaller than a predetermined reference value (a hatched region in the drawing) from the distorted circular image shown in FIG. 5 and applying distortion correction thereto. FIG. If this panoramic image is an image obtained based on a distorted circular image photographed at the point P (i), the image constitutes the i-th frame F (i) of the moving image. . The azimuth angle of 0 ° to 360 ° shown on the lower side of the panoramic image corresponds to the azimuth angle of 0 ° to 360 ° shown on the circumference of the distorted circular image in FIG. That is, the panoramic image shown in FIG. 6 is an image in which the viewpoint is viewed at the position of the fisheye lens 10 shown in FIG.

  The data of the distorted circular image taken by the video camera 20 is sent to the data processing unit 30. The data processing unit 30 is configured by a computer, and distorted circular image data is stored in the hard disk device. In the example shown here, the data processing unit 30 has a function of performing a process of creating a panoramic image from the distorted circular image. Therefore, the distorted circular image as shown in FIG. 5 is converted into a panoramic image as shown in FIG. 6 in the data processing unit 30 and stored in the hard disk device. In the case of the example shown here, the panoramic image obtained in this way is used as an omnidirectional image constituting each frame of a moving image.

  Since the distorted circular image shown in FIG. 5 is a distorted donut-shaped image, the panoramic image shown in FIG. 6 is a rectangular image. Therefore, the data processing unit 30 performs image conversion processing with distortion correction. There is a need to do. Since such image conversion processing is a known technique, detailed description thereof is omitted here. Normally, even if an image conversion process with such distortion correction is performed, it is difficult to completely remove the distortion, so that a slight distortion remains in the obtained panorama image. Distortion correction to a level that does not hinder it is possible.

  The procedure for obtaining the panoramic image around the shooting device shown in FIG. 4 at one point on the route R has been described above. The moving image is moved by the video camera 20 while moving the shooting device along the route R. When shooting is performed, a panoramic image (omnidirectional image) as shown in FIG. 6 is used as one frame, and a moving image including a plurality of continuous frames is obtained. The moving image composed of N frames F (1) to F (N) shown in FIG. 3 is obtained by such shooting.

  However, when presenting a moving image to the user, it is not preferable to display the panoramic image shown in FIG. 6 as it is as one frame image. This panoramic image is a peculiar image including all information of the surrounding 360 °, and is far from an image entering a general human visual field. Therefore, in practice, it is necessary to determine a specific line-of-sight direction, cut out only a part of the visual field in the line-of-sight direction, and present it to the user.

  FIG. 7 is a plan view illustrating a definition example of the line-of-sight direction at the specific point P (i) on the route R. The line-of-sight direction is defined as the direction of the line-of-sight vector E (i) extending from the point P (i). In the case of the example shown here, the direction from the point P (i) upward in the figure is 0 °, and the azimuth angle φ (i) from 0 ° to 360 ° is defined clockwise. In other words, the angle φ (i) (where 0 ° ≦ φ (i) <360 °) formed by the line-of-sight vector E (i) is defined as the azimuth angle with reference to the axis facing the 0 ° direction. It will be.

  The azimuth angle φ (i) is a parameter indicating in which direction the virtual user located at the i-th point P (i) on the route R is looking. In the case of the illustrated example, the path R is a straight line extending in the left-right direction (horizontal direction) in the figure, and therefore the azimuth angle φ (i) = 90 ° indicates the direction toward the node B, and the azimuth angle φ (i) = 270 ° indicates a direction toward the node A.

  Thus, if the point P (i) indicating the current position of the virtual user and the azimuth angle φ (i) indicating the line-of-sight direction are determined, the visual field image Q (P (i), φ (i) viewed from the virtual user is determined. )) Can be extracted. FIG. 8 is a plan view showing an operation of cutting out such a field image Q (P (i), φ (i)). A visual field image Q (P (i), φ (i)) surrounded by a thick line in the figure indicates that the virtual user located at the point P (i) on the route R shown in FIG. ) = 90 ° is an image that should appear in the field of view of the virtual user when it is oriented in the direction indicated by 90 ° (the direction toward Node B).

  In order to obtain such an image, a field of view within the range of azimuth angles “φ (i) −Δ / 2” to “φ (i) + Δ / 2” is configured from the i-th frame image F (i). What is necessary is just to perform the process which cuts out the part to perform. Here, Δ is a predetermined cut-out angle, which is an angle corresponding to the lateral width of the cut out view field image Q (P (i), φ (i)). In the case of the illustrated example, since the setting is such that the azimuth angle φ = 90 °, the visual field image corresponding to the horizontal width Δ is cut out with the position of φ = 90 ° as the center.

  When a virtual user who is moving from the point P (i) shown in FIG. 7 toward the node B is looking in the moving direction (the direction toward the node B), to present a moving image that enters the field of view. , Cut out the field images shown at the azimuth angle φ = 90 ° from the frame images F (i), F (i + 1), F (i + 2),..., And display these field images in order (according to the shooting order). What is necessary is just to perform the process. Then, in the scenery shown in the thick frame in FIG. 8, a moving image in which the sculpture image placed near the azimuth angle of 90 ° gradually approaches is presented.

  Of course, the virtual user can change the line-of-sight direction (azimuth angle φ) during movement. When a change occurs in the line-of-sight direction (azimuth angle φ), the cut-out position indicated by a thick frame in FIG. 8 moves to the left and right. The virtual user can also make the line-of-sight direction reverse to the movement direction. For example, when a virtual user moving from the point P (i) shown in FIG. 7 toward the node B performs the setting of the azimuth angle φ = 270 °, the moving direction is the direction toward the node B. The direction is a direction toward the node A, and a field-of-view image corresponding to the horizontal width Δ is cut out from the frame image as shown in FIG. 8 with the position at φ = 270 ° as the center. As a result, a moving image in which the sculpture image placed near the azimuth angle of 270 ° gradually moves away is presented.

  On the other hand, the user can set the moving direction to an arbitrary direction. For example, for the virtual user located at the point P (i) shown in FIG. 7, when the moving direction is set to the direction toward the node A and the line-of-sight direction is set to the azimuth angle φ = 270 °, FIG. A moving image in which a sculpture image placed at a position near the azimuth angle of 270 ° gradually approaches is presented. In order to present such a moving image, a visual field image indicated by an azimuth angle φ = 270 ° is cut out from each frame image F (i), F (i-1), F (i-2),. A process for displaying these visual field images in order (in the order opposite to the shooting order) may be performed.

  As described above, the basic method for presenting a moving image when a virtual user moves on one route R (route between nodes A and B) shown in FIG. 7 has been described. A similar video presentation process may be performed for the route. For example, the visit path of the museum shown in FIG. 2 is composed of six routes R1 to R6. If each of the routes R1 to R6 is filmed in the same manner as in the above example, Even if the virtual user advances in which direction on the viewing path in which direction, and in which direction the line of sight is directed, appropriate video presentation can be performed.

  As described above, the controller 50 as shown in FIG. 9 is prepared in order to efficiently perform the instruction input for moving the virtual user in the desired direction on the viewing path and directing the line of sight in the desired direction. Is preferred. The controller 50 is a general device used as an input device for a computer game, and is provided with five buttons B1 to B5 on the right side, three buttons B6 to B8 on the center, and one button B9 on the left side. ing. Here, the illustrated button B1 functions as a forward button, the button B2 functions as a backward button, the button B3 functions as a left button, the button B4 functions as a right button, and the button B5 functions as a stop button.

  Here, the forward button B1 functions as a button for giving an instruction to start an operation of moving the virtual user in the first direction on the route, and the backward button B2 moves the virtual user in the second direction on the route. The stop button B5 functions as a button for giving an instruction to stop the virtual user.

  For example, when the virtual user is in a stopped state at the point P (i) on the route R shown in FIG. 7, when the forward button B1 is pressed, the virtual user is directed toward the node B at a predetermined speed. The moving video starts to play. That is, the field-of-view image indicated by a predetermined azimuth angle φ is cut out from each frame image F (i), F (i + 1), F (i + 2),... ) Is displayed. When the stop button B5 is pressed, the reproduction of the moving image is stopped, and the frame image displayed at that time is displayed as a still image.

  On the other hand, when the virtual user is stopped at the position of the point P (i) on the route R shown in FIG. 7, when the backward button B2 is pressed, the virtual user is directed toward the node A at a predetermined speed. The moving video starts to play. That is, the field-of-view image indicated by a predetermined azimuth angle φ is cut out from each frame image F (i), F (i-1), F (i-2),. They are displayed in the reverse order of the shooting order. When the stop button B5 is pressed, the reproduction of the moving image is stopped, and the frame image displayed at that time is displayed as a still image.

  In addition, when the backward button B2 is pressed in a state in which a moving image that is moving toward the node B at a predetermined speed is being played by pressing the forward button B1, the virtual user's The moving direction is reversed in the direction toward the node A. Similarly, when the forward button B1 is pressed in a state in which a moving image that is moving toward the node A at a predetermined speed is being played by pressing the backward button B2, the virtual user at that time is pressed. Will be reversed in the direction toward Node B.

  Here, for convenience of explanation, they are referred to as the forward button B1 and the backward button B2. However, since the virtual user's line-of-sight direction can be directed to any direction of 360 °, “forward” or “reverse” referred to here. Does not indicate the front and back in the general sense of “going in the direction of the gaze” or “going in the direction opposite to the direction of the gaze”, but simply “the first direction (in this example, It means "going in the direction from node A to B)" or "going in the second direction (in this example, going from node B to A)".

  In order to change the viewing direction of the virtual user, an operation of pressing the left button B3 or the right button B4 may be performed. Whether the virtual user is moving or stopped, the azimuth angle φ is updated so that the line-of-sight direction changes to the left at a predetermined speed while the left button B3 is pressed, and the right button B4 is pressed. Meanwhile, the azimuth angle φ is updated so that the line-of-sight direction changes to the right at a predetermined speed. For example, if the azimuth angle φ indicating the line-of-sight direction is set to 90 °, the azimuth angle φ is decreased to 89 °, 88 °, 87 °,... While the left button B3 is pressed ( When the angle reaches 0 °, a process of decreasing from 360 ° is performed, and while the right button B4 is pressed, the azimuth angle φ is increased to 91 °, 92 °, 93 °,. When the angle reaches 360 °, a process of increasing from 0 ° is performed.

  Thus, by operating the buttons B1 to B5, the user can freely walk in the virtual space constituting the corridor of the museum and turn his gaze in an arbitrary direction. On the display screen, the visual field image viewed from the virtual user is displayed as a moving image based on the user's instruction.

  As shown in FIG. 2, the museum's visiting path is composed of a plurality of routes R1 to R6 having nodes as both end points. Therefore, in the example shown here, when a virtual user reaches any node position, the virtual user is automatically stopped at the node and a message indicating that the node has been reached is displayed on the display screen. I have to. Also, when the user performs a process of selecting a new route starting from the reaching node and the user inputs a route selection instruction, the virtual user is allowed to play a video showing the state of moving to the selected route. ing. By adopting such a method, the user can give an instruction to proceed in a desired direction even when there is a path including a branch as in the vicinity of the nodes N2 and N5 shown in FIG.

<<< §2. Basic embodiment of the present invention >>
Now, in §1, the basic method of video presentation employed by the video presentation device according to the present invention has been described. If a moving image is presented using this basic method, an image such as a visual field image Q (P (i), φ (i)) shown in a thick frame in FIG. 8 is displayed on the display screen. It will be. In addition, a moving image is presented in accordance with the movement of the virtual user and the change in the viewing direction.

  However, in the example described in §1, there is a problem in that the moving object is not included as a subject, and thus there is a lack of realism. For example, in the case of the example shown in FIG. 8, the frame image F (i) is an omnidirectional image, and thus includes a 360 ° image, but the interior image inside the museum building and the exhibits are included. Only some paintings and sculptures are shown, and other viewers other than virtual users are not shown. Of course, if you want to concentrate on viewing only paintings and sculptures, it would be preferable to prepare a moving image that does not include moving objects as the subject's video because it would be an obstacle. However, in order to give the user a sense of realism that “actually visits the museum and appreciates it with other viewers”, a video including other viewers other than the virtual user as a subject is used. Is preferred. In particular, in the case of a moving image presentation device for giving a user a real experience of visiting various historical sites and sightseeing spots, it is preferable that a moving image including a moving object such as a person or a vehicle is presented.

  The present invention is based on the premise that a moving image including a moving object is presented to the subject. The method of preparing a moving image including a moving object in the subject is basically the same as the method described in §1. However, it is only necessary to shoot moving images in an environment where moving objects exist at the shooting site. For example, in the case of the art museum illustrated here, as shown in FIG. 4, an actual visitor (may be a general visitor or an extra hired for photography) is walking around the corridor. Shooting using an omnidirectional camera may be performed. Then, the viewer is reflected in the field of view image Q (P (i), φ (i)) shown in the thick frame in FIG. 8, and it is possible to present a moving image including a moving object in the subject. . As a result, it is possible to give a high sense of realism when the user performs a pseudo-viewing experience of the museum.

  However, if the basic method described in §1 is used as it is and a moving image presentation including other viewers (moving objects) is performed, there is a problem that the movements of other viewers (moving objects) become unnatural. For example, when moving image shooting is performed while moving the omnidirectional camera as shown in FIG. 4 from the node A to the node B on the route R shown in FIG. 3, as described in §1, each point P (1 ), P (2),..., P (i),..., P (N-1), P (N), frame images F (1), F (2),. , F (i),..., F (N−1), F (N) are obtained. These moving images composed of N frame images have natural movements as long as they are played forward (shooting order), but when played back (reverse to the shooting order), the movements of the moving objects are reversed. Become unnatural.

  For example, when a visual field image of a virtual user moving from the point P (i) shown in FIG. 7 toward the node B is presented as a moving image, each frame image F (i), F (i + 1), F (i + 2), Are taken out in this order, and the visual field images corresponding to the line-of-sight direction (azimuth angle φ) are cut out and sequentially displayed. Therefore, the movement of other viewers (moving objects) is natural. However, when the visual field image of the virtual user moving from the point P (i) toward the node A is presented as a moving image, each frame image F (i), F (i-1), F (i-2) , ... are taken out in this order, and the visual field image corresponding to the line-of-sight direction (azimuth angle φ) is cut out and displayed sequentially, so that the video is played back in reverse, and other viewers (moving objects) The movement will become unnatural.

  The present invention proposes a new proposal for dealing with such a problem. By including a moving object as a subject, the movement of the moving object can be seen naturally while giving the viewer a high sense of realism. It is an object of the present invention to provide a moving image presentation device capable of presenting a moving image.

  The basic concept of the present invention is that when the moving image presentation related to the route R as shown in FIG. Two types of moving images, that is, a return moving image Mb shot while moving toward A, are prepared, and either one of the moving images that can be played forward according to the moving direction desired by the user is reproduced. To do. Then, regardless of the direction of movement of the virtual user, the moving image is always played in forward order, so that an image including a moving object is presented to the user, giving a high sense of reality, It is possible to present videos that make the movement look natural.

  Hereinafter, the configuration and operation of the video presentation device 100 according to the basic embodiment of the present invention will be described with reference to the block diagram of FIG. As shown in the figure, the moving image presentation device 100 includes a moving image storage unit 110, a route information storage unit 120, a moving image playback unit 130, a moving direction setting unit 140, and an instruction input unit 150, and a field of view viewed from a moving viewpoint. As a video.

  Here, the moving image storage unit 110 is a component that stores moving images taken while moving along a path R connecting the first node and the second node. However, the same path R is shot while moving in the forward direction from the first node to the second node and moving in the backward direction from the second node to the first node. The main feature is that the two types of moving images, that is, the return moving image Mb, are stored.

  FIG. 11 is a moving image frame chart showing the relationship between the forward moving image Ma and the return moving image Mb stored in the moving image storage unit 110. In §1, referring to FIG. 3, each point P (1), P (2),..., P (i),. 1), an assembly of frame images F (1), F (2),..., F (i),..., F (N-1), F (N) photographed in P (N) Explained the structure of the video. N frame images F (1) to F (N) constituting the forward moving image Ma shown in FIG. 11 are exactly the same as the N frame images F (1) to F (N) shown in FIG. .

  That is, in order to prepare the N frame images F (1) to F (N) constituting the forward moving image Ma, first, as shown in FIG. 4 equipped with the fisheye lens 10 (or an omnidirectional mirror). While moving the omnidirectional camera in the forward direction from node A to B, a distorted circular image corresponding to a hemispherical field of view located above a predetermined horizontal plane is captured (see FIG. 5). Then, a rectangular panoramic image (omnidirectional image) as shown in FIG. 6 may be obtained by cutting out an area having an elevation angle equal to or smaller than a predetermined reference value from the distorted circular image and applying distortion correction thereto. . A panoramic image obtained from the captured image at the point P (i) becomes the frame image F (i).

  On the other hand, the N frame images G (1) to G (N) constituting the return moving image Mb can be prepared by substantially the same method. However, the moving direction of the omnidirectional camera at the time of shooting is the return direction from node B to A. After all, both the forward moving image Ma and the return moving image Mb are aggregates of frame images taken at each point on the same route R, but the moving direction at the time of shooting is reversed. Here, if the frame rate at the time of shooting (for example, 30 frames / second) and the moving speed of the omnidirectional camera (for example, 150 mm / second) are unified, the forward moving image Ma and the returning moving image Mb are both the same N A pair of frames photographed at the same point have a corresponding relationship.

  In FIG. 11, the arrows connecting the frame images F (1) to F (N) constituting the forward moving image Ma and the frame images G (1) to G (N) constituting the backward moving image Mb vertically These one-to-one correspondences are shown. For example, the frame F (1) and the frame G (N) photographed at the point P (1) correspond to each other, and the frame F (2) and the frame G (N−1) photographed at the point P (2) Corresponds. If expressed by a general formula, the frame F (i) photographed at the i-th point P (i) corresponds to the frame G (N−i + 1).

  In order to accurately match the shooting points of the individual frame images constituting the forward moving image Ma and the individual frame images constituting the backward moving image Mb, a rail is laid on the route R, and all the frames shown in FIG. It is preferable to take a picture while moving the azimuth camera along the rail in the forward direction and the backward direction.

  Here, when the moving image shot by the above-described method is a moving image that does not include a moving object as a subject, the corresponding frame images F (i) and G (N−i + 1) are basically the same. It becomes an image of the contents. Since the video that does not include moving objects only shows the still life of the museum interior, paintings, sculptures, etc., the frame images taken at the same point will be the same if the lighting environment at the time of shooting does not change .

  On the other hand, when a moving image including a moving object is photographed as a subject, the motion of the moving object differs for each photographing take, so that the corresponding frame images F (i) and G (N−i + 1) are related to the background portion. Even if the contents are the same, the moving object part (foreground part) is different. Of course, if an extra acting as a viewer is employed as a moving object and the same performance is performed during the shooting of the outbound video Ma and the backward video Mb, the moving body part (foreground part) It is possible to prepare videos that are somewhat similar. However, as long as the live-action shooting is performed, it is impossible to completely match the contents of the frame images F (i) and G (N−i + 1), and in the present invention, it is not necessary to dare to match both. .

  As described above, the moving image storage unit 110 illustrated in FIG. 10 captures the moving image Ma moving in the forward direction on the route R and moving while moving on the same route R in the backward direction. The return path moving image Mb including a moving object in the subject is stored. In addition, since embodiment shown here concerns on the apparatus which performs the moving image presentation about the visit route containing 6 path | routes R1-R6 as shown in FIG. Will store the forward moving image Ma and the returning moving image Mb for each of the six routes R1 to R6. In FIG. 10, for convenience of illustration, only the state in which the forward moving image M1a and the return moving image M1b are stored for the route R1 is clearly shown. However, in practice, two types of moving images are similarly displayed for the routes R2 to R6. Will be stored.

  As for a specific route, which direction is the forward direction or the backward direction can be freely determined by the designer of this device, so in consideration of convenience when shooting each video, What is necessary is just to determine suitably.

  On the other hand, the route information storage unit 120 is a component that stores route information RT indicating the connection relation of each route through each node. In general, when handling moving images for a plurality of routes connecting three or more nodes, the moving image storage unit 110 stores moving images for a plurality of routes connecting three or more nodes, respectively. It is necessary to keep. In the case of the example shown here, as described above, the visit path is formed by the six nodes N1 to N6 and the six paths R1 to R6 connecting a predetermined pair of these nodes, In the moving image storage unit 110, moving images (outbound moving image Ma and return moving image Mb) for the six routes R1 to R6 are stored. The route information storage unit 120 stores route information RT indicating the connection relationship of each route. Specifically, for example, if information such as “nodes N1, N2: route R1” is prepared as the route information RT, it can be shown that the route R1 exists between the nodes N1, N2. .

  The instruction input unit 150 is a component that inputs a user instruction, and can give instructions to the virtual user such as movement / stop, setting of the movement direction, and change of the line-of-sight direction. The moving direction setting instruction is given to the moving direction setting unit 140. Based on the given instruction, the movement direction setting unit 140 sets either the forward direction or the backward direction as the movement direction on the route R where the virtual user is currently located.

  On the other hand, an instruction to move / stop the virtual user and an instruction to change the line-of-sight direction are given to the moving image reproducing unit 130. The moving image playback unit 130 plays a function of playing back a moving image stored in the moving image storage unit 110 based on a given instruction and displaying it on the screen of the display device 200. For example, when a movement instruction is given to the virtual user, the movement direction set in the movement direction setting unit 140 is referred to, and when the forward direction is set, the forward moving image Ma is sequentially played as a reproduction target. When the backward direction is set, the forward moving image Mb is reproduced in the forward direction as the reproduction target. When a stop instruction is given to the virtual user, the moving image playback is stopped, and the currently displayed frame image is displayed as a still image.

  In the case of the basic embodiment shown here, each moving image stored in the moving image storage unit 110 is omnidirectional with a 360 ° field of view taken using an omnidirectional camera while moving along the path R. The image is composed of images (see FIG. 6) recorded in units of frames. Therefore, as shown in FIG. 8, the moving image reproduction unit 130 cuts out a visual field image Q corresponding to a predetermined line-of-sight direction (azimuth angle φ) from each frame constituting the moving image, and displays this image on the screen of the display device 200. Will be displayed. The instruction input unit 150 has a function of inputting an instruction indicating a specific line-of-sight direction, and the moving image reproduction unit 130 determines a position to cut out the visual field image Q based on the instruction given from the instruction input unit 150. The line-of-sight direction (azimuth angle φ) for determination is recognized.

  Actually, the instruction input unit 150 includes a controller 50 as shown in FIG. 9 and an interface for fetching operation instructions from the controller 50. As described in §1, the controller 50 includes a forward button B1, a backward button B2, a left button B3, and a right button B4. Then, the movement direction setting unit 140 sets the movement direction to the first direction based on the operation instruction for the forward button B1, and sets the movement direction to the second direction based on the operation instruction for the backward button B2. Process.

  In addition, the moving image reproduction unit 130 performs a process of changing the line-of-sight direction to the left based on the operation instruction of the left-pointing button B3 and changing the line-of-sight direction to the right based on the operation instruction of the right-pointing button B4. More specifically, the moving image reproducing unit 130 has a function of storing the current azimuth angle φ, and a process of reducing φ when the left-pointing button B3 is pressed (from 360 ° when 0 ° is reached). And a process of increasing φ when the right button B4 is pressed (a process of increasing from 0 ° when 360 ° is reached).

  In the embodiment shown here, as illustrated in FIG. 2, it is possible to present a moving image in a state in which a visit route composed of a plurality of routes is freely moved. Therefore, the instruction input unit 150 has a function of inputting a user's route transfer instruction to move from the first route to the second route via a specific node. When an instruction is input, a function is provided for referring to the route information RT in the route information storage unit 120 so that the moving image reproduction for the second route is performed continuously with the moving image reproduction for the first route. doing. For example, in the example shown in FIG. 2, when a route transfer instruction for moving the route R2 from the node N2 to the node N3 after the route R6 is moved from the node N5 to the node N2 is input, Reference numeral 130 recognizes the route transfer instruction by referring to the route information RT, and continues the moving image reproduction for the route R6 and then reproduces the moving image for the route R2.

  As described above, according to the moving picture presentation apparatus shown in FIG. 10, the moving picture presentation in a state in which the line of sight is directed in an arbitrary direction while moving the visit path of the corridor of the museum shown in FIG. It can be performed. In addition, regardless of which route is moved in which direction, since one of the forward moving image Ma and the backward moving image Mb (which can be played forward) is played, the moving image is always played forward. . For this reason, it is possible to present a moving image in which the motion of the moving object always looks natural.

  For example, in the example shown in FIG. 11, the virtual user turns the line of sight near the point P (i) on the route R toward the azimuth angle φ = 90 ° (direction facing the node B), and the forward direction (to the node B). Consider the case of moving in the direction of heading. In this case, since the forward direction is set in the movement direction setting unit 140, the moving image reproduction unit 130 performs forward reproduction with the outbound moving image Ma as a reproduction target.

  FIG. 12 is a diagram showing a playback screen for such a forward moving image Ma. In the forward playback, the frame images F (i−1), F (i), and F (i + 1) constituting the forward moving image Ma are displayed in this order, and therefore, the person on the left side facing the front is displayed on the display screen. Will be presented, and a natural video will be presented in which the person on the right side facing the back moves away. Actually, when the frame rate is about 1/30 seconds, an extreme transition of the person position as in the illustrated example is not seen due to a difference of one frame, but here, for convenience of explanation, the transition of the person position is not seen. Is exaggerated. Each image shown in FIG. 12 should actually be a field-of-view image cut out from a part of the panoramic image as shown in FIG. 8, but here, the azimuth angle φ == Considering the state fixed at 90 °, for the sake of convenience, these visual field images will be referred to as frame images F (i−1), F (i), and F (i + 1).

  If such sequential playback is performed, the display order of the frame images follows the flow of time at the time of shooting, so that the user does not feel discomfort and is grasped as a very natural moving image. On the other hand, if the same frame image is played back in reverse, an unnatural movie is presented in which the left person facing the front moves away and the right person facing the back approaches. become.

  On the other hand, FIG. 13 is a diagram showing a playback screen of the return trip video Mb. In sequential playback, the frame images G (N−i), G (N−i + 1), and G (N−i + 2) that form the backward moving image Mb are displayed in this order. A natural video approaching the woman who was approaching will be presented (again, for convenience of explanation, the transition of the person position is exaggerated and each field image is called a frame image). If such sequential playback is performed, the display order of the frame images follows the flow of time at the time of shooting, so that the user does not feel discomfort and is grasped as a very natural moving image. On the other hand, if the same frame image is played back in reverse, an unnatural movie is presented in which a woman facing the front moves away.

  As described above, when moving in the forward direction (direction toward the node B) on the route R shown in FIG. 11, the moving image reproducing unit 130 sequentially reproduces the forward moving image Ma as shown in FIG. When moving in the backward direction (direction toward node A) on the route R shown in FIG. 13, the backward moving image Mb as shown in FIG. That is, even when moving in any direction, the moving image is reproduced in order, so that natural moving image presentation is always performed.

  The configuration of the video presentation device 100 according to the basic embodiment of the present invention has been described above with reference to the block diagram of FIG. 10, but actually, each of these components uses a computer and its peripheral devices. Can be configured. That is, the moving image storage unit 110, the route information storage unit 120, and the moving direction setting unit 140 can be configured by a storage device such as a computer memory or a disk device, and the moving image playback unit 130 is based on a predetermined program. The instruction input unit 150 can be configured by an input device for a computer having a user interface. Therefore, practically, the moving picture presentation apparatus 100 is configured by incorporating a predetermined program into a computer device.

<<< §3. Embodiment for setting transition period >>
In §2 described above, for the same route R, both the forward moving image Ma and the return moving image Mb are prepared, and one of these two types of moving images is selected according to the moving direction of the virtual user, The basic feature of the present invention that a natural moving image is presented by always performing progressive playback has been described. After all, in the present invention, the forward moving image Ma and the return moving image Mb are switched and presented according to the moving direction of the virtual user.

  By the way, the frame image that forms the forward moving image Ma and the frame image that forms the return moving image Mb are different in content because they contain different moving objects even if they are images taken at the same point. What has happened is as already mentioned. For example, the frame image F (i) shown in FIG. 12 is an image constituting one frame of the forward moving image Ma taken at the point P (i), and the frame image G (N−i + 1) shown in FIG. Similarly, it is an image constituting one frame of the return moving image Mb photographed at the point P (i). Since both are images taken at the same point P (i) and showing the field of view in the same line-of-sight direction, the background portion is the same, but the images are obtained by different shooting takes, so the moving body portion (foreground portion) ) Is different. That is, two persons are shown in the frame image F (i) shown in FIG. 12, whereas another completely different person is shown in the frame image G (N−i + 1) shown in FIG. ing.

  Now, let us consider a case where the virtual user reverses the moving direction at the point P (i) and starts moving in the backward direction while moving the route R shown in FIG. 11 in the forward direction. In order to give an instruction for reversing the moving direction in this way, the instruction input unit 150 has a function of inputting a reversing instruction for reversing the moving direction, and the moving direction setting unit 140 is based on the input of this reversal instruction. Thus, a process of switching the moving direction from the forward direction to the backward direction or from the backward direction to the forward direction may be performed. Specifically, as described in §2, if the controller 50 as shown in FIG. 9 is used as the instruction input unit 150, the forward button B1 is pressed to give a movement instruction in the forward direction, and the forward direction. While the moving video is being presented, the reverse instruction can be input by pressing the backward button B2, and the moving forward is presented while the moving video is being presented. It is also possible to input a reverse instruction by pressing the button B1.

  FIG. 14 is a time chart showing a basic switching mode of a moving image by the moving image reproducing unit 130 when switching from the forward direction to the returning direction is performed at the point P (i) by such inversion instruction input. The horizontal axis of the time chart shown in this application is a time axis representing the time t flowing from the left to the right. The upper band in FIG. 14 shows each frame image of the forward moving image Ma, and the lower band shows the return moving image Mb. Each frame image is shown. In the chart, the left direction of the upper band indicating the forward moving image Ma is the first frame F (1), and the right direction is the Nth frame F (N). Similarly, the left direction of the lower band indicating the backward moving image Mb is the first frame G (1), and the right direction is the Nth frame G (N).

  In this example, the virtual user is moving in the forward direction from the node A side of the route R shown in FIG. 11 to the point P (i) until the inversion instruction input time t0. The hatched portion of the forward moving image Ma shown in the upper band is reproduced in order. However, when an inversion instruction input is given at time t0, the movement direction is reversed in the backward direction. Then, after the inversion instruction input time t0, the virtual user moves in the backward direction from the point P (i) of the route R shown in FIG. 11 toward the node A, and the moving image reproducing unit 130 is in the lower part of the chart. The hatched portion of the return path video Mb shown in the band is reproduced in order. After all, the hatched portion in this chart is the frame image that is actually displayed. The symbol “order” written in the hatched portion indicates that forward reproduction is performed.

  As described above, when the inversion instruction input is given at the time point t0 and the point P (i), the moving image to be reproduced is switched from the forward moving image Ma to the backward moving image Mb at the time point t0. Here, looking at the transition of the frame image before and after the switching, the frame image F (i−1) → the frame image F (i) shown in FIG. 12 → the frame image G (N−i + 1) shown in FIG. 13 → Display is performed in the order of frame image G (N−i + 2). For this reason, discontinuity occurs in the moving moving object when switching from the frame image F (i) to the frame image G (N−i + 1). That is, although two persons are shown up to the frame image F (i), the frame image G (N−i + 1) is replaced with a completely different person.

  Of course, from the user's point of view, the moving direction of the virtual user is reversed and the content of the moving image is switched due to the input of the reversal instruction given by the user, and the moving image is switched to the moving image after the switching (returning moving image Mb). The movements of new characters appearing naturally are natural, so even if people are replaced, there will be no great discomfort. However, sudden change of people at time t0 is not preferable for more natural video presentation. In order to alleviate such a sudden moving phenomenon of a moving object, a transition period T is set when switching between the presentation of the forward moving picture Ma and the presentation of the return moving picture Mb, and during the transition period T, What is necessary is just to show the synthesized image which blended the moving image after switching. If it does so, the moving body in an image | video can be switched smoothly and the unnaturalness of the moving body change at the time of switching can be reduced.

  FIG. 15 is a time chart showing a switching mode through such a transition period T. The difference from the time chart shown in FIG. 14 is that a transition period T is provided from the inversion instruction input time t0 to the time point t1, and in this transition period T, the frame image of the forward moving image Ma and the frame image of the return moving image Mb This is a point that a composite image with Brent is displayed. That is, in the embodiment described in §3, the moving image reproducing unit 130 sequentially reproduces the hatched portion of the forward moving image Ma shown in the upper band of the chart until the inversion instruction input time t0, and after time t1, The hatched portion of the backward moving image Mb shown in the lower band is sequentially played back. As described above, the symbol “order” described in the hatched portion of the drawing indicates that forward reproduction is performed.

  Then, the moving image reproducing unit 130 sets a predetermined period after the inversion instruction is input to the transition period T, and during this transition period T, the moving image before switching (shown in FIG. 15) that is a reproduction target according to the moving direction before switching. In the case of the example, the pre-switching image constituting the forward moving image Ma) and the post-switching moving image (returning moving image Mb in the case of the example shown in FIG. 15) to be reproduced according to the moving direction after switching are configured and the pre-switching image. A composite image obtained by blending the post-switching image taken at the same position (in the example shown in FIG. 15, the point P (i)) is displayed. In the case of the example shown in FIG. 15, during the transition period T, the i-th frame image F (i) constituting the forward moving image Ma and the (N−i + 1) -th forming the corresponding backward moving image Mb. The composite image obtained by blending the frame image G (N−i + 1) is displayed.

  FIG. 16 is a diagram illustrating the principle of the image blending process performed by the moving image playback unit 130. The frame image F (i) shown in the figure is an image constituting the forward moving image Ma photographed at the point P (i) as shown in FIG. 12, and the frame image G (N−i + 1) is shown in FIG. In this way, it is an image that constitutes the return moving image Mb photographed at the point P (i). As described above, these two frame images are both images taken at the same point P (i), but the frame image F (i) includes two persons as moving objects. The image G (N−i + 1) includes another person as a moving object. Since both images are visual field images cut out with respect to the same line-of-sight direction (azimuth angle φ), although the background portion is common, the moving body portion (foreground portion) is different.

  The composite image B (i) is obtained by blending these two images F (i) and G (N−i + 1) based on a predetermined ratio. Thus, as a method for obtaining a composite image B (i) by blending two images, a method called “α blend” is generally known. In this method, assuming that the ratio of the image G (N−i + 1) after switching is α and the ratio of the image F (i) before switching is (1−α), B (i) = (1−α) · F ( i) A composite image B (i) can be obtained by combining both images based on the equation + α · G (N−i + 1). Specifically, for each of the pixel values of the corresponding pixels of both images (in the case of a color image, the pixel value of each color), a synthesis operation based on the above formula is performed, and the obtained value is used as a new pixel value. A collection of pixels may be a composite image.

  Here, if the value α is gradually increased from 0 to 1 as the transition period T elapses, the composite image B (i) gradually changes from the image F (i) to the image G (N−i + 1). Will change. A composite image B (i) shown in FIG. 16 shows a state when α = 0.5 (a state at an intermediate point in the transition period T). As described above, since both images are images having a common background part, if the value α is gradually increased from 0 to 1 during the transition period T, the background image remains constant. A moving image display (so-called transition effect having a dissolve effect) is performed in which a person disappears gradually, and another person gradually appears.

  In the transition period T of the time chart shown in FIG. 15, the hatched portions of the bands shown in the upper and lower stages are triangular, which indicates such a change in blend ratio over time. That is, the vertical width of the hatched portion indicates the blend ratio of each image, and the vertical width (corresponding to (1-α)) of the hatched portion of the forward moving image Ma shown in the upper portion decreases with time, and the lower portion. The vertical width (corresponding to α) of the hatched portion of the return moving image Mb shown increases with time. In addition, the symbol “static” written in the hatched portion of the figure is a still image in which the virtual user is stationary (here, the background image means a constant image, and the moving image gradually changes in terms of moving objects). Is) to be presented.

  That is, in the example shown in the figure, the i-th frame image F (i) constituting the forward moving image Ma is displayed immediately before the inversion instruction input time t0. The composite image B (i) obtained by blending the frame image F (i) and the (N−i + 1) th frame image G (N−i + 1) of the corresponding return moving image Mb is displayed. become. Then, after the time point t1 when the transition period T ends, the backward moving image Mb is sequentially reproduced from the frame image G (N−i + 1).

  When the transition period T is set in this way, the transition of the frame image before and after switching from the forward moving image Ma to the backward moving image Mb is as follows. First, frame image F (i-1) → frame image F (i) shown in FIG. 12 is displayed by forward playback of forward moving image Ma, and transition period T is entered at time t0. In the transition period T, the composite image B (i) shown in FIG. 16 is displayed (the blend ratio α gradually increases from 0 to 1), and after the transition period ends, the frame image G (N -I + 1) → frame image G (N−i + 2) The forward moving image Mb is sequentially played back.

  Therefore, when viewed from the user, if a reverse instruction input is given while moving the route R in the forward direction, the position of the virtual user stops at the point P (i), and the frame image F (i) shown on the upper left of FIG. Is displayed on the frame image G (N−i + 1) shown in the upper right of FIG. 16, and when the frame image G (N−i + 1) is completely switched, the route R is moved in the backward direction. A video will be displayed. As a result, in the transition period T, the two persons shown so far are replaced with another person by a transition having a dissolve effect. That is, the effect that the moving body in the video is smoothly switched is obtained, and the unnaturalness of the moving body change at the time of switching from the forward moving image Ma to the backward moving image Mb can be eliminated.

  Eventually, to implement the embodiment for setting the transition period described in §3, as shown in FIG. 11, the first moving image Ma taken at the position P (1) of the first node A is taken. A total of N frame images from the frame F (1) to the Nth frame F (N) taken at the position P (N) of the second node B, and the return video Mb From the first frame G (1) taken at the position P (N) of the second node B to the Nth frame G (N) taken at the position P (1) of the first node A Up to a total of N frame images. At this time, the i-th frame F (i) of the forward moving image Ma and the (N−i + 1) th frame G (N−i + 1) of the return moving image Mb are the same point P (i) on the route R. A photographed image is obtained and frames corresponding to each other are configured. Then, during the transition period T, the moving image playback unit 130 may perform a process of creating a composite image by combining frames corresponding to each other and displaying the composite image.

  The example shown in FIG. 15 is an example of switching from the forward moving image Ma to the backward moving image Mb (that is, an example in which a reverse instruction is input while moving in the forward direction on the route R shown in FIG. 11). The switching process for the example of switching from the moving image Mb to the forward moving image Ma (that is, the example in which the reverse instruction is input while moving in the backward direction on the route R shown in FIG. 11) can be performed in the same manner.

  In short, during the transition period T, the moving image reproduction unit 130 images the frame immediately before the transition period T of the pre-switching moving image (in the example of FIG. 15, outbound moving image Ma) (in the case of FIG. 15, the frame image F (I)) and a corresponding frame image (frame image G (N−i + 1) in the example of FIG. 15) of the post-switching moving image (in the example of FIG. 15, the return moving image Mb) corresponding thereto A fixed-position image (in the case of the example of FIG. 15, an image taken at the point P (i) in the example of FIG. 15) is displayed, and after the transition period T, the corresponding frame of the post-switching video (example of FIG. 15) is displayed. In this case, the forward playback may be started from the frame image G (N−i + 1)).

  However, the composite image displayed in the transition period T is not necessarily a fixed position image (an image in a state where the virtual user is stopped). When the fixed position image is displayed during the transition period T as in the example illustrated in FIG. 15, the virtual user has an effect of “stopping for a while and then starting to move in the opposite direction” at the point P (i). Video presentation becomes possible. However, the moving image presented when the reverse instruction input is given is not necessarily limited to the effect of “moving in the reverse direction after stopping for a while”. In the following, two variations with different effects are described.

  The first variation is that, when a reversal instruction input is given, “it proceeds for a while and starts moving in the reverse direction at a point P (i + d) that has passed the point P (i) at the time of reversal instruction input”. This is a method of giving a production effect. FIG. 17 is a time chart showing a moving image switching mode when such an effect is adopted. The difference from the chart shown in FIG. 15 is that the transition period T is divided into the first half of the times t0 to t1 and the second half of the times t1 to t2, and the playback order of each moving image is changed every half.

  That is, the moving image reproducing unit 130 continues the forward moving image Ma (moving image before switching) shown in the upper part of FIG. 17 in the first half of the transition period T (time t0 to t1) as in the state before the transition period T. In the second half of the transition period T (time t1 to t2), this is played back in reverse from the last frame of the first half (the period from the time t1 to t2 in the zone of the forward moving picture Ma is a time axis) In reverse order). More specifically, frame images F (i), F (i + 1), F (i + 2),..., F (i + d) are sequentially played back at time t0 to t1, and the playback order is reversed at time t1. From time t1 to time t2, frame images F (i + d),..., F (i + 2), F (i + 1), and F (i) are played back in reverse.

  On the other hand, the moving image reproduction unit 130 relates to the return moving image Mb (moving image after switching) shown in the lower part of FIG. 17, during the transition period T, the returning moving image corresponding to each frame image of the moving image Ma (moving image before switching). Each frame image of Mb (moving image after switching) is reproduced. That is, in the first half of the transition period T (time t0 to t1), the backward moving image Mb is reversely played back from the frame image G (N−i + 1) (the interval from the time t0 to t1 in the band of the backward moving image Mb is a frame). In the latter half of the transition period T (time points t1 to t2), the images are sequentially played back. More specifically, at time points t0 to t1, frame images G (N−i + 1), G (N−i), G (N−i−1),..., G (N−i + 1−d) Backward playback is performed, and the playback order is reversed at time t1. From time t1 to time t2, frame images G (N−i + 1−d),..., G (N−i−1), G (N−i) , G (N−i + 1) and forward playback.

  Then, after the transition period T ends (after time t2), the backward moving image Mb (moving image after switching) may be continuously played forward. Note that the symbol “order” described in the hatched portion in FIG. 17 indicates that forward playback is performed (reproduction according to the frame order in the shooting order), and the symbol “reverse” is reverse playback. (Reproduction according to the frame order opposite to the shooting order) is performed (the same applies to the following figures). The forward moving image Ma and the return moving image Mb have opposite path moving directions at the time of shooting, and the composite image is an image obtained by combining frame images taken at the same point in both images. Therefore, when the forward moving image Ma and the backward moving image Mb are combined to create a composite image, if one is played forward, the other must be played backward. This is why the “forward” / “reverse” symbols shown in the upper and lower bands in the transition period T in FIG. 17 are contradictory.

  When the moving image presentation is performed in this way during the transition period T, the virtual user continues to move from the point P (i) even after the inversion instruction input is given at the time point t0, and at the time point t1, the point P When (i + d) is reached, the movement direction is reversed, the movement returns to the point P (i) at the time t2, and the movement is continued as it is. Moreover, since the blend ratio α is gradually increased from 0 to 1 during the transition period T, the displayed composite image is gradually changed from the frame image of the forward moving image Ma to the frame image of the backward moving image Mb. It will switch. Therefore, the effect that the moving object in the video is smoothly switched is obtained, and the unnatural change of the moving object at the time of switching from the forward moving image Ma to the backward moving image Mb is eliminated, as in the method shown in FIG. That is, the moving object is naturally replaced by a transition having a dissolve effect.

  In practice, the video playback unit 130 gradually decreases the video playback speed from the speed immediately before the transition period in the first half of the transition period T, and gradually increases the playback speed in the second half of the transition period T. It is preferable to increase the number of times. Then, after the reverse instruction input is given at time t0, the moving speed of the virtual user gradually decreases, stops at time t1, and then gradually increases in the reverse direction. It is possible to provide an effect of moving while moving, and to express more natural movement for the virtual user.

  The second variation is a method of giving an effect of “invert immediately and start moving in the opposite direction” when an inversion instruction input is given. FIG. 18 is a time chart showing a moving image switching mode when such a production effect is adopted. The difference from the chart shown in FIG. 15 is that at the time of inversion instruction input t0, playback of the forward moving image Ma is immediately switched to reverse playback. That is, during the transition period T, the video playback unit 130 plays back the forward video Ma (video before switching) backward from the frame immediately before the transition period, and each frame of the forward video Ma and the corresponding return video Mb. A moving image composed of a combined frame obtained by combining each frame of (moving image after switching) is displayed. After the transition period T ends, the backward moving image Mb may be continuously played back in sequence.

  More specifically, during the transition period T, frame images F (i), F (i-1), F (i-2),... (Id) and reverse playback (the frames from the time point t0 to t1 in the band of the forward moving image Ma are frames arranged in reverse order along the time axis), and the backward moving image Mb (moved video after switching) ), Frame images G (N−i + 1), G (N−i + 2), G (N−i + 3),..., G (N−i + 1 + d) are sequentially reproduced. Then, after the transition period T ends (after time t1), the backward moving image Mb (moving image after switching) may be continuously played forward.

  When a moving image is presented in such a way during the transition period T, the virtual user will immediately reverse from the point P (i) when a reverse instruction input is given at time t0. . Moreover, since the blend ratio α is gradually increased from 0 to 1 during the transition period T, the displayed composite image is gradually changed from the frame image of the forward moving image Ma to the frame image of the backward moving image Mb. It will switch. Therefore, the effect that the moving object in the video is smoothly switched is obtained, and the unnatural change of the moving object at the time of switching from the forward moving image Ma to the backward moving image Mb is eliminated, as in the method shown in FIG. That is, the moving object is naturally replaced by a transition having a dissolve effect.

  As described above, with reference to FIGS. 15, 17, and 18, the three presentation methods of moving image presentation performed during the transition period T have been described, but each of these methods has advantages and disadvantages. This will be briefly described below.

  First, when the method shown in FIG. 15 is adopted, since the movement of the virtual user immediately stops at the position P (i) at the time point t0 when the inversion instruction input is given, the merit of quick response of the movement of the virtual user to the inversion instruction. Is obtained. However, during the transition period T, the movement of the virtual user remains stopped, and there is a demerit that the moving image indicating the state of moving on the route R is temporarily interrupted.

  In addition, when the method shown in FIG. 17 is adopted, the virtual user is in a state of continuing movement, and there is an advantage that the moving state moving image is not interrupted. However, even after the reverse instruction input is given, the virtual user continues to move for a while, and a slight delay occurs until the moving direction is reversed, which causes a disadvantage that the responsiveness of the virtual user to the operation input is lowered.

  On the other hand, when the method shown in FIG. 18 is adopted, since the virtual user immediately reverses and moves, there is a merit that the moving video is not interrupted, and there is also a merit that quick response of the virtual user's movement to the reversal instruction. can get. However, as compared with the method shown in FIG. 17, there is a demerit that the unnatural motion of the moving object becomes remarkable. For example, in FIG. 18, the moving image displayed immediately after time t0 is a composite image close to α = 0, so the moving object (two persons shown in FIG. 12) reflected in the outbound moving image Ma is It is displayed more clearly than the moving object (one person shown in FIG. 13) shown in the return path video Mb. However, during the transition period T, the forward moving picture Ma is played back in reverse, so that this clear moving object is shown in reverse, and the unnaturalness of the movement becomes noticeable.

  Of course, also in the method shown in FIG. 18, since the blend ratio α gradually increases with time, the two persons on the outbound video Ma will gradually disappear, but at least the transition period At the beginning of T, the unnatural movement of these two persons will be conspicuous. On the other hand, in the case of the method shown in FIG. 17, since the forward moving image Ma is played forward in the first half of the transition period T, the unnatural motion of the moving object does not become remarkable. Although the backward moving image Mb is played back in the first half, the unnatural movement of the person on the backward moving image Mb is not so noticeable because the blend ratio α is small. Further, in the second half, the forward moving picture Ma is played back in reverse, but at that time, the blend ratio α is large, so the unnatural movement of the two persons is not so noticeable.

  Here, three types of presentation methods of moving image presentation performed during the transition period T are illustrated, but of course, the presentation method of moving image presentation performed during the transition period T is not limited to these three types of methods. In addition to this, various production methods can be employed. In short, during the transition period T, a synthesized image obtained by blending the pre-switching video and the post-switching video can be presented, and the moving object can be smoothly switched, and the unnaturalness of the moving object's motion at the time of switching What is necessary is that it can be reduced.

<<< §4. Embodiment using background video >>>
In §3, an embodiment has been described in which a transition period T is set at the time of reversal switching of the moving direction, and a composite image of the forward moving image Ma and the backward moving image Mb is displayed to smoothly switch the video. Here, in addition to the forward moving image Ma and the returning moving image Mb, a background moving image Mc that does not include a moving object is prepared, and during the transition period T, a background moving image is displayed between the pre-switching moving image and the switching moving image. An embodiment will be described in which blending with Mc is performed to further smoothly switch moving objects.

  FIG. 19 is a block diagram showing a configuration of a moving image presentation apparatus 100 ′ according to an embodiment using this background moving image. The difference from the moving image presentation device 100 shown in FIG. 10 is that the moving image storage unit 110 ′ stores the background moving image Mc in addition to the forward moving image Ma and the return moving image Mb, and the moving image reproducing unit 130 ′ changes. In the period T, the blending is performed by inserting the background moving image Mc between the pre-switching moving image and the post-switching moving image. Hereinafter, these differences will be described in detail.

  FIG. 20 is a moving image frame chart showing the relationship between the forward moving image Ma, the return moving image Mb, and the background moving image Mc stored in the moving image storage unit 110 ′. The background moving image Mc is further added to the chart shown in FIG. Is. In the case of the example shown in FIG. 20, the forward moving image Ma is the Nth image captured at the position of the second node B from the first frame F (1) captured at the position of the first node A. The frame is composed of a total of N frame images up to the frame F (N), and the backward moving image Mb is taken from the first frame G (1) taken at the position of the second node B to the first node. The frame is composed of a total of N frame images up to the Nth frame G (N) taken at the position A, and the background moving image Mc is the first taken at the position of the first node A. A total of N frame images from the frame H (1) to the Nth frame H (N) photographed at the position of the second node B.

  Here, the outbound video Ma and the inbound video Mb are videos that include moving objects in the subject, while the background video Mc is a video that does not include moving objects in the subject. The background video Mc is a video shot while moving in the forward direction on the route R, and each point P (1), P (2),..., P (i),. N-1) and P (N), images that do not include moving objects are frame images H (1), H (2),..., H (i),. -1) and H (N).

  Eventually, the outbound video Ma and the background video Mc are common in that they are composed of a total of N frame images taken in the order of the points P (1), P (2),. The moving image Ma is a video in which a moving object such as another viewer is reflected, whereas the background moving image Mc does not reflect such a moving object, and is only an image of the interior of the museum and exhibits. Such a background moving image Mc may be taken in a state where there is no general viewer such as a closed day of a museum.

  Note that the background video Mc is an image that does not include a moving object, and therefore, if the lighting environment is constant regardless of whether it is shot while moving in the forward direction or moving in the return direction, it is substantially the same. It becomes a video of the content (only the frame order is reversed). Therefore, the background moving image Mc in the example shown in FIG. 20 is obtained by shooting while moving in the forward direction, but conversely, the moving image obtained by performing shooting while moving in the backward direction. May be used as the background video Mc.

  Since the moving image is not included in the background moving image Mc, unnaturalness does not occur even when the backward reproduction is performed. FIG. 21 is a diagram showing a playback screen for the background video Mc shown in FIG. The frame images H (i−1), H (i), and H (i + 1) of the background moving image Mc shown in FIG. 21 are the frame images F (i−1) and F (i) of the forward moving image Ma shown in FIG. ), Which corresponds to an image obtained by removing two moving persons from F (i + 1). In other words, the frame images H (i−1), H (i), and H (i + 1) are the same images as the background portions of the frame images F (i−1), F (i), and F (i + 1). It is.

  When progressive playback is performed for the background moving image Mc shown in FIG. 21, the frames H (i-1), H (i), H (i + 1) are displayed in this order, and the points P (i-1), P (i ), A state of moving to P (i + 1) is presented. On the other hand, when reverse playback is performed, the frames H (i + 1), H (i), and H (i-1) are displayed in this order, and the points P (i + 1), P (i), and P (i-1) are displayed. The state of moving to is presented. In either case, there is no sense of incongruity because no moving object is shown.

  Actually, when the frame rate is about 1/30 seconds, the transition of the scenery as shown in the example in the figure is not seen due to the difference of one frame, but in FIG. Exaggerated. In addition, each image shown in FIG. 21 is actually a visual field image cut out from a part of the omnidirectional image as shown in FIG. 8, but here, the azimuth angle φ indicating the line-of-sight direction is shown. For convenience, these field images will be referred to as frame images H (i−1), H (i), and H (i + 1).

  The instruction input unit 150 illustrated in FIG. 19 is the same component as the instruction input unit 150 illustrated in FIG. 10 and has a function of inputting a reversal instruction for reversing the moving direction. 19 is also the same component as the movement direction setting unit 140 shown in FIG. 10, and performs a process of switching the movement direction based on the input of the reverse instruction from the instruction input unit 150. Then, similarly to the moving image reproducing unit 130 shown in FIG. 10, the moving image reproducing unit 130 ′ shown in FIG. 19 is sequentially forwarded with the outgoing moving image Ma being reproduced when the forward direction is set in the movement direction setting unit 140. When the return direction is set in the movement direction setting unit 140, the basic function is to perform forward playback with the return moving image Mb as a playback target.

  However, when there is a moving direction reversal instruction input from the instruction input unit 150, the moving image reproduction unit 130 ′ sets a predetermined period after the reversal instruction input as the transition period T, and It has a function of performing smooth switching of moving object display by performing blending with a background moving image Mc interposed therebetween. In other words, during the transition period T, the moving image playback unit 130 ′ and the background video and the pre-switching image constituting the pre-switching video (outbound video Ma or return video Mb) to be played back according to the moving direction before switching. The first synthesized image blended with the background image captured at the same position as the pre-switching image, the background image constituting the background video, and the post-switching that is the playback target according to the moving direction after switching A second composite image obtained by blending a post-switching image that is formed at the same position as the background image and that constitutes the video (inbound video Mb or forward video Ma) is converted into the first composite image and the second composite image. Process to display in order.

  FIG. 22 is a time chart showing an example of a moving image switching mode by the moving image reproducing unit 130 ′ when the inversion instruction is input in the moving image presentation device 100 ′ shown in FIG. The horizontal axis of this chart is a time axis representing the time t flowing from left to right as in the time charts shown so far, and the upper band indicates each frame image of the forward moving image Ma, and the middle band. Indicates each frame image of the background moving image Mc, and the lower band indicates each frame image of the return moving image Mb. In the chart, the left direction of the upper band indicating the forward moving image Ma is the first frame F (1), and the right direction is the Nth frame F (N). Similarly, the left direction of the middle band indicating the background moving image Mc is the first frame H (1), the right direction is the Nth frame H (N), and the left direction of the lower band indicating the backward moving image Mb is The first frame G (1) and the right direction are the Nth frame G (N).

  In this example, until the reverse instruction input time t0, the virtual user is moving in the forward direction from the node A side of the route R shown in FIG. 20 toward the point P (i), and the moving image reproducing unit 130 ′ The hatching portion of the forward moving image Ma shown in the upper band of the chart is reproduced in order. However, when an inversion instruction input is given at time t0, after the transition period from time t0 to t2, the hatched portion of the backward moving image Mb shown in the lower band of the chart is sequentially played back after time t2. As described above, the symbol “order” described in the hatched portion of the drawing indicates that forward reproduction is performed.

  In this embodiment, the transition period is composed of a first transition period T1 (time points t0 to t1) and a second transition period T2 (time points t1 to t2) subsequent thereto. Then, in the first transition period T1, smooth switching from the forward moving picture Ma (moving picture before switching) to the background moving picture Mc is performed, and in the second transition period T2, the moving picture moving back Mb (moving picture after switching) from the background moving picture Mc. ) Will be smoothly switched to.

  As shown in FIG. 20, the i-th frame F (i) of the forward moving image Ma, the (N−i + 1) th frame G (N−i + 1) of the return moving image Mb, and the i-th frame of the background moving image Mc. The frame H (i) is an image taken at the same point P (i) on the route R, and constitutes a frame corresponding to each other. The moving image reproduction unit 130 ′ performs a process of creating a composite image by combining such mutually corresponding frames during the transition periods T1 and T2, and displaying the composite image.

  In the example shown in FIG. 22, during the first transition period T1, the moving image reproduction unit 130 ′ displays the frame F (i) immediately before the transition period of the moving image before switching (outbound moving image Ma) and the background moving image Mc corresponding thereto. A fixed position image composed of a composite frame synthesized by blending the corresponding background frame H (i) is displayed as the first composite image B1 (i), and the background video Mc is displayed during the second transition period T2. The fixed background frame H (i) and the corresponding frame G (N−i + 1) corresponding to the corresponding background frame H (i) of the moving image after switching (inbound moving image Mb) are combined and synthesized. An example in which the position image is displayed as the second composite image B2 (i), and after the transition period, the process of starting forward playback from the corresponding frame G (N−i + 1) of the moving image after switching (return moving image Mb) is performed. It is.

  FIG. 23 is an equation showing the principle of image blending processing for creating a composite frame in the first transition period T1 and the second transition period T2 shown in FIG. During the first transition period T1, the moving image reproduction unit 130 ′ sets the ratio of the background image (frame H (i) in the example shown in FIG. 22) to α and the pre-switching image (in the example shown in FIG. 22). A first composite image B1 (i) is created and displayed by combining both images based on the equation shown in the upper part of FIG. 23, where the ratio of the frame F (i)) is (1-α), and As the first transition period T1 elapses, the value α is gradually increased from 0 to 1. On the other hand, during the second transition period T2, the ratio of the post-switching image (frame G (N−i + 1) in the example shown in FIG. 22) is β, and the background image (frame H (in the example shown in FIG. 22) The ratio of i)) is set to (1-β), and a second synthesized image B2 (i) is created and displayed by synthesizing both images based on the expression shown in the lower part of FIG. As the transition period T2 elapses, the value β is gradually increased from 0 to 1. Actually, for each of the pixel values of the corresponding pixels of both images (in the case of a color image, the pixel value of each color), the composition operation based on the above equations is performed, and the obtained value is used as a new pixel value. A collection of pixels may be a composite image.

  As described above, if the value α is gradually increased from 0 to 1 as the first transition period T1 elapses, the composite image B1 (i) becomes the frame image F (i) shown in FIG. Gradually changes to a frame image H (i) shown in FIG. As described above, since both images are images having a common background portion, when the value α is gradually increased from 0 to 1 during the first transition period T1, the background image is constant. The moving image display (so-called transition with a fade-out effect) in which the two people gradually disappear is performed.

  In the first transition period T1 of the time chart shown in FIG. 22, the hatched portions of the bands shown in the upper stage and the middle stage are triangular, indicating such a change in blend ratio over time. That is, the vertical width of the hatched portion indicates the blend ratio of each image, and the vertical width (corresponding to (1-α)) of the hatched portion of the forward moving image Ma shown in the upper row decreases with time, The vertical width (corresponding to α) of the hatched portion of the background video Mc shown increases with time. In addition, the symbol “static” written in the hatched portion of the figure is a still image in which the virtual user is stationary (here, the background image means a constant image and the moving object gradually disappears) Is a moving image).

  That is, in the example shown in the figure, the i-th frame image F (i) constituting the forward moving image Ma is displayed immediately before the inversion instruction input time t0, so the first transition period T1 following this is displayed. During the period, the first composite image B1 (i) obtained by blending the frame image F (i) and the i-th frame image H (i) of the background moving image Mc corresponding thereto is displayed. become.

  When the process of the first transition period T1 is completed in this way, the process of the second transition period T2 is subsequently executed. In the second transition period T2, as described above, the process of gradually increasing the value β from 0 to 1 with the progress is performed. Then, the composite image B2 (i) gradually changes from the frame image H (i) shown in FIG. 21 to the frame image G (N−i + 1) shown in FIG. Since both images are images having a common background portion, if the value β is gradually increased from 0 to 1 during the second transition period T2, the background image remains constant and a new A moving image display in which people gradually appear (so-called transition having a fade-in effect) is performed.

  In the second transition period T2 of the time chart shown in FIG. 22, the hatched portions of the bands shown in the middle and lower stages are triangular, which indicates such a change in blend ratio over time. That is, the vertical width of the hatched portion indicates the blend ratio of each image, and the vertical width (corresponding to (1-β)) of the hatched portion of the background video Mc shown in the middle row decreases with time, and in the lower row. The vertical width (corresponding to β) of the hatched portion of the return moving image Mb shown increases with time. Again, the symbol “static” described in the hatched portion of the figure means that a still image in which the virtual user is stationary is presented.

  After all, in the example shown in the figure, since the i-th frame image H (i) constituting the background moving image Mc is displayed at the end of the first transition period T1, the second transition period following this is displayed. During T2, a second composite image B2 (in which the frame image H (i) and the (N−i + 1) th frame image G (N−i + 1) of the return moving image Mb corresponding thereto are blended. i) will be displayed. Then, after the time point t2 when the second transition period T2 ends, the backward moving image Mb is sequentially reproduced from the frame image G (N−i + 1).

  Such a moving image switching process looks as follows when viewed from the user. That is, when the user gives a reverse instruction input while moving the route R in the forward direction, the position of the virtual user stops at the point P (i), and the frame image F (i) shown in FIG. 12 is shown in FIG. A gradually changing moving image (moving image in which two persons gradually disappear) is displayed in the frame image H (i), and then the frame image H (i) shown in FIG. 21 to the frame image shown in FIG. A gradually changing moving image (moving image in which a new person gradually appears) is displayed on G (N−i + 1), and when the frame image G (N−i + 1) is completely switched, the route R is moved in the backward direction. A video will be displayed. As a result, an effect of smoothly switching the moving object in the video can be obtained, and the unnaturalness of the moving object change at the time of switching from the forward moving image Ma to the return moving image Mb can be eliminated.

  Note that the example shown in FIG. 22 is an example of switching from the forward moving image Ma to the backward moving image Mb via the background moving image Mc (that is, an example in which a reverse instruction is input while moving in the forward direction on the route R shown in FIG. 20). However, of course, switching from the return movie Mb to the outbound movie Ma via the background movie Mc (that is, an example in which an inversion instruction is input while moving in the return direction on the route R shown in FIG. 20). Processing can be performed in exactly the same way.

  However, the switching mode of the moving image that mediates the background moving image Mc is not limited to the example shown in FIG. FIG. 24 is a time chart showing a modification of the switching mode shown in FIG. The feature of this modified example is that the moving image reproducing unit 130 ′ provides an intermediate transition period T3 between the first transition period T1 and the second transition period T2, and the corresponding background is displayed during the intermediate transition period T3. Only the frame H (i) is displayed.

  When viewed from the user, the two persons appearing in the frame image F (i) shown in FIG. 12 gradually disappear (first transition period T1), and then eventually disappear after the person disappears completely as shown in FIG. The frame image H (i) is displayed for a while (during the intermediate transition period T3). Subsequently, the frame image H (i) shown in FIG. 21 is changed to the frame image G (N−i + 1) shown in FIG. A moving image that gradually changes, that is, a moving image in which a new person gradually appears is displayed (second transition period T2), and when the frame image G (N−i + 1) is completely switched, the route R is moved in the return direction. A moving video will be displayed.

  In both the switching mode shown in FIG. 22 and the modification example shown in FIG. 24, the video presentation with the effect that the virtual user “stops for a while and starts moving in the opposite direction” at the point P (i) is presented. Although it is possible, as described in §3, the video to be presented when the inversion instruction input is given does not necessarily need to be limited to the staging effect of “stop for a while and then move in the reverse direction”. . In the following, two variations with different effects are described. These two variations basically correspond to the two variations described in §3.

  The first variation is that, when a reversal instruction input is given, “it proceeds for a while and starts moving in the reverse direction at a point P (i + d) that has passed the point P (i) at the time of reversal instruction input”. This is a method of giving a production effect. FIG. 25 is a time chart showing a moving image switching mode when such an effect is adopted.

  The feature of the first variation is that the moving image reproduction unit 130 ′ continuously reproduces the moving image before switching (forward moving image Ma) during the first transition period T1, and each frame of the moving image before switching. And a corresponding background frame of the background moving image Mc are displayed as a first combined image, and the switched moving image (return moving image Mb) is displayed during the second transition period T2. A moving image composed of a combined frame obtained by sequentially reproducing the frames corresponding to the final corresponding background frame in one transition period T1 and combining the frames of the moving image after switching and the corresponding background frame of the background moving image is used as the second combined image. Displayed, and after the end of the second transition period T2, the moving image after switching is continuously played forward.

  In other words, during the first transition period T1 (time t0 to t1) shown in FIG. 25, the video playback unit 130 ′ continues the forward video Ma (video before switching) in the same manner as the state before the transition period. Progressive playback is performed, and the background video Mc that has been played forward is blended. All the moving images are sequentially played back, and the moving images in a state of moving from the point P (i) to the point P (i + d) are blended. At this time, the blend ratio α is gradually increased from 0 to 1 so that the forward moving image Ma gradually changes to the background moving image Mc.

  Subsequently, during the second transition period T2 (time points t1 to t2), the background moving image Mc is played back in reverse, and the forward moving image Mb (moving image after switching) that has been played forward is blended. Thereby, the moving image in a state of returning from the point P (i + d) to the point P (i) is blended. At this time, the blend ratio β is gradually increased from 0 to 1 so that the background moving image Mc gradually changes to the return moving image Mb.

  After all, in the first transition period (time t0 to t1), the forward moving image Ma is reproduced in the order of frame images F (i), F (i + 1), F (i + 2),..., F (i + d), The background moving image Mc is sequentially played back as frame images H (i), H (i + 1), H (i + 2),..., H (i + d), and a blending process between them is performed. In the subsequent second transition period (time points t1 to t2), the background moving image Mc is reversely played back with the frame images H (i + d),..., H (i + 2), H (i + 1), and H (i). However, the backward moving image Mb is sequentially played back with the frame images G (N−i + 1−d),..., G (N−i−1), G (N−i), and G (N−i + 1). Blend processing is performed.

  Then, after the end of the second transition period T2 (after time t2), the backward moving image Mb (moving image after switching) may be continuously played forward. As described above, in the example shown here, the forward moving image Ma and the background moving image Mc have the same path moving direction at the time of shooting. Therefore, in the first transition period T1, both the forward moving image Ma and the background moving image Mc are sequentially reproduced. By doing so, a composite image can be obtained. On the other hand, the backward moving image Mb and the background moving image Mc are reverse in the moving direction of the path at the time of shooting. Therefore, in the second transition period T2, the backward moving image Mb is reproduced in the forward direction while the backward moving image Mb is reproduced in the forward direction. Thus, a composite image can be obtained.

  When moving images are presented in this way during the transition periods T1 and T2, the virtual user continues to move from the point P (i) even after the inversion instruction input is given at the time point t0, and at the time point t1. When the point P (i + d) is reached, the movement direction is reversed, the point P (i) is returned to the point P (i) at time t2, and the movement is continued as it is. In addition, during the first transition period T1, the blend ratio α is gradually increased from 0 to 1, so that the moving body included in the forward moving image Ma gradually disappears while moving. As a result, during the second transition period T2, the blend ratio β is gradually increased from 0 to 1, so that the new moving body gradually moves while moving in the background video Mc that does not include the moving body. The effects that appear in

  In practice, the moving image playback unit 130 ′ gradually decreases the playback speed of the moving image from the speed immediately before the transition period in the first transition period T1, and increases the playback speed in the second transition period T2. It is preferable to gradually increase it. Then, after the reverse instruction input is given at time t0, the moving speed of the virtual user gradually decreases, stops at time t1, and then gradually increases in the reverse direction. It is possible to provide an effect of moving while moving, and to express more natural movement for the virtual user.

  The second variation is a method of giving an effect of “invert immediately and start moving in the opposite direction” when an inversion instruction input is given. FIG. 26 is a time chart showing a moving image switching mode when such a production effect is adopted.

  The feature of the second variation is that the video playback unit 130 ′ plays back the pre-switching video (outbound video Ma) from the frame immediately before the transition period during the first transition period T1, and plays back the pre-switching video. A moving image composed of a combined frame obtained by combining each frame and each frame of the background moving image Mc corresponding to the frame is displayed as a first combined image, and a switched moving image (in the period of the second transition period T2) The moving image Mb) is a moving image composed of synthesized frames obtained by sequentially playing back the moving image Mb) from the frame corresponding to the final corresponding background frame of the first transition period T1 and combining the frames of the moving image after switching and the corresponding background frame of the background moving image Mc. The image is displayed as the second composite image, and after the second transition period T2, the moving image after switching is continuously played back in sequence.

  That is, during the first transition period T1 (time t0 to t1) shown in FIG. 26, the moving image playback unit 130 ′ plays back the forward moving image Ma (moving image before switching) from the frame immediately before the transition period, The background video Mc reproduced in the reverse direction is blended. All the moving images are played back in reverse, and the moving images in a reverse movement state from the point P (i) to the point P (id) are blended. At this time, the blend ratio α is gradually increased from 0 to 1 so that the forward moving image Ma gradually changes to the background moving image Mc.

  Subsequently, during the second transition period T2 (time points t1 to t2), the background moving image Mc is played back in reverse, and the forward moving image Mb (moving image after switching) that has been played forward is blended. Thereby, the moving image in a state of further moving from the point P (id) to the point P (i-2d) is blended. At this time, the blend ratio β is gradually increased from 0 to 1 so that the background moving image Mc gradually changes to the return moving image Mb.

  Eventually, in the first transition period (time t0 to t1), the forward moving image Ma is converted into frame images F (i), F (i-1), F (i-2),..., F (id). The background video Mc is played back in reverse with the frame images H (i), H (i-1), H (i-2),..., H (id). Blend processing is performed. Then, in the subsequent second transition period (time points t1 to t2), the backward moving image Mb is reproduced while the background moving image Mc is reversely played back with the frame images H (id),..., H (i-2d). The frame images G (N−i + 1 + d),..., G (N−i + 1 + 2d) are sequentially played back, and the blending process between them is performed.

  Then, after the end of the second transition period T2 (after time t2), the backward moving image Mb (moving image after switching) may be continuously played forward. As described above, also in the example shown here, the forward moving image Ma and the background moving image Mc have the same path movement direction at the time of shooting. Therefore, both the forward moving image Ma and the background moving image Mc are reversely fed during the first transition period T1. A composite image can be obtained by reproduction. On the other hand, the backward moving image Mb and the background moving image Mc are reverse in the moving direction of the path at the time of shooting. Therefore, in the second transition period T2, the backward moving image Mb is reproduced in the forward direction while the backward moving image Mb is reproduced in the forward direction. Thus, a composite image can be obtained.

  When the moving image presentation is performed in this way during the transition periods T1 and T2, the virtual user moves immediately from the point P (i) when the inversion instruction input is given at the time point t0. become. In addition, during the first transition period T1, the blend ratio α is gradually increased from 0 to 1, so that the moving body included in the forward moving image Ma gradually disappears while moving. As a result, during the second transition period T2, the blend ratio β is gradually increased from 0 to 1, so that the new moving body gradually moves while moving in the background video Mc that does not include the moving body. The effects that appear in

  As mentioned above, with reference to FIG. 22, FIG. 24, FIG. 25, and FIG. 26, several presentation methods of animation presentation performed during the transition periods T1 and T2 have been described. There is. This will be briefly described below.

  First, when the method shown in FIG. 22 or FIG. 24 is adopted, since the movement of the virtual user immediately stops at the position P (i) at the time point t0 when the inversion instruction input is given, the quick response of the movement of the virtual user to the inversion instruction The advantage of sex is obtained. However, during the transition periods T1 and T2, the movement of the virtual user remains stopped, and there is a demerit that the moving image indicating the state of moving on the route R is temporarily interrupted.

  In addition, when the method shown in FIG. 25 is adopted, the virtual user is in a state of continuing movement, and there is an advantage that the moving state moving image is not interrupted. However, even after the reverse instruction input is given, the virtual user continues to move for a while, and a slight delay occurs until the moving direction is reversed, which causes a disadvantage that the responsiveness of the virtual user to the operation input is lowered.

  On the other hand, when the method shown in FIG. 26 is adopted, since the virtual user immediately reverses and moves, there is a merit that the moving video is not interrupted, and there is also a merit that quick response of the virtual user's movement to the reversal instruction. can get. However, as compared with the method shown in FIG. 25, there is a demerit that the unnatural motion of the moving object becomes remarkable. For example, in FIG. 26, since the moving image displayed immediately after time t0 is a composite image close to α = 0, the moving object (two persons shown in FIG. 12) shown in the outbound moving image Ma is clearly displayed. Is done. However, during the first transition period T1, the forward moving picture Ma is played back in reverse, so that this clear moving object is shown in reverse, and the unnaturalness of the movement becomes noticeable.

  On the other hand, if the method shown in FIG. 25 is adopted, the forward moving image Ma and the backward moving image Mb including the moving object are always reproduced in the forward direction, so that the moving object does not move unnaturally. Since the background moving image Mc does not include moving objects, it does not become unnatural even if it is played backward. Therefore, in each method described above, the moving body moves unnaturally only in the first transition period T1 in which the forward moving image Ma is played back in the reverse direction in the method shown in FIG.

  As mentioned above, although several presentation methods of the video presentation performed during transition period T1, T2 were illustrated, of course, the presentation method which performs the video switching which mediated the background video is not limited to these methods. Various other production methods can be employed. For example, the method of providing the intermediate transition period T3 employed in the method shown in FIG. 24 can be applied to the method shown in FIG. 25 and the method shown in FIG.

<<< §5. Various modifications >>
As mentioned above, although the moving image presentation apparatus based on this invention was demonstrated based on some embodiment, here, some modifications are hung up.

(1) Method of creating an omnidirectional image In §1, a region having an elevation angle equal to or smaller than a predetermined reference value is cut out from a distorted circular image obtained by photographing using a photographing device as shown in FIG. Although an example in which an omnidirectional image (panoramic image) is obtained by performing the above is shown, of course, the apparatus used to create the omnidirectional image is not limited to the photographing apparatus shown in FIG. For example, in the photographing apparatus shown in FIG. 4, a distorted circular image is formed on the imaging surface of the video camera using the fisheye lens 10, but an omnidirectional mirror or the like may be used instead of the fisheye lens 10.

  Moreover, in §1, although the example which produces an omnidirectional image based on the real image image which image | photographed the real facility was shown, the virtual facility is created on a computer and the three-dimensional which shows the virtual path | route in this virtual facility is shown. An image created based on the CG image may be used as the omnidirectional image. In this case, in the omnidirectional image storage unit 110, a rectangular panoramic image created based on a three-dimensional CG image indicating a virtual route is stored as an omnidirectional image.

(2) Adjustment of elevation angle In the embodiments described so far, only the azimuth angle φ (i) is used as a parameter indicating the line-of-sight direction, but it is also possible to add a parameter called elevation angle Ψ (i). . In this case, the line-of-sight vector E (i) is determined by the azimuth angle φ (i) and the elevation angle Ψ (i). As described above, when the elevation angle ψ (i) is used as a parameter, the cut frame by the moving image reproducing units 130 and 130 ′ is determined according to both the azimuth angle φ (i) and the elevation angle ψ (i). Therefore, for example, in the case of the example shown in FIG. 8, the vertical dimension of the cutout frame (outer frame of the field image Q (P (i), φ (i))) indicated by a thick line is set shorter, and this cutout frame Is determined based on the azimuth angle φ (i), and the vertical position is determined based on the elevation angle Ψ (i). Further, the controller 50 shown in FIG. 9 may be provided with buttons for giving an instruction to change the elevation angle Ψ (i), such as an upward button or a downward button.

(3) Correspondence relationship between frames constituting each moving image In the example shown in FIG. 20, each of the forward moving image Ma, the returning moving image Mb, and the background moving image Mc is composed of N frame images. As indicated by the facing arrows, the individual frame images have a corresponding relationship. If a moving image is shot using a shooting device that lays a rail on a route to be imaged and moves on the rail at a constant speed, as shown in the example in FIG. A moving image in which the three frames, ie, the frame forming the moving image Mb and the frame forming the background moving image Mc, completely correspond to each other can be obtained. However, when the moving speed of the photographing apparatus is different, a complete correspondence between the frames is obtained. It becomes impossible.

  However, even in such a case, it is possible to obtain a moving image in which the three frames completely correspond by the adjustment process after shooting. For example, if the forward moving image Ma is composed of 1000 frames and the backward moving image Mb is composed of 500 frames due to a difference in moving speed at the time of shooting, the processing of thinning out every other frame of the outbound moving image Ma is performed. These are all moving images composed of 500 frames, and the frames correspond one-to-one. Alternatively, if a process of interpolating a new frame between the frames of the return moving image Mb is performed, all of the moving images are composed of 1000 frames, and the frames correspond one-to-one.

  Of course, even when the forward moving image Ma is composed of 1000 frames and the backward moving image Mb is composed of 980 frames, the frames can be made to correspond one-to-one by performing appropriate interpolation processing. Is possible.

  For practical use, it is preferable to move the shooting device on the rail to shoot each moving image. However, when such a shooting environment cannot be obtained, for example, while driving a car equipped with an in-vehicle camera, You can also prepare each video by taking a picture or taking pictures with your own video camera while the photographer walks. In such a shooting environment, the correspondence between frames between the moving images is not accurate, but as described above, it is possible to prepare a moving image having the correct correspondence between frames by a subsequent correction process. .

  Note that if the rail is not used during shooting, the shooting location will also shift slightly, and the position of the background shown in each movie will shift slightly. There is no sense of incongruity.

(4) Acceptance / rejection of omnidirectional image In the embodiments described so far, the individual frame images constituting the moving image are configured by omnidirectional images (panoramic images) as shown in FIG. An example is shown in which a portion corresponding to the angle φ) is cut out and displayed as a field image Q (P (i), φ (i)). However, in carrying out the present invention, it is not always necessary to prepare each moving image as an omnidirectional image, and a moving image having a normal angle of view shot by a normal video camera may be used. In this case, the individual frame images that make up the movie are also images with a normal angle of view, so the direction of the line of sight cannot be freely specified, but the movie can be played with the direction of the line of sight fixed. is there. In this case, it is not necessary to cut out a part of the field image Q (P (i), φ (i)).

  Of course, instead of the omnidirectional image, for example, a moving image composed of a frame of a semi-directional image having an angle of view of 180 ° can be used. In this case, the specified range of the line-of-sight direction (azimuth angle φ) is narrowed, but other than that, substantially the same effect as the embodiments described so far can be obtained.

(5) Omission of Route Information Storage Unit 120 The embodiment described so far is an example in which a moving state of a visit route having a plurality of routes is presented as a moving image as shown in FIG. Although the route is switched by referring to the route information RT stored in 120, the present invention is of course applicable to the case where a moving image is presented only for a single route. In this case, the route information RT is not necessary, and therefore it is not necessary to provide the route information storage unit 120.

(6) Configuration of Controller The controller 50 shown in FIG. 9 is an example of a device that constitutes a part of the instruction input unit 150. In practice, various other types of devices are used as the controller. It is possible to use. In particular, the form and arrangement of the operation buttons and the like can be freely set by design. In §1, the role of the buttons B6 to B9 is not described, but these buttons have various functions such as a function for setting the moving speed and a function for specifying the elevation angle ψ (i) described above. Functions can be assigned.

  A joystick can be used instead of the button. For example, if one joystick is tilted forward, a forward instruction is given; if it is tilted backward, a backward instruction is given; if it is tilted leftward, a leftward instruction is given; Can be used. Furthermore, it can be used such that the moving speed is changed based on the inclination angle.

  In the above-described embodiment, when the forward button B1 is pressed, the virtual user continues to move in the first direction (for example, the forward direction) even when the button is released, and when the backward button B2 is pressed, the movement is performed. The direction is reversed, and even if the button is released, it continues to move in the second direction (for example, the return direction), but what instructions are given by each button operation? It is possible to negotiate freely according to the intention of the designer. For example, it is possible to perform a handling in which the virtual user stops only when the forward button B1 or the backward button B2 is continuously pressed and moves in the first direction or the second direction and the finger is released from the button. is there.

  In short, the controller 50 may be able to input an instruction input for designating the moving direction of the virtual user, in other words, an inversion instruction for inverting the moving direction by some method. The inversion instruction input is not necessarily performed while the virtual user is moving, and can be performed even when the virtual user is stopped. For example, when the virtual user presses the stop button B5 to stop while moving in the forward direction, and then instructs to move in the backward direction by pressing the reverse button B2, the operation of pressing the reverse button B2 is a reverse instruction input. It will be an operation of.

(7) Coordination between the line-of-sight direction and the direction of travel In the embodiments described so far, the moving direction and the line-of-sight direction of the virtual user can be specified independently. For example, the controller 50 shown in FIG. In the example used, the moving direction is designated by operating the forward button B1 and the backward button B2, and the line-of-sight direction (azimuth angle φ) is designated by operating the left button B3 or the right button B4. In other words, the line-of-sight direction can be changed regardless of the switching of the moving direction, and the user can give an instruction to freely change the line-of-sight direction even during the transition period.

  However, normally, as a human behavior, it is natural to make the line of sight substantially coincide with the moving direction (that is, it is common to move the line of sight toward the moving direction). Therefore, it is possible to adopt an operation in which the moving direction is reversed according to the line-of-sight direction.

  For example, when the angle formed by the line-of-sight direction with respect to the moving direction exceeds 90 ° or exceeds 145 °, a critical condition is set in advance, and when the critical condition is exceeded, the moving direction is reversed. By doing so, when the virtual user looks back while moving, the moving direction is reversed, and the user moves in the opposite direction (turned back). If such an operation is performed, it is possible to provide a virtual experience with a sense of reality that is closer to human behavior.

  Specifically, when the controller 50 shown in FIG. 9 is used as the instruction input unit 150, the movement direction setting unit 140 determines whether or not the instruction input unit 150 uses the left button B3 or the right button B4. The moving direction may be switched when the angle of the line-of-sight direction with respect to the moving direction becomes equal to or greater than a predetermined reference.

(8) Switching during route transfer In §3 and §4, a transition period is set at the time of video switching when the virtual user reverses the moving direction, and the blended image is displayed during this transition period. Although the method of reducing the unnaturalness of the moving body change has been described, this method can also be used when moving images are switched by route transfer.

  For example, as shown in FIG. 2, let us consider a case where a moving state of a visit route having a plurality of routes R1 to R6 is presented as a moving image. Now, assume that the virtual user has moved the route R2 from the node N2 to N3, changed to the route R3, and moved from the node N3 to N4. In this case, a moving image taken along the route R3 is displayed after the moving image taken along the route R2, but it is usually difficult to continuously shoot while turning the corner (node N3). Therefore, the moving image of the route R2 and the moving image of the route R3 are generally shot as different takes, and discontinuity occurs in the moving object. This is the same when passing through a branch point such as the node N2.

  Even if continuous shooting is performed as one take, the distortion characteristics of the distorted circular image obtained by the omnidirectional imaging device differ depending on the individual cutout positions, so that the moving image shot along the path R2 and the path R3 In the captured moving image, the position and size of the same subject will deviate. In other words, when switching from the route R2 to the route R3 through the node N3, discontinuity may occur not only on the moving object but also on the stationary background. For example, when changing only the direction of travel without changing the direction of the line of sight on the map when passing through the node N3, in theory, a continuous background image should be displayed. There will be a sense of incongruity due to the deviation.

  Therefore, as described in §3 and §4 when switching videos by route transfer, if a transition period is set and a blended composite image is displayed during this transition period, unnaturalness at the time of switching videos Can be reduced. In the case of the above example, when the user gives an instruction to move from the route R2 to the route R3, it is only necessary to stop at the node N3 and present a composite image that transitions from the image of the route R2 to the image of the route R3. . Alternatively, a period during which the node N3 is moved slightly before (position on the route R2) to a position slightly past the node N3 (position on the route R3) is set as a transition period and stopped at the node N3. Instead, the composite image that transitions from the image of the route R2 to the image of the route R3 may be presented while continuously moving.

10: Fisheye lens 20: Video camera 30: Data processing unit 40: Cart 50: Controller 100, 100 ': Movie presentation device 110, 110': Movie storage unit 120: Route information storage unit 130, 130 ': Movie reproduction unit 140: Movement direction setting unit 150: Instruction input unit 200: Display device A: Node B as end point of route: Nodes B1 to B9 as end points of route: Button B (i): Composite image B1 (i): First composition Image B2 (i): Second composite image E (i): Eye vector F (1), F (2), F (i-1), F (i), F (i + 1), F (N-1) ), F (N): frame image of forward moving image / field-of-view image G (1), G (2), G (N−i), G (N−i + 1), G (N−i + 2) cut out therefrom , G (N-1), G (N): Frame of the return video Image / view image H (1), H (2), H (i-1), H (i), H (i + 1), H (N-1), H (N) cut out from the background image Frame image / field-of-view image clipped from Ma: outbound video M1a: outbound video of route R1 Mb: inbound movie M1b: inbound movie of route R1 Mc: background images N1 to N6: node O: top vertex P (1) , P (2), P (i-1), P (i), P (i + 1), P (N-1), P (N): point Q (P (i), φ (i) on the route ): Field image R, R1 to R6: Route RT: Route information T: Transition period T1: First transition period T2: Second transition period T3: Intermediate transition period t: Time t0: Inversion instruction input time t1 to t3 : Specific time point α: Blend ratio of composite image β: Blend ratio of composite image Δ: Cutting angle φ (i): Gaze direction (azimuth angle)

Claims (20)

A video presentation device that presents a field of view as viewed from a moving viewpoint,
A video storage unit that stores a video shot while moving along a path connecting the first node and the second node;
An instruction input unit for inputting a user instruction including an inversion instruction for inverting the moving direction;
Based on the input user instruction, the forward direction from the first node to the second node and the return direction from the second node to the first node as moving directions on the route And a process of switching the movement direction from the forward direction to the backward direction or from the backward direction to the forward direction based on the input of the reverse instruction. A moving direction setting section;
A video playback unit that plays back the video stored in the video storage unit and displays the video on a screen of a display device;
With
In the moving image storage unit, an outbound moving image including a moving object on the subject taken while moving on the route in the outward direction, and a returning route including a moving object on the subject captured while moving on the route in the returning direction. Contains videos and
When the forward direction is set in the movement direction setting unit, the video playback unit performs forward playback with the forward video as a playback target, and the return direction is set in the movement direction setting unit. In order to play the forward video as a playback target,
The moving image reproduction unit further sets a predetermined period after the inversion instruction is input as a transition period, and during this transition period, a pre-switching image constituting a pre-switching moving image that is a reproduction target according to a moving direction before switching The ratio of the post-switching image is α and the post-switching image is composed of a post-switching video that is a playback target according to the moving direction after switching and is captured at the same position as the pre-switching image. Create a composite image by blending both images with a ratio of (1-α), and display the blended composite image by gradually increasing the value α from 0 to 1 over the transition period,
The forward moving image is composed of a total of N frame images from the first frame imaged at the first node position to the Nth frame imaged at the second node position. ,
The return moving image is composed of a total of N frame images from the first frame photographed at the second node position to the Nth frame photographed at the first node position. ,
The i-th frame F (i) of the forward moving image and the (N−i + 1) th frame G (N−i + 1) of the return moving image are images taken at the same point on the route. Configure frames that correspond to each other,
A moving picture presentation apparatus, wherein the moving picture reproducing unit creates a synthesized image by synthesizing frames corresponding to each other during the transition period, and displays the synthesized image. The video presentation device according to claim 1 ,
The video playback unit
During the transition period, a fixed position image consisting of a composite frame obtained by combining the frame immediately before the transition period of the pre-switching video and the corresponding frame of the post-switching video corresponding thereto is displayed.
A moving picture presenting apparatus characterized by starting progressive playback from the corresponding frame of the moving picture after switching after the end of the transition period. The video presentation device according to claim 1 ,
The video playback section
In the first half of the transition period, the video before switching is continuously played forward, and in the second half of the transition period, the video before switching is played backward from the last frame of the first half,
During the transition period, display a video image composed of synthesized frames obtained by synthesizing each frame of the video before switching and each frame of the video after switching corresponding to this,
A moving picture presenting apparatus characterized in that, after the transition period ends, the moving picture after switching is continuously played in order. The video presentation device according to claim 3 ,
A video in which the video playback unit gradually decreases the playback speed from the speed immediately before the transition period in the first half of the transition period and gradually increases the playback speed in the second half of the transition period Presentation device. The video presentation device according to claim 1 ,
The video playback unit
During the transition period, the pre-switching video is played back in reverse from the frame immediately before the transition period, and a video image composed of the combined frames of the pre-switching video frames and the corresponding post-switching video frames is displayed. And
A moving picture presenting apparatus characterized in that, after the transition period ends, the moving picture after switching is continuously played in order. A video presentation device that presents a field of view as viewed from a moving viewpoint,
  A video storage unit that stores a video shot while moving along a path connecting the first node and the second node;
  An instruction input unit for inputting a user instruction including an inversion instruction for inverting the moving direction;
  Based on the input user instruction, the forward direction from the first node to the second node and the return direction from the second node to the first node as moving directions on the route And a process of switching the movement direction from the forward direction to the backward direction or from the backward direction to the forward direction based on the input of the reverse instruction. A moving direction setting section;
  A video playback unit that plays back the video stored in the video storage unit and displays the video on a screen of a display device;
  With
  In the moving image storage unit, an outbound moving image including a moving object on the subject taken while moving on the route in the outward direction, and a returning route including a moving object on the subject captured while moving on the route in the returning direction. A moving image and a background moving image that is photographed while moving in the forward direction or the backward direction on the route, and not including a moving object in the subject, are stored;
  When the forward direction is set in the movement direction setting unit, the video playback unit performs forward playback with the forward video as a playback target, and the return direction is set in the movement direction setting unit. In order to play the forward video as a playback target,
  The moving image reproduction unit further sets a predetermined period after the inversion instruction is input as a transition period, and during this transition period, a pre-switching image constituting a pre-switching moving image that is a reproduction target according to a moving direction before switching , A first synthesized image obtained by blending a background image that forms the background moving image and is captured at the same position as the image before switching, a background image that configures the background moving image, and a moving direction after switching A second composite image obtained by blending a post-switching image that is a playback target and that is captured at the same position as the background image is displayed in the order of the first composite image and the second composite image. An apparatus for presenting a moving image. The video presentation device according to claim 6 ,
The video playback section
Within the transition period, set a first transition period and a second transition period following this,
The ratio of the background image is α, the ratio of the pre-switching image is (1−α), a first composite image is created by combining both images, and this is displayed within the first transition period; and , Gradually increase the value α from 0 to 1 over the course of the first transition period,
The ratio of the post-switching image is β, the ratio of the background image is (1-β), and a second composite image is created by combining both images, and this is displayed within the second transition period, and The moving picture presenting apparatus is characterized in that the value β is gradually increased from 0 to 1 as the second transition period elapses. The video presentation device according to claim 7,
A total of N frame images from the first frame shot at the first node position to the Nth frame shot at the second node position of the forward moving image stored in the moving image storage unit Consists of
A total of N frame images from the first frame captured at the second node position to the Nth frame captured at the first node position for the return path video stored in the video storage unit. Consists of
A total of N frame images from the first frame shot at the first node position to the Nth frame shot at the second node position of the background moving image stored in the moving image storage unit Consists of
The i-th frame F (i) of the forward moving image , the (N−i + 1) th frame G (N−i + 1) of the backward moving image, and the i-th frame H (i) of the background image It is an image taken at the same point on the top, constituting a mutually corresponding frame,
A moving picture presentation apparatus in which a moving picture reproducing unit creates a synthesized image by synthesizing frames corresponding to each other during a transition period and displays the synthesized image. The video presentation device according to claim 8 ,
The video playback section
During the first transition period, a fixed position image composed of a composite frame obtained by combining the frame immediately before the transition period of the pre-switching video and the corresponding background frame of the background video corresponding thereto is displayed as the first composite image,
During the second transition period, a fixed position image composed of a synthesized frame obtained by synthesizing the corresponding background frame of the background moving image and the corresponding frame of the switched moving image corresponding to the corresponding background frame is displayed as the second synthesized image. And
A moving picture presenting apparatus characterized by starting progressive playback from the corresponding frame of the moving picture after switching after the end of the transition period. The video presentation device according to claim 9 ,
A moving picture presentation apparatus, wherein the moving picture reproduction unit provides an intermediate transition period between the first transition period and the second transition period, and displays only the corresponding background frame during the intermediate transition period. The video presentation device according to claim 8 ,
The video playback section
During the first transition period, the pre-switching video is continuously played forward and a video image composed of a composite frame obtained by synthesizing each frame of the pre-switching video and the corresponding background frame of the background video is used as the first composite image. Display
During the second transition period, the post-switching video is sequentially played back from the frame corresponding to the final corresponding background frame of the first transition period, and each frame of the post-switching video and the corresponding background frame of the background video are combined. A moving image composed of the synthesized frames is displayed as a second synthesized image,
A moving picture presenting apparatus characterized in that, after the end of the second transition period, the moving picture after switching is continuously played forward. The moving image presentation device according to claim 11 ,
A moving image characterized in that the moving image playback unit gradually decreases the playback speed from the speed immediately before the transition period during the first transition period, and gradually increases the playback speed during the second transition period. Presentation device. The video presentation device according to claim 8 ,
The video playback section
During the first transition period, the moving image before switching is played back in reverse from the frame immediately before the transition period, and the moving image is composed of synthesized frames obtained by combining each frame of the moving image before switching and each frame of the background moving image corresponding thereto. As the first composite image,
During the second transition period, the post-switching video is sequentially played back from the frame corresponding to the final corresponding background frame of the first transition period, and each frame of the post-switching video and the corresponding background frame of the background video are combined. A moving image composed of the synthesized frames is displayed as a second synthesized image,
A moving picture presenting apparatus characterized in that, after the transition period ends, the moving picture after switching is continuously played in order. In the moving image presentation device according to any one of claims 1 to 13 ,
Each moving image stored in the moving image storage unit is composed of an image that is recorded in units of frames, which is an omnidirectional image having a 360 ° field of view, which is taken using an omnidirectional camera while moving along a route. ,
The instruction input unit has a function of inputting an instruction indicating a specific gaze direction,
A moving picture presenting apparatus, wherein the moving picture reproducing unit cuts out a visual field image corresponding to the line-of-sight direction from each frame constituting the moving picture and displays the image on a display screen. The video presentation device according to claim 14 ,
The instruction input unit includes a controller having a forward button, a backward button, a left button, and a right button,
The movement direction setting unit sets the movement direction to the first direction based on the operation instruction for the forward button, sets the movement direction to the second direction based on the operation instruction for the backward button,
A video presentation device, wherein the video playback unit changes the line-of-sight direction to the left based on the operation instruction for the left button, and changes the line-of-sight direction to the right based on the operation instruction for the right button. The moving picture presentation device according to claim 15 ,
The moving direction setting unit switches the moving direction when the angle of the line-of-sight direction with respect to the moving direction is equal to or greater than a predetermined reference based on an operation instruction for the left button or a right button. Video presentation device. In the moving image presentation apparatus in any one of Claims 14-16 ,
From the distorted circular image obtained by photographing the hemispherical field located above the predetermined horizontal plane using the omnidirectional camera equipped with the fisheye lens or the omnidirectional mirror, the moving image storage unit has an elevation angle of a predetermined reference value or less. A moving picture presenting apparatus for storing a moving picture in which a rectangular panoramic image obtained by cutting out an area and correcting the distortion is stored as an omnidirectional image. In the moving image presentation apparatus in any one of Claims 14-16 ,
An omnidirectional image storage unit stores a moving image in which a rectangular panoramic image created based on a three-dimensional CG image indicating a virtual route is an omnidirectional image. In the moving image presentation device according to any one of claims 1 to 18 ,
The video storage unit stores videos for a plurality of paths connecting three or more nodes, respectively.
A route information storage unit for storing route information indicating a connection relation of each route via each node;
The instruction input unit has a function of inputting a user's route transfer instruction to move from the first route to the second route via the node;
A function of performing a video playback for the second route by referring to the route information when the video playback unit receives the route transfer instruction, and continues the video playback for the first route. A moving picture presentation apparatus comprising: A program causing a computer to function as the moving picture presentation device according to claim 1 .