JP7716232B2 - Image processing device, control method thereof, and program - Google Patents

Description

The present invention relates to an image processing device, a control method thereof, and a program.

Recently, attention has been focused on technology that uses multiple cameras installed in different positions to capture synchronized images from multiple viewpoints and generate virtual viewpoint images using the multi-view images obtained through this capture. For example, Patent Document 1 discloses technology in which multiple cameras are positioned to surround a subject and images of the subject captured by these cameras are used to generate an image from any viewpoint. This technology for generating virtual viewpoint images from multi-view images allows viewers to view highlight scenes of soccer or basketball games from various angles, for example, providing a greater sense of realism to viewers than with regular images. Furthermore, when filming music events, live streaming, or music videos, it is possible to create images of artists from various angles.

Japanese Patent Application Laid-Open No. 2008-015756

When filming music events, live broadcasts, music videos, etc., it is common to switch between multiple images captured simultaneously from multiple cameras. For example, a first camera is used to capture what is known as a "long shot" ranging from a long shot including the subject's surroundings to a bust shot of the subject. A second camera is used to capture what is known as a "close-up shot" ranging from a bust shot to a close-up of the subject. By switching between the images captured by the first and second cameras, it is possible to generate images that correspond to various subject sizes. In this case, for example, the first camera could be used as a virtual viewpoint (referred to herein as a virtual camera) that generates the virtual viewpoint image described above, and the second camera could be used as a physical camera (referred to herein as a real camera) that captures images that are not used for the virtual viewpoint image.

Generally, video switching devices that switch between two videos and output one video instantly switch from one video to another, resulting in a significant change in the video when switching. This can cause viewers to feel uncomfortable. One known method for reducing the sense of discomfort felt by viewers when switching between videos is to add visual effects such as fade-in and fade-out when switching between videos. However, this still means that the video from the first camera and the video from the second camera are used when switching, and it is impossible to avoid unnatural changes in the video caused by switching between videos.

One aspect of the present invention provides technology that reduces unnatural changes in images when switching between two images for output.

An image processing device according to one aspect of the present invention has the following configuration:
an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
The first generation means generates a virtual viewpoint for the period based on information about the first viewpoint, information about the second viewpoint, and a ratio of the elapsed time from the start of the period to the total time of the period.
An image processing device according to another aspect of the present invention has the following configuration:
an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
a setting means for setting a ratio in accordance with a user operation received during the period;
and
The first generating means generates a virtual viewpoint for the period based on the information about the first viewpoint, the information about the second viewpoint, and the ratio set by the setting means.
An image processing apparatus according to still another aspect of the present invention has the following configuration:
an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
The first generation means generates a virtual viewpoint at each time during the period based on information about the first viewpoint at each time and information about the second viewpoint at each time.

This invention reduces unnatural changes in the images when switching between two images for output.

FIG. 1 is a diagram showing an example of the overall configuration of an image processing system according to a first embodiment. 10 is a flowchart showing a distribution video determination process according to the first embodiment. FIG. 10 is a diagram showing a timeline for switching from a virtual viewpoint image to a real camera image. FIG. 4 is a diagram showing an example of generation of virtual camera information according to the first embodiment. FIG. 4 is a diagram showing an example of generation of virtual camera information according to the first embodiment. FIG. 4 is a diagram showing an example of generation of virtual camera information according to the first embodiment. 10A to 10C are views showing another example of generation of virtual camera information according to the first embodiment. FIG. 3 is a diagram illustrating an operation unit for specifying a switching ratio according to the first embodiment. FIG. 10 is a diagram showing an example of the overall configuration of an image processing system according to a second embodiment. 10 is a flowchart showing a distribution video determination process according to the second embodiment. FIG. 10 is a diagram showing an example of generation of virtual camera information according to the second embodiment. FIG. 10 is a diagram showing an example of generation of virtual camera information according to the second embodiment. FIG. 10 is a diagram showing an example of generation of virtual camera information according to the second embodiment. FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claimed invention. While the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

First Embodiment
An image processing device that changes an output image from a first viewpoint image to a second viewpoint image will be described below. In the first embodiment, the first viewpoint is the viewpoint of a virtual imaging device for generating a virtual viewpoint image from multiple images captured by multiple imaging devices, and the second viewpoint is the viewpoint of a physical imaging device that captures the image. That is, the first viewpoint image is a virtual viewpoint image, and the second viewpoint image is an image captured by a real camera (hereinafter, referred to as a real camera image). An example will be described below in which, in an image processing system that generates a virtual viewpoint image, a new virtual viewpoint image is generated that smoothly connects the two images when switching from the virtual viewpoint image to the real camera image.

FIG. 1 is a block diagram showing an example configuration of an image processing system for generating a virtual viewpoint video according to the first embodiment. The camera group 101 is composed of multiple imaging devices (hereinafter referred to as cameras) that capture multi-viewpoint images of the shooting range in order to generate a virtual viewpoint video. Each of the multiple cameras has an internal imaging element and a lens in front of it. The multiple cameras are fixedly installed around the shooting range, facing the shooting range. A camera control unit 102 controls each camera in the camera group 101. A camera control unit 102 is provided for each camera in the camera group 101, and is connected to each camera in the camera group 101 via a camera control cable and a camera image output cable. The multiple camera control units 102 are connected to each other via a local network cable or the like, for example, in a daisy chain, and transmit images from the camera group 101 to an image processing device 103 connected downstream. The network configuration for connecting the multiple camera control units 102 is not limited to a daisy chain, and may be a star-type network configuration in which each camera control unit is connected to an image processing device.

The image processing device 103 has the function of generating and outputting a virtual viewpoint image, which is an image from a virtual viewpoint, based on the images (multi-viewpoint images) acquired by the camera group 101. The functional configuration of the image processing device 103 is described below.

The image acquisition unit 104 acquires captured images (multi-view images) acquired by the camera group 101 from the camera control unit 102. The image acquisition unit 104 also acquires, in advance, captured images obtained by the camera group 101 capturing an image of a capturing area that does not include the subject (foreground) as background images and stores the images in the background image storage unit 105. The separation unit 106 separates the subject (foreground) included in the captured image from the captured image of the capturing area. The separation unit 106 performs separation by, for example, background subtraction. More specifically, the separation unit 106 compares the captured image with a background image acquired in advance and stored in the background image storage unit 105, and identifies the difference as the foreground, which is the subject, thereby separating the foreground from the background. The separation unit 106 stores the separated image including the foreground (hereinafter referred to as the foreground image) in the foreground image storage unit 107. Note that the method of separating the foreground and background used by the separation unit 106 is not limited to the separation method using background subtraction described above, and any well-known separation method, such as a separation method using a distance image, can be used.

The foreground image storage unit 107 stores multiple foreground images (multiple foreground images acquired from multiple cameras (i.e., multiple viewpoints)) separated by the separation unit 106 from images captured by the group of cameras 101 installed around the shooting area. The 3D model generation unit 108 acquires the foreground images from the foreground image storage unit 107 and generates a 3D model of the foreground. The 3D model generation unit 108 generates a 3D model of the foreground using, for example, a volume intersection method from the foreground images acquired from multiple viewpoints. The generated foreground 3D model and its position information are stored in the 3D model storage unit 109.

The virtual camera information generation unit 110 generates virtual camera information in response to user operations that specify the position of the virtual viewpoint, the direction of the line of sight, etc., received from a user interface such as a joystick or various input units. The virtual camera information includes information on the position, attitude (line of sight direction), and angle of view (focal length) of the virtual viewpoint (hereinafter also referred to as the virtual camera) of the virtual viewpoint video, as well as time information. In other words, the function of the virtual camera information generation unit 110 is to generate time-specific information about the virtual viewpoint required to generate the virtual viewpoint video in response to operation of the virtual camera by the operator using an input unit such as a joystick.

The virtual viewpoint video generation unit 111 generates a virtual viewpoint video based on the time, position, attitude, and angle of view of the virtual camera represented by the virtual camera information generated by the virtual camera information generation unit 110 or the automatic virtual camera information generation unit 117 (described below). For example, to generate a virtual viewpoint video, the virtual viewpoint video generation unit 111 acquires a foreground image for that time from the foreground image storage unit 107 and a foreground 3D model for that time from the 3D model storage unit 109, and generates a foreground image corresponding to the position, attitude, and angle of view of the virtual camera. The virtual viewpoint video generation unit 111 also acquires a background image stored in the background image storage unit 105, acquires a prepared background 3D model, and generates a background image corresponding to the position, attitude, and angle of view of the virtual camera. The virtual viewpoint video generation unit 111 combines the generated foreground image and background image and outputs the result as a virtual viewpoint video. The virtual viewpoint video is provided to the video switching unit 115 and becomes one of the candidate images to be output as the final video.

The real camera 112 is a camera that can capture images within the shooting range of the virtual camera independently of the camera group 101. The real camera 112 is used not to capture images necessary for the virtual viewpoint video, but to capture close-up images of a subject. Note that in this embodiment, the term "real camera" is used to distinguish it from the camera group 101 that captures images necessary for the virtual viewpoint video, and the virtual camera that does not actually exist but is virtually placed in a position as if capturing the virtual viewpoint video. The captured image obtained by the real camera 112 is provided to the image switching unit 115, which will be described later, and becomes one of the image candidates to be output as the final image.

The real camera information acquisition unit 113 acquires information including the position, orientation (line of sight direction), and angle of view (focal length) of the real camera 112. The real camera information acquisition unit 113 estimates the position and orientation of the real camera 112, for example, from the position at which a marker placed within the range in which the real camera 112 moves is reflected in an image captured by the real camera 112. However, this is not limited to this, and for example, an image of the marker may be obtained by connecting a camera that captures markers for position estimation to the real camera 112 separately from the real camera. Alternatively, without placing a marker, the position and orientation of the real camera 112 may be estimated by identifying characteristic locations whose positions are known from images captured by the real camera 112.

The video determination unit 114 selects and determines an output video from multiple output video candidates. The video determination unit 114 has input units such as a switch for selecting video output and a fader for adjusting volume, etc. The video determination unit 114 can also add various video effects (transitions) when switching between videos. For example, it can determine to output a virtual viewpoint video, switch from a virtual viewpoint video to a real camera video, or determine to add video effects such as fade-in or fade-out when switching. The video determination unit 114 transmits channel information specifying the selected video and information indicating the video effects to be executed when switching to the video switching unit 115. The video switching unit 115 selects a video from the video candidates based on the information from the video determination unit 114 and outputs it to the video output unit 116. The video output unit 116 outputs the video supplied from the video switching unit 115 to the outside.

When switching the output video from a virtual camera to a real camera, the automatic virtual camera information generation unit 117 automatically generates virtual camera information to obtain a virtual viewpoint video that connects the videos before and after the switch. The virtual camera information generated by the automatic virtual camera information generation unit 117 is one of the video effects when switching videos. When the positions, attitudes (directions of gaze), and angle of view (focal length (zoom value)) of the virtual camera and real camera are different, new virtual camera information is automatically generated from the virtual camera information and real camera information, smoothing the change in video when switching videos.

Next, the hardware configuration of the image processing device 103 that realizes the above-described functional configuration will be described using FIG. 10. The image processing device 103 has a CPU (Central Processing Unit) 1001, ROM (Read Only Memory) 1002, RAM (Random Access Memory) 1003, auxiliary storage device 1004, display unit 1005, operation unit 1006, communication I/F 1007, and bus 1018.

The CPU 1001 realizes each function of the image processing device 103 shown in FIG. 1 by controlling the entire image processing device 103 using computer programs and data stored in the ROM 1002 and RAM 1003. The image processing device 103 may have one or more pieces of dedicated hardware separate from the CPU 1001, and at least some of the processing by the CPU 1001 may be performed by the dedicated hardware. Examples of dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 1002 stores programs that do not require modification. The RAM 1003 temporarily stores programs and data supplied from the auxiliary storage device 1004, as well as data supplied from the outside via the communication I/F 1007. The auxiliary storage device 1004 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The display unit 1005 is composed of, for example, an LCD display or LEDs, and displays a GUI (Graphical User Interface) that allows the user to operate the image processing device 103. The operation unit 1006 is composed of, for example, a keyboard, mouse, joystick, touch panel, etc., and receives operations from the user to input various instructions to the CPU 1001. The communication I/F 1007 is used for communication with devices external to the image processing device 103. For example, if the image processing device 103 is connected to an external device via a wired connection, a communication cable is connected to the communication I/F 1007. If the image processing device 103 has the function of wirelessly communicating with external devices, the communication I/F 1007 is equipped with an antenna. The bus 1018 connects each unit of the image processing device 103 to transmit information.

In this embodiment, the display unit 1005 and operation unit 1006 are assumed to exist inside the image processing device 103, but at least one of the display unit 1005 and operation unit 1006 may exist as a separate device outside the image processing device 103. In this case, the CPU 1001 may operate as a display control unit that controls the display unit 1005 and an operation control unit that controls the operation unit 1006.

Next, the process of switching between images from a virtual camera and a real camera performed by the image processing device 103 configured as described above will be described with reference to Figure 2. Figure 2 is a flowchart showing the output image determination process performed by the image processing device of the first embodiment. Note that Figure 2 omits the process of storing images acquired by the camera group 101 in the foreground image storage unit 107 and the process of storing foreground images separated by the separation unit 106 in the 3D model storage unit 109.

In step S201, the virtual viewpoint video generation unit 111 acquires the virtual camera information generated by the virtual camera information generation unit 110. In step S202, the virtual viewpoint video generation unit 111 generates a virtual viewpoint video based on the acquired virtual camera information. In step S203, the video switching unit 115 acquires output video switching information from the video determination unit 114. The switching information indicates, for example, the channel of the output video after switching determined by the video determination unit 114, the time of switching, etc. In step S204, the video switching unit 115 determines whether to stop the output video based on the switching information acquired in step S203. If it is determined that the output video should be stopped (YES in step S204), the video switching unit 115 stops the video output in step S205. If it is determined that the output video should not be stopped (NO in S204), the processing proceeds to step S206.

In step S206, the video switching unit 115 determines whether to switch the output video based on the switching information acquired in step S203. If it is determined not to switch the output video (NO in step S206), in step S207 the video switching unit 115 continues outputting the video without switching the output video. Then, the process returns to step S201. On the other hand, if it is determined to switch the output video (YES in step S206), the process proceeds to step S208.

In step S208, the video switching unit 115 determines whether to automatically generate virtual camera information when switching the output video. If it is determined that virtual camera information should not be automatically generated (NO in step S208), in step S209, the video switching unit 115 immediately switches the video to be output to the video output unit 116 based on the switching information to the post-switching video indicated by the switching information. For example, switching is performed from the virtual viewpoint generated by the virtual camera information generation unit 110 to the virtual viewpoint video generated by the virtual viewpoint video generation unit 111 to the real camera video captured by the real camera 112. Then, processing returns to step S201. On the other hand, if it is determined that virtual camera information should be automatically generated (YES in step S208), processing proceeds to step S210.

The switching information from the video determination unit 114 is also provided to the virtual camera information automatic generation unit 117. In step S210, the virtual camera information automatic generation unit 117 acquires switching conditions from the switching information received from the video determination unit 114. The switching conditions include, for example, transition period information indicating the period (start time and end time) for automatically generating virtual camera information. The virtual camera information automatic generation unit 117 acquires virtual camera information necessary to generate a virtual viewpoint from the virtual camera information generation unit 110 and real camera information from the real camera information acquisition unit 113. In step S211, the virtual camera information automatic generation unit 117 generates new virtual viewpoint information (virtual camera information) for switching videos based on the virtual camera information, real camera information, and switching conditions. In step S212, the virtual viewpoint video generation unit 111 generates a virtual viewpoint video based on the new virtual viewpoint generated by the virtual camera information automatic generation unit 117. The video switching unit 115 outputs the virtual viewpoint video obtained from this new virtual viewpoint and then starts outputting the selected video (in this example, the real camera video). Processing then returns to step S201.

The relationship between the virtual viewpoint video, real camera video, and output video over time when switching the output video from the virtual camera to the real camera is explained below using Figure 3. Figure 3 is a diagram showing a timeline of the video switching process in the first embodiment. In Figure 3, the first virtual viewpoint video 301 is a virtual viewpoint video generated by the virtual viewpoint video generation unit 111 based on the virtual camera information (also referred to as first virtual camera information) generated by the virtual camera information generation unit 110. The real camera video 302 is a video captured and output by the real camera 112. The second virtual viewpoint video 303 is a virtual viewpoint video generated by the virtual viewpoint video generation unit 111 based on the virtual camera information (also referred to as second virtual camera information) generated by the virtual camera information automatic generation unit 117. The output video 304 is a video selected by the video switching unit 115 from among the candidate videos of the first virtual viewpoint video 301, real camera video 302, and second virtual viewpoint video 303, and output. The horizontal axis represents time.

The virtual viewpoint video generation unit 111 generates and outputs a first virtual viewpoint video 301 in accordance with virtual camera information generated by the virtual camera information generation unit 110 in response to virtual camera operation by the operator. The real camera 112 also outputs a real camera video 302 that it has captured. Note that the real camera 112's position, orientation, zoom, etc. are controlled by the cameraman during capture. At time t0, the video determination unit 114 outputs switching information 310 to the video switching unit 115, indicating that switching will be performed from the first virtual viewpoint video 301 to the real camera video 302 after t2-t0 seconds, and then over t7-t2 seconds using the second virtual viewpoint video 303. In the example of Figure 3, the period from time t2, when output of the first virtual viewpoint video ends, to time t7, when output of the real camera video 302 begins, is set as a transition period.

The switching information 310 received by the video switching unit 115 instructs that the output video be switched from the first virtual viewpoint video 301 to the real camera video 302, and that the second virtual viewpoint video 303 be used as the switching condition. Note that the second virtual viewpoint video 303 is a virtual viewpoint image generated by the virtual viewpoint video generation unit 111 based on the virtual camera information generated by the automatic virtual camera information generation unit 117. Furthermore, the switching condition sets the time from t2 to t7 as the transition period for switching the video (the period during which the second virtual viewpoint video is output).

When switching information 310 including the switching conditions described above is output from the image determination unit 114, YES is determined in steps S206 and S208 of FIG. 2. When the automatic virtual camera information generation unit 117 receives this switching condition, it generates a new virtual viewpoint (also referred to as a second virtual viewpoint) for creating a second virtual viewpoint video 303 for switching from the first virtual viewpoint video 301 to the real camera video 302 from time t2 to time t7. More specifically, to create virtual viewpoint information from time t2 to time t7, the automatic virtual camera information generation unit 117 first acquires virtual camera information from the virtual camera information generation unit 110 and real camera information from the real camera information acquisition unit 113. The virtual camera information includes information on the position, line of sight direction, and angle of view of the virtual viewpoint used by the virtual viewpoint video generation unit 111 to generate the first virtual viewpoint video 301. The real camera information includes information on the position, attitude, and angle of view of the real camera 112 capturing the real camera video 302. Until time t2, the video switching unit 115 selects the first virtual viewpoint video 301 and outputs it to the video output unit 116. At time t2, the video switching unit 115 switches the video to be output to the video output unit 116 from the first virtual viewpoint video 301 to the second virtual viewpoint video 303. Furthermore, at time t7, the video switching unit 115 switches the video to be output to the video output unit 116 from the second virtual viewpoint video 303 to the real camera video 302. The video output unit 116 outputs the video sent from the video switching unit 115.

An example of the process for automatically generating virtual camera information by the virtual camera information automatic generation unit 117 will be described in detail using Figures 4A to 4C. Figures 4A to 4C show an example of the process for automatically generating virtual camera information in the first embodiment. Figure 4A shows the positions and orientations at each time between t0 and t10 of the virtual camera for generating the first virtual viewpoint video 301, the virtual camera for generating the second virtual viewpoint video 303, and the real camera 112 that captures the real camera video 302. The positions of the virtual camera and real camera are described below, but other camera information (orientation, zoom state, etc.) can also be calculated in a similar manner. The timeline from t0 to t10 corresponds to the timeline shown in Figure 3.

In Figures 4A to 4C, first virtual camera information 401 indicates the position indicated by the position information of the first virtual camera generated by the virtual camera information generation unit 110 with a black dashed arrow. The first virtual camera moves moment by moment along the black dashed arrow in the direction of the arrow between t0 and t10. Real camera information 403 indicates the position indicated by the position information of the real camera 112 acquired by the real camera information acquisition unit 113 with a white dashed arrow. The real camera 112 moves moment by moment along the white dashed arrow in the direction of the arrow between t0 and t10. The virtual camera information automatic generation unit 117 generates second virtual camera information 402 starting from the virtual camera information at time t2, gradually approaching the real camera information at each time. In Figures 4A to 4C, the movement of the second virtual camera based on the second virtual camera information 402 is indicated by a black solid arrow.

The following describes, using Figures 4B and 4C, how the automatic virtual camera information generation unit 117 generates the position of the second virtual camera 2 from the position of the first virtual camera, which moves from moment to moment, and the position of the real camera 112. Below, we explain an example of generating information about the second virtual viewpoint based on information about the first virtual camera, information about the real camera, and the ratio between the elapsed time from the start of the transition period and the total time of the transition period.

4a in Figure 4B shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t2. At time t2, the position of the second virtual camera is the same as the position of the first virtual camera. 4b shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t3. The position of the second virtual camera at time t3 is determined based on the ratio of the elapsed time (t3 - t2) to the total time of the transition period (t7 - t2). More specifically, the position of the second virtual camera at time t3 is the position on the line segment connecting the position of the first virtual camera at time t2 and the position of real camera 112 at time t3, which is located a distance from the first virtual camera toward real camera 112 by the ratio (t3 - t2) / (t7 - t2). In other words, the position of the second virtual camera during the transition period is generated by taking a weighted average based on the ratio between the position of the first virtual viewpoint and the position of the real camera 112. 4c shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t4. The position of the second virtual camera at time t4 is generated in the same manner as time t3. That is, the position of the second virtual camera at time t4 is the position on the line segment connecting the position of the first virtual camera at time t2 and the position of the real camera 112 at time t4, moved from the first virtual camera toward the real camera 112 by a ratio of (t4 - t2) / (t7 - t2).

4d in Figure 4C shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t5. The position of the second virtual camera at time t5 is also generated in a similar manner to that of time t3. That is, the position of the second virtual camera at time t5 is the position on the line segment connecting the position of the first virtual camera at time t2 and the position of real camera 112 at time t5, which is moved from the first virtual camera toward real camera 112 by a ratio of (t5 - t2) / (t7 - t2). 4e in Figure C shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t6. The position of the second virtual camera at time t6 is also generated in a similar manner. That is, the position is the position on the line segment connecting the position of the first virtual camera at time t2 and the position of real camera 112 at time t6, which is moved from the first virtual camera toward real camera 112 by a ratio of (t6 - t2) / (t7 - t2). 4f in Figure 4C shows the positions of the first virtual camera, real camera 112, and second virtual camera at time t7. The position of the second virtual camera at time t7 is a position on the line segment connecting the position of the first virtual camera at time t2 and the position of real camera 112 at time t7, moved from the first virtual camera toward real camera 112 by a ratio of (t7 - t2) / (t7 - t2). In other words, at time t7, which is the end time of the transition period, the position of the second virtual camera and the position of real camera 112 are the same.

As described above, according to the first embodiment, a transition period from time t2 to time t7 is provided when switching from the virtual viewpoint image captured by the first virtual camera to the real camera image captured by real camera 112. During this transition period, information about the second virtual camera, which moves from the position of the first virtual camera to the position of real camera 112, is generated based on the information about the first virtual camera and the information about the real camera during the transition period. Therefore, when switching from the image captured by the first virtual camera to the image captured by real camera 112, even if the positions of the first virtual camera and the real camera are far apart, virtual camera information that interpolates between them can be automatically generated during the transition period. As a result, it is possible to provide a natural image when switching from the image captured by the virtual camera to the image captured by the real camera. While the process of switching from the virtual camera image to the real camera image has been described, the same process can also be applied when switching from the real camera image to the virtual camera image. In this case, the position of the second virtual camera at the first time of the transition period will be the same as that of the real camera 112, and the position of the second virtual camera will gradually approach the position of the first virtual camera.

Note that in Figures 4A to 4C, the position of the second virtual camera during the transition period is set to gradually approach the position of the real camera, independent of the position of the first virtual camera, except at the start of the transition period, but this is not limited to this. For example, the second virtual camera information 402 may be automatically generated using a method such as that shown in Figure 5.

Figure 5 shows another example of a method for generating a virtual camera path for a virtual viewpoint video in the first embodiment. Similar to Figures 4A to 4C, Figure 5 shows the positions of the first virtual camera, the second virtual camera, and the real camera 112 at each time between times t0 and t10. In this example, a method is described for generating second virtual camera information 402 using information from the first virtual camera and the real camera 112 at the same time. Similar to the method described in Figures 4A to 4C, at time t2, the positions of the first virtual camera and the second virtual camera are the same.

The position of the second virtual camera at time t3 is a position on the line segment connecting the positions of the first virtual camera and real camera 112 at time t3, moved from the first virtual camera toward real camera 112 by a ratio of (t3 - t2) / (t7 - t2). Similarly, the position of the second virtual camera at time t4 is a position on the line segment connecting the positions of the first virtual camera and real camera 112 at time t4, moved from the first virtual camera toward real camera 112 by a ratio of (t4 - t2) / (t7 - t2). Similarly, the position of the second virtual camera at time t5 is a position on the line segment connecting the positions of the first virtual camera and real camera 112 at time t5, moved from the first virtual camera toward real camera 112 by a ratio of (t5 - t2) / (t7 - t2). Similarly, the position of the first virtual camera at time t6 is a position on the line segment connecting the positions of the first virtual camera and real camera 112 at time t6, moved from the first virtual camera toward real camera 112 by a ratio of (t6 - t2) / (t7 - t2). Similarly, the position of the second virtual camera at time t7 is a position on the line segment connecting the positions of the first virtual camera and real camera 112 at time t7, moved from the first virtual camera toward real camera 112 by a ratio of (t7 - t2) / (t7 - t2). As explained in Figure 4C (4f), at time t7, which is the end time of the transition period, the position of the second virtual camera and the position of real camera 112 are the same.

As described above, with the method shown in FIG. 5, the virtual camera position when switching from the virtual viewpoint image captured by the first virtual camera to the real camera image captured by real camera 112 is calculated based on the positions of the first virtual camera and real camera 112 at the same time. With this method, when switching from virtual camera image to real camera image, or from real camera image to virtual camera image, the position of the second virtual camera is always calculated from the positions of the first virtual camera and real camera 112 at the same time. Therefore, even if the second virtual camera changes direction from the real camera position to virtual camera position 1 while moving from the position of the first virtual camera to the position of real camera 112, the switching can be performed without any sense of incongruity.

Furthermore, in the above method for automatically generating the two pieces of virtual camera information, the start and end times for switching between images are specified, but this is not limited to this; the start time for switching and the time required for switching (length of transition period) may also be specified. This makes it easy to specify the time required for switching in advance and to standardize switching times when generating the same images.

Furthermore, in the method for automatically generating the two pieces of virtual camera information described above, the movement of the second virtual camera when switching between images is determined based on the ratio between the elapsed time and the movement period, but this is not limited to this. For example, instead of the ratio between the elapsed time and the movement period described above, a ratio specified by user operation (hereinafter referred to as the transition ratio) may be used at each time during the transition period. For example, the image determination unit 114 may be provided with an input unit having a fader that can specify the image before switching and the image after switching and specify the transition ratio, and the position of the second virtual viewpoint may be generated in response to user operation on the input unit.

Figure 6 shows an example of an input unit 600 that can specify a transition ratio. User operations via the input unit 600 are output to the video determination unit 114. The input unit 600 has a pre-switching button switch 601 and a post-switching button switch 602, each equipped with button switches for channels 1 to 4. A fader 603 is provided across the pre-switching button switch 601 and the post-switching button switch 602. The fader 603 moves in response to user operations, and indicates the transition ratio when switching videos according to its position. In this embodiment, the virtual viewpoint video from the first virtual camera is assigned to channel 1, and the real camera video from the real camera 112 is assigned to channel 2.

In Figure 6(a), the fader 603 is in the top position, and in this case, the video of the channel specified by the pre-switching button switch 601 is output. The pre-switching button switch 601 for channel 1 is lit, indicating that the video of channel 1 (first virtual viewpoint video 301) has been selected as the video to be output from the video switching unit 115. Meanwhile, channel 2 has been selected by the post-switching button switch 602, and channel 2 is lit. This indicates that channel 2 (real camera video 302) has been selected as the video to be output after switching. When the fader 603 is moved from the top to the bottom, the output video switches from the virtual viewpoint video captured by the first virtual camera to virtual viewpoint video 2 captured by the second virtual camera. The position of the second virtual camera is generated by the method described above with reference to Figures 4A to 4C or 5, based on a transition ratio corresponding to the position of the fader 603. The transition ratio can be set, for example, based on the distance from the top to bottom position of the fader 603 and the distance from the top to the current position of the fader 603.

In the example of Figure 6(b), fader 603 is located 2/5 of the way from the top to the bottom. In this case, the position of the second virtual camera is 2/5 of the way along the line segment connecting the position of the first virtual camera and the position of real camera 112 at that time, moving from the first virtual camera toward real camera 112 (similar to 4c in Figure 4B). Note that the time when fader 603 starts moving from the state in Figure 6(a) is the start time of the transition period described above, and the time when fader 603 reaches the bottom, as shown in Figure 6(c), is the end time of the transition period. In other words, when fader 603 reaches the bottom, the image from the second virtual camera switches to the image from real camera 112, completing the image switch.

As described above, by operating the fader 603, it is possible to specify the transition ratio used by the virtual camera information automatic generation unit 117 to generate virtual camera information when switching videos. This makes it easy to control the switching time and the speed at which the virtual camera approaches the state of the real camera.

The above describes switching from virtual viewpoint video to real camera video, but this is not limiting; the above processing can also be applied to switching from real camera video to virtual viewpoint video. That is, one of the first viewpoint for obtaining the video before switching and the second viewpoint for obtaining the video after switching can be the viewpoint of a virtual imaging device for generating the virtual viewpoint video, and the other can be the viewpoint of a physical imaging device that captures the video. In this case, switching occurs from the real camera video to a virtual viewpoint video from the second virtual camera, and then to a virtual viewpoint video from the first virtual camera. The virtual viewpoint video is generated as if switching had occurred from virtual viewpoint camera information 2 to virtual viewpoint camera information 1. Furthermore, when switching between two virtual viewpoint videos from two virtual viewpoints, or when switching between two own-camera videos from two real cameras, the virtual viewpoint video from the second virtual camera generated by the virtual camera information automatic generation unit 117 can be used.

As described above, according to the first embodiment, when switching from a first image obtained from a first viewpoint to a second image obtained from a second viewpoint, a new virtual camera is generated to complement the transition between the first and second viewpoints. Then, by using a virtual viewpoint image from the new virtual viewpoint between the first and second images, it is possible to achieve a transition between the first image and the second image obtained after the switch, as if they were captured from a single viewpoint (camera). Furthermore, smoothly switching between the virtual viewpoint image and the image from the real camera enables more dynamic visual expression that cannot be captured with a real camera.

Second Embodiment
In the first embodiment, a process for generating information about a virtual viewpoint (second virtual camera) based on information about a first virtual camera and information about a real camera was described. Information about the virtual viewpoint includes position, orientation (direction of line of sight), focal length (zoom value), and the like, but in the process of the first embodiment, these are generated by the same process without any particular distinction. In the second embodiment, position information and orientation information of the virtual viewpoint are generated by independent processes. Note that the same reference numerals are used for components equivalent to those in the first embodiment, and detailed descriptions thereof will be omitted.

As described above, in the first embodiment, the position information of the second virtual camera is generated from the position information of the first virtual camera and the position information of the real camera 112 so as to move between them, and the attitude of the second virtual camera can be generated using a similar method. However, the method of the first embodiment has the problem that, depending on the attitude and focal length of the second virtual camera, the subject to be photographed may not be included in the shooting range of the second virtual camera. In the second embodiment, to solve this problem, the position of the second virtual camera and the attitude and focal length information of the second virtual camera are controlled independently.

Figure 7 is a block diagram showing an example configuration of an image processing system according to the second embodiment. This configuration adds an object identification unit 701 to the configuration of the first embodiment (Figure 1). The object identification unit 701 identifies the object captured by the virtual camera or real camera 112. That is, the object identification unit 701 identifies the object captured in the image captured by the virtual camera or real camera 112 based on camera information from the virtual camera information generation unit 110, real camera information acquisition unit 113, and automatic virtual camera information generation unit 117, as well as information from the 3D model storage unit 109. The image acquisition unit 104 also provides the image acquired from the camera control unit 102 to the image switching unit 115. This enables the image switching unit 115 to use the image captured by the camera group 101 used for the virtual viewpoint image as image output.

Figure 8 is a flowchart showing the output video determination process according to the second embodiment. Processes equivalent to those in the first embodiment (Figure 2) are assigned the same step numbers. In step S801, the automatic virtual camera information generation unit 117 references the switching information and determines whether the transition ratio of the position and orientation of the second virtual camera is different during the transition period from the first virtual viewpoint video 301 to the real camera video 302. If it is determined that the transition ratio is not different (NO in step S801), the process proceeds to step S211. On the other hand, if it is determined that the transition ratio is different (YES in step S801), the process proceeds to step S802.

In step S802, the automatic virtual camera information generation unit 117 generates information about the position, attitude, and angle of view of the second virtual camera when switching from virtual camera footage to real camera footage, based on information about the first virtual camera, information about the real camera 112, and the switching conditions. The automatic virtual camera information generation unit 117 acquires the position transition period for switching from the position of the first virtual camera to the position of the real camera 112, and the attitude transition period for switching from the attitude of the first virtual camera to the attitude of the real camera 112, which are included in the switching conditions. In the switching conditions, for example, the position transition period and the attitude transition period are set independently of each other and are indicated by start and end times, respectively. The automatic virtual camera information generation unit 117 calculates the position and attitude of the second virtual camera at each time. Note that, as in the first embodiment, an input unit 600 equipped with a fader 603 for specifying the switching ratio may be used. In this case, a separate fader 603 is provided for each condition that needs to be controlled independently.

The orientation of the second virtual camera may be calculated using a transition ratio different from the position transition ratio so as to prioritize displaying the subject included in the output image after switching. Figures 9A and 9B show an example of the process of generating virtual camera information in step S802 so as to prioritize displaying the subject included in the output image after switching. The positions and orientations of the first virtual camera, second virtual camera, and real camera 112 at each time are as shown in Figure 4A. Note that, in the first virtual camera, subject 901 is present within its shooting range as the subject primarily being photographed, and in the real camera 112, subject 902 is present within its shooting range as the subject primarily being photographed. During the position transition period (times t2 to t7), the position of the second virtual camera transitions from the position of the first virtual camera to the position of real camera 112, as in the first embodiment. Meanwhile, the attitude and focal length (zoom value) of the second virtual camera are abruptly changed during the attitude transition period from time t2 to time t4 so that the angle of view is equivalent to that of real camera 112. Thereafter, the attitude and focal length of the second virtual camera are set so that the angle of view is equivalent to that of real camera 112 from time t4 to time t7. Note that equivalent angle of view refers to an attitude and angle of view set so that the same subject appears in approximately the same position in the images obtained from each viewpoint. Alternatively, it refers to an attitude and angle of view set so that the same subject appears at approximately the same size in the images obtained from each viewpoint. Alternatively, it refers to an attitude and angle of view set so that the same subject appears in approximately the same position and size in the images obtained from each viewpoint.

The subject identification unit 701 can determine the position of the foreground in the virtual viewpoint image captured by the first virtual camera based on the information on the position, orientation, and focal length of the first virtual camera from the virtual camera information generation unit 110 and the position of the foreground from the 3D model storage unit 109. Similarly, the position of the foreground in the real camera image captured by the real camera 112 can be determined based on the information on the position, orientation, and focal length of the real camera 112 and the position of the foreground from the 3D model storage unit 109. During the transition period in which the virtual viewpoint image from the second virtual camera is being output, the automatic virtual camera information generation unit 117 of this embodiment calculates the orientation of the second virtual camera as if the second virtual camera were capturing the post-switching image, i.e., the image with the same angle of view as the image from the real camera 112.

In Figure 9A, 9a shows the position 911 and orientation 912 of the first virtual camera at time t2, and the position 931 and orientation 932 of the real camera 112 at time t2. At time t2, the position and orientation of the second virtual camera are the same as the position 931 and orientation 932 of the first virtual camera. 9b in Figure 9A shows the position 913 and orientation 914 of the first virtual camera, the position 933 and orientation 934 of the real camera 112, and the position 951 and orientation 954 of the second virtual camera at time t3. The orientation 954 of the second virtual camera at time t3 is determined based on the orientation 912 (orientation 952) of the first virtual camera at time t2 and the orientation 953 that enables the second virtual camera to obtain an angle of view equivalent to that of the real camera 112 at time t3. In other words, the orientation 954 of the second virtual camera at time t3 is an orientation that is tilted from orientation 952 to orientation 953 by a ratio of (t3 - t2) / (t4 - t2) between orientation 952 and orientation 954.

In Figure 9B, 9c shows the position 915 and orientation 916 of the first virtual camera, the position 935 and orientation 936 of the real camera 112, and the position 955 and orientation 956 of the second virtual camera at time t4. As with time t3, the orientation 956 of the second virtual camera at time t4 is determined based on the orientation 912 of the first virtual camera at time t2 and the orientation that allows the second virtual camera to obtain an angle of view equivalent to that of the real camera 112 at time t4. However, at time t4, (t4 - t2) / (t4 - t2) = 1, and therefore the orientation 956 that allows the second virtual camera to obtain an angle of view equivalent to that of the real camera 112 is determined as the orientation of the second virtual camera at time t4.

9d in Figure 9B shows the position 917 and orientation 918 of the first virtual camera, the position 937 and orientation 938 of the real camera 112, and the position 957 and orientation 958 of the second virtual camera at time t5. The orientation 958 of the second virtual camera at time t5 is determined so as to obtain the same angle of view as the real camera 112 at time t5. Similarly, 9e in Figure 9C shows the position 919 and orientation 920 of the first virtual camera, the position 939 and orientation 940 of the real camera 112, and the position 959 and orientation 960 of the second virtual camera at time t6. The orientation 960 of the second virtual camera at time t6 is determined so as to obtain the same angle of view as the real camera 112 at time t6. 9f in Figure 9C shows the position 921 and orientation 922 of the first virtual camera and the position 941 and orientation 942 of the real camera 112 at time t7. At time t7, the position and orientation of the second virtual camera are the same as the position 941 and orientation 942 of the real camera 112.

<Other Embodiments>
In the above embodiments, the real camera 112 has been described as a camera that is different from the group of cameras 101 that generate the virtual viewpoint video and is brought in around the shooting range of the virtual viewpoint video, but this is not limited to this. For example, as in the second embodiment, if the images from some or all of the cameras in the group of cameras 101 are sent to the image switching unit 115 and can be selected as output images, the real camera 112 may be any one of the group of cameras 101. This makes it possible to easily generate a new virtual viewpoint video for the transition period between the images, even when switching from the virtual viewpoint video to real camera image from one of the real cameras in the group of cameras 101 that generate the virtual viewpoint video.

Furthermore, the virtual viewpoint during the transition period may be generated for each frame captured by the real camera 112 during the transition period (or for each frame of the virtual viewpoint image captured by the first virtual viewpoint), or at a predetermined time interval (e.g., every 0.5 seconds).

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

101: Camera group, 102: Camera control unit, 103: Image processing device, 104: Image acquisition unit, 105: Background image storage unit, 106: Separation unit, 107: Foreground image storage unit, 108: 3D model generation unit, 109: 3D model storage unit, 110: Virtual camera information generation unit, 111: Virtual viewpoint image generation unit, 112: Real camera, 113: Real camera information acquisition unit, 114: Image determination unit, 115: Image switching unit, 116: Image output unit, 117: Automatic virtual camera information generation unit

Claims (15)

an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
The first generating means generates a virtual viewpoint for the period based on information about the first viewpoint, information about the second viewpoint, and a ratio between an elapsed time from the start of the period and an entire time of the period.
1. An image processing device comprising: 2. The image processing device according to claim 1, wherein the first generating means generates virtual viewpoint information for the period based only on information about the first viewpoint at the start time of the period. an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
a setting means for setting a ratio in accordance with a user operation received during the period ;
and
The image processing device according to claim 1, wherein the first generating means generates a virtual viewpoint for the period based on the information of the first viewpoint, the information of the second viewpoint, and the ratio set by the setting means. 4. The image processing device according to claim 1, wherein the first generating means generates a virtual viewpoint for the period by taking a weighted average of the information of the first viewpoint and the information of the second viewpoint based on the ratio. 5. The image processing device according to claim 1, wherein the first generation means generates a virtual viewpoint at each time during the period based on information about the first viewpoint at the start time of the period and information about the second viewpoint at each time. an acquisition means for acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition means acquiring information about a first viewpoint for acquiring the first image and information about a second viewpoint for acquiring the second image at a time corresponding to the time of the first image;
a setting means for setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generating means for generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generating means for generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output means for switching between and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
The image processing device is characterized in that the first generation means generates a virtual viewpoint at each time during the period based on information about the first viewpoint at each time and information about the second viewpoint at each time. further comprising an identification means for identifying a subject from the image captured from the second viewpoint,
7. The image processing device according to claim 1, wherein the first generating means generates information about the direction of the line of sight included in the information about the virtual viewpoint for the period based on the position of the subject identified by the identifying means. The image processing device described in claim 7, characterized in that the first generation means generates gaze direction information included in the virtual viewpoint information for the period based on the gaze direction of the virtual viewpoint to obtain an image of a shooting range in which the position of the subject shown in the virtual viewpoint image is the same as the position of the subject shown in the image obtained from the second viewpoint of the subject, and the gaze direction of the first viewpoint at the start of the period. The image processing device described in claim 7 or 8, characterized in that the first generation means generates information about the focal length of the virtual viewpoint for the period based on the focal length of the line of sight of the virtual viewpoint for obtaining an image of a shooting range in which the size of the subject shown in the virtual viewpoint image is the same as the size of the subject shown in the image obtained from the second viewpoint of the subject, and the focal length of the line of sight of the first viewpoint at the start of the period . 10. The image processing device according to claim 1, wherein one of the first image and the second image is a virtual viewpoint image generated based on a plurality of images captured by a plurality of imaging devices and a virtual viewpoint. the second generating means further includes a connection means for connecting to a plurality of imaging devices for obtaining a plurality of images for generating a virtual viewpoint video;
The image processing device according to claim 10 , wherein the virtual viewpoint video is generated based on the plurality of images. an acquisition step of acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition step acquiring information about a first viewpoint for capturing the first image and information about a second viewpoint for capturing the second image at a time corresponding to the time of the first image;
a setting step of setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generation step of generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generation step of generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output step of switching and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
In the first generation step, a virtual viewpoint for the period is generated based on information about the first viewpoint, information about the second viewpoint, and a ratio between the elapsed time from the start of the period and the total time of the period.
2. A method for controlling an image processing apparatus comprising: an acquisition step of acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition step acquiring information about a first viewpoint for capturing the first image and information about a second viewpoint for capturing the second image at a time corresponding to the time of the first image;
a setting step of setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generation step of generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generation step of generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output step of switching and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
a setting step of setting a ratio in accordance with a user operation received during the period;
and
In the first generating step, a virtual viewpoint for the period is generated based on the information on the first viewpoint, the information on the second viewpoint, and the ratio set in the setting step.
2. A method for controlling an image processing apparatus comprising: an acquisition step of acquiring information relating to a first image and a second image, at least one of which is an image captured by an imaging device, the acquisition step acquiring information about a first viewpoint for capturing the first image and information about a second viewpoint for capturing the second image at a time corresponding to the time of the first image;
a setting step of setting a period from the end of output of the first video to the start of output of the second video when switching the video to be output from the first video to the second video;
a first generation step of generating virtual viewpoint information for the period based on the first viewpoint information for the period and the second viewpoint information for the period;
a second generation step of generating a virtual viewpoint video for the period based on information about a virtual viewpoint for the period;
an output step of switching and outputting the first video, the virtual viewpoint video for the period, and the second video in that order;
and
In the first generation step, a virtual viewpoint at each time in the period is generated based on information about the first viewpoint at each time and information about the second viewpoint at each time.
2. A method for controlling an image processing apparatus comprising: A program for causing a computer to function as the image processing device according to any one of claims 1 to 11 .