JP7709299B2 - Processing system, control method thereof, and program - Google Patents

Description

This disclosure relates to a processing system and its control method and program.

Recently, technology has been gaining attention that involves synchronous shooting using multiple cameras installed in different positions, and then generating virtual viewpoint content using the multiple viewpoint images obtained from the shooting. With this technology, for example, it is possible to view highlight scenes of a soccer or basketball game from a variety of angles, providing the user with a more realistic experience than with normal images.

Patent document 1 discloses a technology in which multiple cameras are arranged to surround a subject, and images of the subject captured by these multiple cameras are used to generate an image from any viewpoint.

JP 2008-015756 A

Traditionally, there has been a so-called tally (tally lamp, tally light). A tally is a red lamp attached to television cameras and the like that lights up when the device is in use, and is usually linked to a switcher or similar device. It lets the performers (subjects) know that they are being filmed, so they can use it as a cue for what to say or do.

However, in the system that generates the virtual viewpoint content described above, there is no actual camera at the position of the virtual viewpoint. Therefore, for example, when filming in a studio, the performers do not know where to look, which creates limitations in terms of directing.

This disclosure has been made in consideration of the above problems, and aims to provide a technology that supports the subject (performer) in capturing images that are used to generate virtual viewpoint images.

In order to solve this problem, for example, the processing system of the present disclosure has the following configuration.
1. A processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, comprising:
A plurality of imaging means for imaging a space including a subject;
A means for identifying a position of the virtual viewpoint;
and a presentation means for presenting a presentation for determining a direction of the subject in the space based on the position of the virtual viewpoint .

The present disclosure provides a technology that assists a subject (performer) in capturing images used to generate a virtual viewpoint image.

1 is a diagram showing the overall configuration of an image processing system according to a first embodiment. 1A and 1B are diagrams showing a configuration and a part of a space surrounding a subject in a first embodiment. FIG. 11 is an explanatory diagram for explaining a marker display. 6 is a flowchart showing a marker display process according to the first embodiment. 13A to 13C are explanatory diagrams for explaining a marker display process according to the second embodiment. FIG. 13 is a diagram for explaining a configuration for marker display in the third embodiment. FIG. 13 is a diagram for explaining a configuration for presenting a marker in the fourth embodiment. FIG. 13 is a diagram for explaining a method of presenting a marker in the fifth embodiment. FIG. 2 is a diagram showing the hardware configuration of each device constituting the system of the embodiment. FIG. 13 is a diagram showing a hardware configuration of an apparatus according to a first modified example of the first embodiment. 10 is a flowchart showing a processing procedure according to a first modified example of the first embodiment. 13A and 13B are diagrams showing a configuration in a space surrounding a subject and a part thereof in a second modified example of the first embodiment. 13A and 13B are diagrams showing a configuration and a part of a space surrounding a subject in third and fourth modified examples of the first embodiment. 13 is a flowchart showing a processing procedure according to a fifth modified example of the first embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

[First embodiment]
A system for capturing images using multiple cameras installed in a facility such as a stadium, concert hall, or studio will be described with reference to FIG.

Figure 1 is an overall configuration diagram of an image processing system 100 to which the first embodiment is applied. This system 100 has an imaging device 1, a real space information retention device 2, an image processing device 3, a user terminal 4, and a marker output device 5, which are connected via a network. The network may be either wired or wireless.

In the above configuration, the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 can be realized by an information processing device represented by a personal computer (PC) and an application program executed on the information processing device. FIG. 9 is a hardware block diagram of the information processing device 200. The information processing device 200 has a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display device 215, an operation unit 216, and a communication interface (I/F) 217, which are connected to a system bus 218. When the power supply of this device is turned on, the CPU 211 executes a boot program in the ROM 212, loads an OS (operating system) from the auxiliary storage device 214 (e.g., a hard disk) to the RAM 213, and executes the OS. As a result, the information processing device 200 can input various instructions from the user via the operation unit 26 and execute the corresponding processing. Furthermore, the CPU 211 can execute an application program pre-stored in the auxiliary storage device 214 under the control of the OS. For example, depending on the type of application program, the information processing device 200 functions as a real space information holding device 2, an image processing device 3, a user terminal 4, or a marker output device 5. For example, when the information processing device 200 functions as an image processing device 3, the communication I/F 217 is communicably connected to a network (to communicate with the user terminal 4 and the marker output device 5), as well as the imaging device 1 and the real space information holding device 2. When the information processing device 200 functions as the marker output device 5, the communication I/F 217 is connected to a network (to communicate with the user terminal 4 and the marker output device) and a projector. When the information processing device 200 functions as the real space information holding device 2, the auxiliary storage device 214 may be configured to store real space information (described in detail below) and then function as a file server.

The imaging device 1 is made up of multiple cameras installed around a competition field, studio, etc., which capture images synchronously and transmit them to the image processing device 3.

The real space information holding device 2 holds information about a predetermined range of space including the subject (performer). Specifically, this information includes three-dimensional model information of objects (background objects) that appear as the background in the virtual viewpoint image, such as the field or spectator seats of a stadium, or studio equipment, three-dimensional space information indicating the range in which the virtual viewpoint can be set, as well as the installation position, shooting direction, focal length, etc. of each of the imaging devices 1. The information held in the real space information holding device 2 is referenced when the image processing device 3 generates a virtual viewpoint image, and so may be provided within the image processing device 3.

The image processing device 3 has a virtual viewpoint image generation unit 301, a virtual camera path calculation unit 302, a virtual camera path information storage unit 303, a marker coordinate calculation unit 304, a display image generation unit 305, and a communication unit 306.

The virtual viewpoint image generating unit 301 generates a three-dimensional model of a foreground object based on images of multiple viewpoints acquired from the image capturing device 1. The virtual viewpoint image generating unit 301 then generates a virtual viewpoint image by mapping a texture that matches the virtual viewpoint acquired from the virtual camera path calculating unit 302 to the generated foreground three-dimensional model and the background three-dimensional model acquired from the real space information holding device 2, and performing rendering. During this generation process, the virtual viewpoint image generating unit 301 calculates the coordinates of the foreground object and background object that will be reflected in the virtual viewpoint image to be generated, and performs texture mapping and rendering only for those coordinates. The virtual viewpoint image generating unit 301 passes these coordinate values to the virtual camera path information holding unit 303, which will be described later, as the subject foreground coordinates and real space background coordinates.

The virtual camera path calculation unit 302 calculates virtual camera path parameters that are continuous in time based on the user's instructions to the virtual camera path instruction unit 403 of the user terminal 4. The virtual camera parameters are at least the position, orientation (direction of gaze) and angle of view (focal length) of the virtual camera, and are associated with frame numbers or time codes attached to the multiple viewpoint images so that it is possible to identify which moment in the shooting scene the parameters are for. When making this calculation, the real space information obtained from the real space information storage unit 2 is referenced, and the virtual camera path is set within a range in which the virtual viewpoint can be set.

The virtual camera information storage unit 303 accumulates the subject foreground coordinates and real space background coordinates received from the virtual viewpoint image generation unit 301, and the virtual camera path parameters calculated by the virtual camera path calculation unit 302.

The marker coordinate calculation unit 304 generates information on the coordinate parameters of the marker information to be displayed in real space based on the virtual camera parameters related to the virtual camera stored in the virtual camera information storage unit 303 and the three-dimensional space information (background information) stored in the real space information storage unit 2.

The display image generating unit 305 generates a display image to be displayed on the image display unit 402 of the user terminal 4. The display image generated here is a virtual viewpoint image generated by the virtual viewpoint image generating unit 301. The virtual viewpoint image is an image that represents the view from a virtual viewpoint.

The communication unit 306 communicates between the image processing device 3, the user terminal 4, and the marker output device 5 via a network (not shown) or the like. Between the image processing device 3 and the user terminal 4, the communication unit 306 transmits and receives image, audio, and text data, as well as instruction information such as a virtual camera path instruction sent from the user when generating a virtual viewpoint image. Between the image processing device 3 and the marker output device 5, the communication unit 306 transmits and receives display parameters such as coordinate information required for marker display, and the shape, color, size, and display information of the marker.

The user terminal 4 has a communication unit 401, an image display unit 402, a virtual camera path instruction unit 403, a user information transmission unit 404, and a marker information transmission unit 405.

The communication unit 401 transmits and receives various information to and from the device communication unit 306 as described above. The image display unit 402 receives and displays the image generated by the display image generation unit 305 described above. The virtual camera pass instruction unit 403 receives instructions for the virtual camera pass from the user, and passes them to the virtual camera pass calculation unit 302 via the communication units 401 and 306. Here, the user does not necessarily need to specify all of the virtual camera parameters strictly for the entire viewing time. For example, it is possible to input instructions based on various perspectives, such as wanting to view a virtual viewpoint image focusing on a specific player or performer, always wanting to view a certain range around the ball, or wanting to view a location where a more noteworthy event is occurring.

The user information transmission unit 404 attaches user information, including a user ID, to the information transmitted from the communication unit 401 to the communication unit 306. The marker information transmission unit 405 attaches marker type parameter information, such as the type of marker (including shape and color), to the information transmitted from the communication unit 401 to the communication unit 306.

The marker output device 5 has a communication unit 501, a control data generation unit 502, a marker control unit 503, and a display data output unit 504. The marker output device 5 displays (or presents) a marker that can be recognized by a person who is a subject. The displayed marker is output at a position related to the virtual camera parameters. In other words, the position of the marker is controlled by the marker output device 5 in conjunction with the virtual camera parameters.

As described above, the communication unit 501 communicates with the device communication unit 306 of the image processing device 3 and the marker coordinate calculation unit 304 and the user terminal 4 via a network or the like.

The control data generation unit 502 integrates the data output from the virtual viewpoint image generation unit 301 and the marker coordinate calculation unit 304 with the data output from the virtual camera path instruction unit 403 and the marker information transmission unit 405, generates control data such as the marker type, display update time, and position coordinates, and outputs it to the marker control unit 503.

The marker control unit 503 converts the marker into display data (screen coordinate system) based on display parameters such as the display position, size, shape, and display content, and outputs it to the display data output unit 504.

The display data output unit 504 is composed of a display device such as a projector, and outputs display data based on the output data of the marker control unit 503.

Figure 2 (a) is a schematic diagram of the real space surrounding a subject in the image processing system of this embodiment. The image processing system of this embodiment has multiple cameras (image capture devices 1) that surround a predetermined range of real space including the subject. It also has a projector (display data output unit 504) to display markers that can be recognized by the subject (human) at a position based on the virtual camera path information.

The image processing system shown in the figure is an example of the case where this configuration is implemented in a circular studio, with cameras 21a to 21f, projectors 22a to 22f, and screens 23a to 23f arranged to surround subject 24. Image processing device 3 generates a virtual viewpoint image from the multiple viewpoint images acquired by cameras 21a to 21f, and calculates the coordinates on the screen on an extension line connecting subject 24 and the virtual viewpoints as marker coordinates. Then, marker output device 5 receives the marker coordinates and displays a marker on the corresponding screen. Specifically, the marker display is projected onto the screen by projectors 22a to 22f, and projector 22a projects onto screen area 23e in front. Similarly, projector 22b displays a marker on screen area 23d, projector 22c displays a marker on screen area 23f, and projector 22d displays a marker on screen area 23b. Projector 22e displays a marker on screen area 23a, and projector 22f displays a marker on screen area 23c. Projectors 22a to 22f are connected via a network, and display data output unit 504 is configured to transmit display data to the projector on which the data should be displayed. Screens are also configured on the floor and top surfaces, and projectors 22b and 22e project onto floor screen 25. Projectors 27a and 27b project onto top screen 26.

Figure 2(b) is a partial cross-sectional view of the circular studio of this embodiment shown in Figure 2(a). As described above, the markers are displayed by projectors onto the screens, with projector 22 projecting onto screens 23 and 25, and projector 27 projecting the type of marker set by the user onto screen 26.

3(a) is a diagram showing a part of the configuration of the image processing system shown in FIG. 2(a). The virtual viewpoint 31 indicates the position of the virtual camera when a certain virtual viewpoint image is generated. Also, the marker 32 is projected onto the screen area 23a by the projector 22e. Now, the subject 24 cannot actually see the position of the virtual camera (virtual viewpoint 31), so when he or she tries to focus his or her gaze on the virtual camera, he or she will see the projected marker 32. The projection position of the marker 32 is controlled by the marker control unit 503 so that it is on an extension of the line connecting the face (eyes) of the subject 24 and the virtual viewpoint 31. Also, the distance from the subject's face to the virtual viewpoint 31 is expressed by the size of the circle of the marker 32b. In this way, it is possible to express where the virtual viewpoint 31 is between the marker projected on the screen and the subject's face by the display form of the marker (here, the size). For example, this can be realized by informing the subject in advance that the virtual camera is located closer to the subject when the circle is enlarged, or closer to the marker when the circle is smaller.

Figures 3(b) to (e) are diagrams showing the positional relationship between the subject, the virtual camera, and the markers, and are side views of Figure 3(a). In Figures 3(b) to (e), the white circle 33 indicates the virtual viewpoint, and the black circle 34 indicates the marker projected onto the screen. As shown in this figure, the subject's face, the virtual viewpoint, and the markers are configured to be aligned in a straight line. The screen is also configured on the floor and top surfaces, so as shown in Figures 3(b) to (e), the subject 24 can align its line of sight with the virtual camera even if the virtual camera is angled from above or below the subject.

Figure 4 is a flowchart showing the processing of the control data generation unit 502 of the marker output device 5.

At S100, the control data generation unit 502 starts the generation process. At S101, the control data generation unit 502 performs a process of generating a setting file for marker display based on the data output from the marker information transmission unit 405. This setting file includes information that defines the type of marker and the display parameters of the marker (color, shape, etc.).

At S102, the control data generation unit 502 counts the number of subjects from the three-dimensional model of the subjects in the virtual viewpoint image output from the virtual viewpoint image generation unit 301, and assigns a subject ID.

At S103, the control data generation unit 502 calculates the coordinates of the subject's face in the world coordinate system from the three-dimensional model information of the subject. If there are multiple subjects, the same process is performed for each person. In addition, the face can be identified from the three-dimensional model of the subject using general image processing techniques based on the shape and feature points of the model.

At S104, the control data generation unit 502 calculates the marker coordinates to be displayed based on the setting file, the subject position (face coordinates), and the virtual camera coordinates. The calculation of the marker coordinates is performed in the world coordinate system, and the marker coordinates that intersect with the screen on an extension of the face coordinates and the virtual camera coordinates are calculated. As a result, the screen to project onto and the projector to project onto are determined.

In S105, the control data generation unit 502 generates display parameter information from the setting file and outputs it to the marker control unit 503 together with the marker coordinates.

As a result, the marker control unit 503 controls the display data output unit 504 to display the marker of the set type at the set position.

The displayed marker is a symbol that indicates to the subject the direction in which the virtual viewpoint exists, and may be an icon representing a virtual viewpoint camera. In this case, the icon may be displayed at a size that corresponds to the distance in real space between the subject and the virtual viewpoint. As a result, the subject is able to grasp not only the direction in which the virtual viewpoint exists, but also the distance to the virtual viewpoint.

In addition, when there are multiple subjects, the marker coordinates can be calculated for each subject and displayed with different settings (color and shape) to ensure that the subjects are all aligned with the same position. This can be achieved by informing each subject of the marker settings in advance. Also, if multiple subjects always aim their gaze at the same position, only one marker may be used.

[First Modification of the First Embodiment]
In the above embodiment, the image processing system has been described as being composed of the real space information holding device 2, the image processing device 3, the user terminal 4, and the marker output device 5 as independent devices, but if the processing capacity is sufficient, these may be realized by one information processing device and application program. The configuration of the image processing system in this case is shown in FIG. 10. The image processing system is composed of an information processing device 300, a plurality of cameras 350a, 350b, ... as the image capture device 1, and a plurality of projectors 360a, 360b, ... as the display data output unit 504. The information processing device 300 has a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a display device 305, an operation unit 306, a camera I/F 307, and a projector I/F. Here, the camera I/F 307 and the projector I/F 308 may be realized as one I/F together. It is assumed that the auxiliary storage device 304 is capable of storing real space information, including an OS. The configuration surrounding the subject for shooting and projection is the same as that shown in FIG.

When the power of this device is turned on, the CPU 301 loads the OS stored in the auxiliary storage device 304 into the RAM 303 according to the boot program stored in the ROM 302, and executes it. As a result, this device functions as a device that performs processing according to instructions from the user. Furthermore, by the CPU 301 loading the image processing program from the auxiliary storage device 304 into the RAM 303 and executing it, this device functions as the virtual viewpoint image generation unit 301, the virtual camera path calculation unit 302, the virtual camera information storage unit 303, the marker coordinate calculation unit 304, the display image generation unit 305, the image display unit 402, the virtual camera path instruction unit 403, the user information transmission unit 404, the marker information transmission unit 405, the control data generation unit 502, and the marker control unit 503 in FIG. 1.

The processing procedure when the CPU 301 executes the image processing program will be described with reference to the flowchart in FIG. 11. To simplify the explanation, it is assumed that the information about the markers has already been set.

At S401, the CPU 301 inputs information related to the virtual viewpoint via the operation unit 306. This information includes the coordinates of the virtual viewpoint in real space, as well as the line of sight and angle of view from the virtual viewpoint.

In S402, the CPU 301 receives captured image data from each of the cameras 350a, 350b, ... via the camera I/F 307. Then, in S403, the CPU 301 generates a virtual viewpoint image by referring to real space information (particularly information such as the installation position and direction of the camera) stored in the auxiliary storage device 304. Then, in S404, the CPU 301 displays the generated virtual viewpoint image on the display device 305.

In S405, CPU 301 calculates the intersection position of the line connecting the coordinates of the subject's face and the coordinates of the virtual viewpoint in real space with the screen surface as the marker display position. Then, in S406, CPU 301 causes the projector that is targeting the calculated marker display position to display a marker at the marker position via projector I/F 308.

Then, in S407, the CPU 301 determines whether or not the user has instructed the application to end, and if not, returns to S401 and repeats the above process.

In this modified example, the real space information retention device 2, image processing device 3, user terminal 4, and marker output device 5 are integrated into the information processing device in FIG. 3, but some of them may be independent devices.

In addition, when generating a virtual viewpoint image of an actor's movements in a movie, for example, it is easier for the actor to act if the sense of realism is felt, so each projector may display not only markers but also the surrounding scenery surrounding the actor.

The presentation position of the marker may be determined without depending on the virtual viewpoint. In particular, the position of the marker may be any position to which the subject's line of sight is directed, and may be specified by the user, for example.
[Second Modification of the First Embodiment]
In the above embodiment, the markers are displayed by projecting them from the projector onto the screen, but the configuration of the display data output unit 504 is not limited to this. As shown in FIG. 12, a plurality of liquid crystal displays may be installed so as to cover the surfaces of the screens 23, 25, 26, and 28. In this case, in S104, the control data generation unit 502 determines the display 28a corresponding to the marker coordinates calculated in the world coordinate system, and calculates the display coordinates on the display screen based on the position and orientation of the display 28a that have been calibrated in advance. Calibration of the display can be performed by calculating the three-dimensional coordinates of the markers displayed on the display by triangulation using the cameras 21a to 21f, and associating them with the coordinates of the markers on the display. In addition, in S105, the marker control unit 503 controls the display data output unit 504 to display the markers 32 of the type set at the display coordinates on the corresponding display 28a. On the screens other than the marker display coordinates on the display and on the screens of other displays on which the markers are not projected, the color of the screen may be displayed in an unobtrusive manner, or the surrounding scenery surrounding the performer may be displayed.

[Third Modification of the First Embodiment]
In the first embodiment, the direction in which the virtual viewpoint exists is displayed using a point marker, but a three-dimensional model of the virtual camera may be projected onto the screen to indicate the attitude of the virtual camera. In this case, the three-dimensional model of the virtual camera is stored in the auxiliary storage device 214 in advance, and is expanded into the RAM 213 when the application program is executed. The type of the marker is the virtual camera three-dimensional model. FIG. 13(a) shows how the three-dimensional model of the virtual camera is projected. In S405, the CPU 301 virtually projects the three-dimensional model of the virtual camera from the coordinates 24f of the face of the subject in the real space toward the screen 23a, and calculates the three-dimensional coordinates of the image 32c. Then, the three-dimensional coordinates of the image 32c are back-projected toward the projector 22d to generate an image of the marker to be displayed by the projector 22d. The virtual camera three-dimensional model may have any shape, texture, or transmittance, and may be a three-dimensional arrow model instead of a camera.

This allows the subject to recognize not only the direction in which the virtual viewpoint is located, but also the orientation of the virtual viewpoint.

[Fourth Modification of the First Embodiment]
In the above embodiment, the direction and posture of the virtual viewpoint are displayed using a point or a three-dimensional model, but a virtual viewpoint image may be displayed to indicate the angle of view of the virtual viewpoint. At this time, the display image generating unit 305 transmits the virtual viewpoint image generated immediately before to the projector or display. The type of the marker is a virtual viewpoint image. FIG. 13B shows how the virtual viewpoint image is projected. In S104, the control data generating unit 502 calculates the marker coordinates on the screen or display, and performs a projective transformation on the virtual viewpoint image so that the marker coordinates are at the center and the virtual viewpoint image is viewed from the subject in a frontal direction. The projective transformation uses an affine transformation or a homography transformation. In S105, the marker control unit 503 controls the display data output unit 504 to project the virtual viewpoint image 32d that has been projected and transformed to the display coordinates on the corresponding screen or display.

This allows the subject to recognize the direction of the virtual viewpoint, as well as the angle of view and appearance from the virtual viewpoint by looking at the virtual viewpoint image. If necessary, the user may change the display settings of the marker to flip the virtual viewpoint image horizontally, displaying the virtual viewpoint image so that the subject himself or herself appears as if in a mirror image.

[Fifth Modification of the First Embodiment]
When there are a large number of virtual viewpoints, such as 10 or more, and when there are multiple virtual viewpoints that move at high speed, displaying the markers as is can be distracting to the subject. In this modified example, virtual viewpoints that are close to each other are displayed together, and the markers of virtual viewpoints that move at high speed are displayed so as to be difficult to visually recognize.

The virtual camera information storage unit 303 transmits multiple virtual camera paths at the same time code to the marker coordinate calculation unit 304. The marker display change processing flow in the marker coordinate calculation unit 304 is shown in FIG. 14. In S201, multiple coordinates and speeds are acquired as virtual camera information. In S202, the distance threshold (radius 2 m) and quantity threshold (= 10) described in the marker display settings are acquired, and it is determined whether there are 10 or more virtual cameras within 2 m of each other among the multiple received virtual cameras. If the answer is No, proceed to S204. If the answer is Yes, in S203, the center of gravity of the set of virtual cameras within 2 m of each other is determined as a representative point, and a marker is displayed only for the representative point. At this time, the color or marker type may be changed to indicate that it is a representative point, or the number of elements in the set may be displayed in text. In S204, the amount of transparency of the virtual camera for a speed of 2 to 10 m/s is determined by linearly changing it based on the minimum speed threshold (2 m/s), maximum speed threshold (10 m/s), and maximum transparency (90%) described in the display settings of the marker. Note that the amount of transparency is 0% for 0 to 2 m/s, and 90% for 10 m/s or more. The method of determining the amount of transparency is not limited to this. Also, the marker display may be changed so that the faster the virtual camera speed is, the harder it is to see by changing the brightness, saturation, etc., not limited to transparency.

This allows the subject to concentrate on their performance even when there are a large number of virtual viewpoints.

Another effect of this modified example is that, when there is a delay between photographing the subject and displaying the marker, the position shift caused by the delay of a quickly moving virtual camera is made less noticeable by making the marker transparent. On the other hand, a virtual camera that does not move quickly has relatively little position shift caused by the delay, making it easier to see without being transparent.

Second Embodiment
In the first embodiment and its modified example described above, the position of the marker visually recognized by the subject is configured so that the subject's line of sight (line of sight) is aligned with the position of the virtual camera, which is one of the virtual camera parameters. The content represented by the marker is not limited to the above.

In this second embodiment, the displayed marker is related to the virtual camera parameters, but rather than directing the subject's gaze towards the virtual camera, it indicates the position to which the subject wishes to direct their gaze (the position to which the subject wishes to direct their gaze). With this configuration, it is possible to generate an image of the subject's gaze directed towards the marker, captured by the virtual camera.

Figure 5(a) is a diagram for explaining this embodiment. As described above, the marker 51 in the figure indicates the position at which the subject wishes to make eye contact (the position at which the subject is desired to make eye contact). The type of marker in this embodiment is different from that in the first embodiment. Therefore, the type of marker is switched by the marker type parameter information transmitted by the marker information transmission unit 405 of the user terminal 4.

Reference numbers 52a and 52b in the figure are virtual cameras. Although the subject 24 cannot actually be seen, an image can be generated by capturing an image of the subject 24 with its gaze focused on the marker 51 using the virtual camera.

Figure 5(b) is a diagram showing a schematic diagram of the positional relationship between the virtual camera and the markers. In this embodiment, the positional relationship between the virtual camera and the markers is defined in advance as a setting file in the marker information transmission unit 405. In the setting file, for example, the virtual camera path 55 and the corresponding marker 53 are defined as angles 57 formed with an extension line of the subject's face. In this way, it is possible to create a free viewpoint video by capturing video of a subject directing their gaze at a certain target from various angles.

[Third embodiment]
In the first embodiment and the modified example, and the second embodiment, the marker is configured to be visible to the subject by projecting it onto a screen using a projector. However, when generating a free viewpoint video in a temporary studio or at a shooting location, it may not always be possible to configure a screen. Therefore, in this embodiment, an indicator is provided on the mounting member on which the camera is mounted, and the position of the virtual camera is recognized by the subject by the way the indicator is displayed.

The third embodiment will be described with reference to FIG. 6. Cameras 61a to 61c are mounted on mounting members 62a to 62c that are arranged to surround the subject in real space. Indicators are provided on these mounting members, and the lighting position in the height direction, blinking speed, display color, etc. are controlled by the marker control unit 503. Reference numeral 64 in the figure indicates the position of the virtual camera, which cannot be seen directly by the subject. However, the subject can roughly recognize the position of the virtual camera by the display of the indicator. For example, when indicators 63a and 63b are currently lit, the subject can recognize that the position of the virtual camera is between real cameras 61a and 61b and roughly at the height of the indicators. Furthermore, when indicator 63a blinks, the subject can recognize that the position of the virtual camera is closer to real camera 61a.

In this way, by configuring an indicator on the mounting member of the real camera, it is possible to configure it so that the subject can recognize the approximate virtual camera position.

When this third embodiment is adopted, the marker output device 5 in FIG. 1 controls the display of the indicator.

[Fourth embodiment]
In the above-described embodiments (including the modified examples), the marker is presented by a marker that two-dimensionally displays the position of the virtual camera in real space. In the present embodiment, a configuration will be described in which the position of the virtual camera itself is presented three-dimensionally so as to be visible to the subject.

The marker in this fourth embodiment is, for example, a flying device capable of autonomous flight with a hovering function, such as a drone. Any marker may be used in this embodiment as long as it is powered, capable of flight, and has a configuration in which coordinate values can be controlled. In other words, the marker in this embodiment receives the virtual camera position coordinates output by the marker control unit 503 as world coordinate values in the real space, and flies to that position. Note that there may be multiple markers.

For example, as shown in FIG. 7, the marker is configured as a flying object 71 having a mark (in the figure, an arrow display 72 and a marking 73 on the side) that indicates the line of sight of the virtual camera.

By configuring in this way, the subject can recognize the position of the virtual camera. In addition, when there are multiple markers and a free viewpoint video is generated, the markers may be included in the angle of view of the virtual camera. However, since the position of the virtual camera and the shape of the markers are known in advance, it is possible to erase the markers and generate a free viewpoint video as an image. At this time, the coordinates of the markers are projected onto each of the cameras 21, and the projected coordinates on the camera video are excluded from the model generation and rendering process, thereby preventing the image of the marker from affecting the model generation and rendering. In addition, the area targeted by the position may be erased by smoothing and synthesizing with the same texture as the surroundings, or the marker may be erased by displaying it in the same color as the background in advance.

[Modification of the fourth embodiment]
In the fourth embodiment described above, a drone was used, but the position and attitude of the virtual camera may be displayed using other devices capable of three-dimensional display. For example, there are aerial displays that emit light in the air by exciting plasma, 3D displays using anaglyphs and polarized shutters, and glasses-type or contact lens-type devices such as Hololens capable of AR display. In either case, a three-dimensional model of the marker is projected onto the coordinates of the left and right eyes of the subject in the virtual space, and an image to be displayed on each device is generated, thereby allowing the virtual camera to be displayed three-dimensionally to the subject. In addition, when there are multiple virtual cameras, the markers displayed for each device may be controlled to make it easier to recognize a specific virtual camera for each subject.

This allows the subject to see the position and attitude of the virtual camera in three dimensions while avoiding the risk of coming into contact with the drone.

[Fifth embodiment]
In the above-mentioned embodiment, the markers show information related to the line of sight of the subject. When the subject is performing in real space, there are cases where the line of sight does not necessarily need to be aligned with something. In such cases, when the subject stands at a certain position in real space and performs a performance, it may be desired for the subject to recognize the position. In this fifth embodiment, the display of the markers is projected onto the floor. The subject can recognize the position linked to the virtual camera based on the content displayed on the floor.

The configuration of this embodiment is shown in Figure 8. The projector 83 is attached to a ball head 84, and the projection position can be controlled by the pan angle, tilt angle, and zoom magnification. The display content 86 includes the standing position, movement direction, and the like. The display position and display content of the marker are controlled by the virtual camera position and marker position using a setting file defined in the marker information transmission unit 405. With this configuration, it is possible to generate a free viewpoint video in which, for example, the movement from one standing position to another is captured by a virtual camera.

In the embodiments described so far, the subject visually recognizes the position of the marker when shooting, but this is not necessarily limited to this configuration. For example, as described above, by superimposing the marker position on the archive video, it is possible to display the relationship between the virtual camera and the line of sight for viewers as well, rather than being limited to a configuration in which it is used only during shooting.

The above can be achieved by preparing multiple streams for each virtual viewpoint. Furthermore, by preparing a video that overlooks the real space and configuring the marker positions to be interactive, the user can switch the camera position and view the free viewpoint video.

In addition, images captured by multiple cameras can be stored as archived footage, and when an operator generates a free viewpoint video at a later date, the subject's facial position (line of sight) can be detected and a marker can be displayed at a position on the extension line in the virtual space. This configuration improves the operator's operability when determining the angle of view of the free viewpoint video.

In addition, the markers in the embodiments described so far will naturally appear in the images captured by each camera, and may be a hindrance in the generated free viewpoint video. However, since the display position, color, shape, etc. of the marker in virtual space are known, it is possible to erase the marker by smoothing the area corresponding to that position with the same texture as the surrounding area and performing a synthesis process. Alternatively, this problem can be easily solved by displaying the marker in the same color as the background in advance, as described above.

Other Examples
The present disclosure can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention.

1...Multiple viewpoint image storage unit, 2...Subject information storage unit, 3...Image processing device, 301...Virtual viewpoint image generation unit, 302...Virtual camera path calculation unit, 303...Virtual camera information storage unit, 304...Marker coordinate calculation unit, 305...Display image generation unit, 306...Device communication unit, 4...User terminal, 401...Device communication unit, 402...Image display unit, 403...Virtual camera path instruction unit, 404...User information transmission unit, 405...Marker information transmission unit, 5...Marker output device, 501...Device communication unit, 502...Control data generation unit, 503...Marker control unit, 504...Display data output unit

Claims (18)

1. A processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, comprising:
A plurality of imaging means for imaging a space including a subject;
A means for identifying a position of the virtual viewpoint;
and a presentation means for providing a presentation for determining a direction of the subject in the space based on the position of the virtual viewpoint . the plurality of photographing means and a screen for projection are arranged so as to surround the subject,
2. The processing system according to claim 1 , wherein the presentation means projects information for determining a direction of the subject onto the screen. 3. The processing system according to claim 2 , wherein the presentation means projects information for determining a direction of the subject at a position where an extension line connecting the subject and the virtual viewpoint intersects with the screen. The processing system according to claim 2, wherein the presentation means projects information for determining the orientation of the subject at a position where the screen intersects with a line that is shifted in a predetermined direction from an extension line connecting the subject and the virtual viewpoint. 5. The processing system according to claim 1 , wherein the presentation means presents information for determining a direction of the subject in a size according to a distance between the subject and the virtual viewpoint. the plurality of photographing means are attached to a mounting member having an indicator that lights up a position set with respect to a position in a height direction, and are arranged so as to surround the subject,
2. The processing system according to claim 1 , wherein the presenting means illuminates a position at which the subject should look. 2. The processing system according to claim 1 , wherein the presenting means is means for positioning a flying device at a position at which the subject's line of sight is directed. The processing system according to claim 7 , wherein the flying device is provided with a mark indicating the line of sight of a virtual viewpoint. The processing system according to claim 1, characterized in that the presentation means is a means for controlling the pan and tilt angles of a projector mounted on a mounting member capable of controlling the pan and tilt angles, and projects information for determining the direction of the subject onto a floor according to a position to which the subject's line of sight is to be directed. 10. The processing system according to claim 1, wherein the presentation means presents information for determining a direction of the subject at the position of the virtual viewpoint. The processing system according to any one of claims 1 to 9 , characterized in that the presentation means presents information for determining the orientation of the subject at a position different from the position of the virtual viewpoint and in a position in the direction in which the subject is facing. 12. The processing system according to claim 1 , wherein the orientation of the subject is a facial orientation of a person who is the subject. 12. The processing system according to claim 1 , wherein the direction of the subject is a direction of a line of sight of a person who is the subject. a user terminal having a designation means for designating the virtual viewpoint and a display means for displaying the virtual viewpoint image;
an image processing device for generating the virtual viewpoint image;
The processing system according to claim 1 , further comprising: a presentation device having the presentation means. 15. The processing system according to claim 1 , wherein the presentation means presents a virtual viewpoint image. 16. The processing system according to claim 1, wherein the presentation means controls a color or a shape in accordance with the number or speed of the virtual viewpoints. 1. A method for controlling a processing system for generating a virtual viewpoint image representing a scene from a virtual viewpoint, comprising:
A step of photographing a space including a subject by a plurality of photographing means;
A step of identifying a position of the virtual viewpoint;
a presentation step of providing a presentation for determining a direction of the subject in the space based on a position of the virtual viewpoint ;
A control method comprising the steps of: A program for causing a computer to execute each step of the method according to claim 17 .