JP6910405B2 - Information processing equipment, information processing methods, and programs - Google Patents

Description

The present invention relates to a system for generating content.

Attention is being paid to a technology that synchronously shoots a subject from multiple viewpoints with a plurality of cameras arranged at different positions and generates free viewpoint content (virtual viewpoint content) using the images of the multiple viewpoints obtained by the shooting. .. In Patent Document 1, in a system in which free viewpoint content is generated and stored based on images of a plurality of viewpoints, an image receiving device reads out video data based on the generated viewpoint information to obtain arbitrary free viewpoint content. The technology to be acquired is described.

The virtual viewpoint content can be generated by combining three-dimensional models of one or more subjects. The method of generating a three-dimensional model of a subject photographed by a plurality of cameras may be the visual volume crossing method shown in Non-Patent Document 1. In the visual volume crossing method, the space to be measured taken by a plurality of cameras is divided by a small cube or a rectangular parallelepiped (hereinafter, referred to as a "voxel"). Then, each voxel is geometrically transformed and projected onto the camera image, and it is determined in the camera image whether or not the voxel is projected within the silhouette of the object to be modeled. Then, the voxels determined to be projected in the silhouette in all the camera images are registered as voxels constituting the target object, and the aggregate of the registered voxels is output as a three-dimensional model.

Japanese Unexamined Patent Publication No. 2015-204512

"Virtual View generation for 3D Digital Video", IEEE MULTIMEDIA, Vol.4, No.1, pp.18-26, 1997

In the generation of virtual viewpoint contents using the 3D model, it is premised that the 3D model can be generated. However, when trying to generate the virtual viewpoint content at a certain point in time, it may not be possible to generate a part of the three-dimensional model and the virtual viewpoint content cannot be generated. In this case, it takes time to determine that the three-dimensional model cannot be generated, and it takes time for the user to recognize that the virtual viewpoint content cannot be generated, which reduces convenience. There was a problem.

The present invention has been made in view of such a problem, and an object of the present invention is to provide a technique for improving user convenience when generating / displaying content.

The information processing device according to one aspect of the present invention is determined by a determination means for determining the time when the virtual viewpoint content can be generated and a determination means by the determination means based on the state of the material data used for generating the virtual viewpoint content. An output means for outputting data for displaying information that can distinguish between a time range in which the virtual viewpoint content can be generated and a time range in which the virtual viewpoint content cannot be generated based on the time specified. Has .

According to the present invention, it is possible to improve the convenience of the user when generating / displaying the content.

The figure which shows the configuration example of a system. The figure which shows the arrangement example of the image pickup part. The figure which shows the example of the screen displayed on the operation part. The figure which shows the example of the correspondence relationship between the type of material data and the acquired data group. The figure which shows the example of the time range management information. The figure which shows the display example of the effective time range in the time indicator part. The figure which shows the example of the image display at the time of straddling two discontinuous effective time ranges. The figure which shows another display example of the effective time range in a time indicator. The figure which shows another configuration example of a system.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each of the embodiments described, a three-dimensional model of one or more subjects is generated from a plurality of images captured by a plurality of imaging devices (cameras), and free viewpoint content (virtual) is generated based on the three-dimensional model. A system for generating (viewpoint content) will be described. Generating a 3D model from images captured by multiple cameras generally requires a lot of computational complexity and time. In particular, when generating a high-resolution three-dimensional model, it is necessary to reduce the size of voxels, the number of voxels increases accordingly, and the number of determinations as to whether or not the voxels are projected in the silhouette also increases. Therefore, in the present embodiment, a material data generation device such as an image processing server that generates and saves a three-dimensional model and texture data holding color information of each voxel of the three-dimensional model in advance as material data is provided. It shall be used to configure the system. According to such a system, it is possible to prevent the process of generating the three-dimensional model from being performed every time the virtual viewpoint content is generated, and to improve the efficiency of the process.

For example, a low-resolution 3D model with a relatively short processing time is generated for live distribution, and a high-resolution 3D model is generated for replay images of highlight scenes. Three-dimensional data with step resolution can be generated. At this time, a texture, which is one of the material data, can also be generated for each of the high resolution and the low resolution. Further, for example, polygon data may be generated. That is, a plurality of different material data can be generated for the same subject.

In this system, for example, a video generator that generates virtual viewpoint content from material data acquires material data that has been generated and saved in advance from the material data generation device when generating virtual viewpoint content, and video. Can be generated. At this time, the video generator cannot know in advance whether or not the material data necessary for generating the virtual viewpoint content exists. For example, it may be difficult to generate a three-dimensional model depending on the quality of the image captured by the imaging device, the operating status of the imaging device, and the like. That is, there may be a case where the material data for a specific subject does not exist at a certain time. Further, as described above, when a plurality of different material data are generated for the same subject, the time from completion of generation to saving is different for each material data. That is, it is possible that the generation of some material data has not been completed at the time requested by the user. Further, in soccer half-time or the like, it is possible that all or a large number of imaging devices stop imaging, and material data is not stored at all in that time range.

Therefore, when the video generation device generates the virtual viewpoint content at the time specified by the user, the video generation device asks the material data generation device whether each material data required at the specified time is saved, and the virtual viewpoint content. Can be generated. If a part of the necessary material data is not saved in the specified time, the virtual viewpoint content cannot be generated in the specified time. That is, after the time is specified by the user, the user is notified that the virtual viewpoint content cannot be generated within the specified time, which may not be convenient for the user.

Therefore, in the system according to the present embodiment, information on the time when the material data is available is generated, and when the user generates the virtual viewpoint content, a set of material data necessary for generating the virtual viewpoint content. Notifies the video generator of information on the available period. According to this, the user can clearly recognize the time interval in which the virtual viewpoint content can be generated, so that it is necessary to specify the time and check each time whether the material data at that time exists. It disappears. In the following, the system for executing such processing and the details of the processing will be described in the case where the state of the competition at the stadium (stadium) is imaged and used for live distribution or generation of virtual viewpoint video for replay. explain. However, the present invention is not limited to this, and a virtual viewpoint image may be generated based on an image captured in, for example, a concert hall, a martial art such as sumo, or a shooting studio. It should be noted that the embodiments described below are merely examples, and any configuration and procedure capable of performing the same processing can be used within the scope of the idea of the present invention.

(System configuration)
FIG. 1 shows a configuration example of the system according to the present embodiment. This system includes, for example, an imaging unit 1, a synchronization unit 2, a material data generation unit 3, a storage unit 4, a time information management unit 5, a video generation unit 6, a video output unit 7, a data group selection unit 8, and an operation unit. It is configured to include 10. The system may be composed of one electronic device or a plurality of electronic devices.

In this system, a plurality of imaging units 1 perform imaging in synchronization with each other with high accuracy based on the synchronization signal by the synchronization unit 2. As shown in FIG. 2, the plurality of imaging units 1 may be installed so as to surround the ground or the subject. Each of the imaging units 1 transmits the captured image to the material data generation unit 3. The material data generation unit 3 generates material data used when generating a virtual viewpoint image by using images captured from a plurality of viewpoints received from the image pickup unit 1, and outputs the material data to the storage unit 4. At this time, the material data generation unit 3 outputs the time information based on the imaging time of the captured image used for generating the material data to the storage unit 4 together with the material data. The time information may be related to the time, but may be arbitrary information related to the time such as a frame number.

The material data includes, for example, one or both of a subject three-dimensional model, which is voxel data indicating the three-dimensional shape of the subject, and a subject texture for coloring each voxel of the subject three-dimensional model. Further, the material data may include one or both of a background three-dimensional model which is polygon data for projecting a background other than the subject and a background texture image to be attached to the polygon of the background three-dimensional model. Further, the material data may include at least one of a subject polygon model for reducing the amount of data transfer and a texture for the subject polygon to be attached to the subject polygon model. In addition, the material data may include subject position information that can be used for analysis of movement of athletes and the like. The three-dimensional model of the material data can be generated by using, for example, the visual volume crossing method, but other methods may be used, and the three-dimensional model may be generated by any other method. In addition, other material data can be generated by any method.

The storage unit 4 stores the material data of one or more types of information acquired from the material data generation unit 3 in association with the time information. By storing the various material data and the time information in association with each other in this way, the storage unit 4 can identify and read the corresponding material data based on the time information and the type information of the material data. The storage unit 4 outputs the type information and the time information of the material data for which storage has been completed to the time information management unit 5.

The time information management unit 5 generates or updates the time range management information based on the type information and the time information of the material data acquired from the storage unit 4. The time range management information records the time range continuously stored in the storage unit 4 for each type of material data. This time range can be defined as a combination of the start time and the end time of the time range in which the material data is continuously stored based on the time information acquired from the storage unit 4. For example, when the material data that has not been saved up to that point is newly saved, the time information management unit 5 saves the time information corresponding to the material data as the start time and the end time. In addition, when the time information management unit 5 saves new data corresponding to a new time within a predetermined period from the time indicated by the time information corresponding to the data, the material data has already been saved. Update the end time of the data and expand the time range. Then, when a predetermined period elapses from the time indicated by the time information of the material data for the material data already stored, the time information management unit 5 ends updating the information in the time range of the material data. Therefore, the time information management unit 5 starts the time when new data is saved together with the time information indicating the time when a period equal to or longer than the latest end time has elapsed from the already stored material data. Generates information on a new time range as the time. In this case, time range management information that specifies a plurality of time ranges is generated for one type of material data. The predetermined period here may be, for example, a period for one frame, but may be an arbitrary period, for example, a period longer than that.

The image generation unit 6 acquires material data from the storage unit 4 based on an instruction from the connected operation unit 10 and generates a virtual viewpoint image. The generated virtual viewpoint video is output and displayed on, for example, the display unit 11 and the video output unit 7 of the operation unit 10.

The operation unit 10 provides a user interface for receiving input from the user, such as a combination of a touch panel, a button / keyboard, and a display, and presenting information to the user. The operation unit 10 includes, for example, a display unit 11, a video type selection unit 12, and a time indicator unit 13 as its functions. The operation unit 10 displays a screen as shown in FIG. 3, for example, and accepts user operations. In FIG. 3, the display unit 11 displays the generated virtual viewpoint image. The video type selection unit 12 selects the video type of the virtual viewpoint video to be output. The time indicator 13 specifies the time of the virtual viewpoint image to be generated. The video type selection unit 12 accepts selection of types such as "live distribution", "low image quality replay", "high image quality replay", "for mobile terminals", and "player position analysis". Here, "live distribution" is selected when generating a virtual viewpoint image for immediate distribution, which is allowed to have low image quality. "Low-quality replay" is selected when generating a virtual viewpoint video for replay playback with the same image quality as live distribution. In "live distribution" and "low image quality replay", a low resolution 3D model is used to generate a virtual viewpoint image. "High-quality replay" is selected when generating a high-quality virtual viewpoint image that is premised on repeated viewing, such as a replay of a highlight scene. In "high image quality replay", a low resolution 3D model is used to generate a virtual viewpoint image. "For mobile terminals" is selected when generating a virtual viewpoint video with a small amount of data. In "for mobile terminals", for example, a virtual viewpoint image is generated using a three-dimensional model such as a polygon instead of a voxel. "Player position analysis" is selected, for example, to check how the players are moving on the field. In the "player position analysis", only the position information is used, and the three-dimensional model of the player who is the subject is not used. The selected video type information is input to the data group selection unit 8 via, for example, the video generation unit 6.

The data group selection unit 8 selects a data group that is a combination of material data necessary for generating a virtual viewpoint image of the image type based on the input image type. For example, the data groups selected for each of the above-mentioned video types may be a combination as shown in the table shown in FIG. For example, when "high image quality replay" is selected, a high-resolution subject three-dimensional model, a high-resolution subject texture, a background three-dimensional model, and a high-resolution background texture are selected as the corresponding data groups. When "player position analysis" is selected, it is sufficient if only the arrangement and movement of the players can be specified, so the subject position information and the background three-dimensional model are selected as the corresponding data groups. The data group selection unit 8 notifies the video generation unit 6 of the selected data group. Then, the video generation unit 6 inquires the time information management unit 5 about the effective time range 20 indicating the time range in which all the data groups are stored in the storage unit 4 based on the notified data group.

As shown in FIG. 5, the time information management unit 5 manages the time range management information that specifies the time range for each of the material data stored in the storage unit 4. When the time information management unit 5 receives an inquiry for designating a data group from the video generation unit 6, it identifies a time range in which all of the designated data group is stored and returns it to the operation unit 10.

For example, when the image type is a high-quality data group, the time information management unit 5 stores all of the high-resolution subject three-dimensional model, the high-resolution subject texture, the background three-dimensional model, and the high-resolution background texture. Identify the time range you are in. That is, in this case, the time information management unit 5 specifies the time range 20h, which is the logical product of the time range of the high image quality data group. Further, when the video type is the player position analysis data group, the time information management unit 5 specifies the time range 20p corresponding to the time range in which the player position information is stored. When the video type is live distribution, as shown in FIG. 5, the latest time information 21 which is the time when the necessary data groups are prepared is specified as the effective time range 20.

The specified effective time range 20 is notified to the operation unit 10 via the video generation unit 6, and the information is displayed on the time indicator unit 13. When a plurality of time ranges are specified as effective time ranges such as the time range 20h and the time range 20p, all or a part of the plurality of time ranges (for example, a predetermined number of time ranges) is the operation unit 10. Will be notified.

When the user specifies the time of the video for generating the virtual viewpoint within the effective time range 20, the designated time is transmitted to the video generation unit 6. At this time, the viewpoint information such as the viewpoint position / posture of the virtual viewpoint to be drawn and the angle of view thereof is determined by using the 3D mouse, and the viewpoint information is also transmitted to the image generation unit 6. It is not necessary to use a 3D mouse, and the viewpoint information may be determined by accepting an operation through a GUI or an operation by a dedicated operation unit, for example. The video generation unit 6 requests and acquires the necessary material data from the storage unit 4 based on the obtained designated time and the preset data type group. At this time, it has been confirmed in advance that the data group at the designated time requested by the video generation unit 6 is stored in the storage unit 4 based on the information in the effective time range 20. Therefore, the video generation unit 6 can surely acquire the target data without confirming the existence of the data. The image generation unit 6 generates a virtual viewpoint image based on the obtained material data and the viewpoint information, and outputs the virtual viewpoint image to the display unit 11 and the image output unit 7 of the operation unit 10.

With such a configuration, it is possible to notify the user of information in a time range in which an appropriate virtual viewpoint image can be generated using the material data stored in the storage unit 4. Then, since the user can specify the time at which he / she wants to generate the virtual viewpoint video from the notified time range, the virtual viewpoint video cannot be generated at the designated time. This makes it possible to improve the convenience of the user when generating / displaying the virtual viewpoint video.

(Example of effective time range display and video playback control)
Subsequently, an example of a method in which the time indicator 13 displays the effective time range 20 and accepts the time designation by the user will be described. As an example, the time indicator 13 displays the effective time range 20 by using the seek bar 13a, which is a bar-shaped graphical interface (GUI), as shown in FIGS. 6A to 6B. Here, as an example, an example in which two effective time ranges of the first effective time range 20a and the second effective time range 20b are displayed will be shown. Note that the example described here is only an example, and an arbitrary display method capable of distinguishing the effective time range from the other time range and specifying an arbitrary time within the effective time range can be used. For example, a configuration may be used in which the time is specified by a counter value using a physical jog dial or the like. In this case, the display of the effective time range may be a list of data indicating one time range according to the start time and the end time.

In the example of FIG. 6A, the time indicator 13 displays the two acquired effective time ranges 20a and 20b in one seek bar 13a as effective ranges by changing the color of a part of the seek bar 13a. .. In this case, the entire range of the seek bar 13a corresponds to the length of the entire expected content, and in the case of sports such as soccer, the time range from the start of the match to the end of the match is displayed by the entire seek bar 13a.

In the example of FIG. 6B, the time indicator 13 displays the time range in which the desired virtual viewpoint image cannot be generated shorter than the length corresponding to the length of the actual time range. In the example of FIG. 6B, the period between the first effective time range 20a and the second effective time range 20b is displayed, but this period does not necessarily have to be displayed. At this time, as shown in FIG. 6B, numerical values indicating the start time and end time of the effective time range 20 may be displayed on the seek bar 13a.

The example of FIG. 6A is useful in that it makes it easy for the user to recognize the time range of the entire content, the ratio of the effective time range 20 in the time range, and the current playback position. On the other hand, for example, when the ratio of the effective time range 20 to the time range of the entire content is small, it is useful in that the user's operation can be facilitated by displaying as in the example of FIG. 6B. Is. As described above, since each display method has different advantages, these two display methods are configured to be switchable, and the time indicator 13 selects which of these display methods to use by the user. You may accept it. Further, the time indicator 13 may automatically select which display method to use based on parameters such as the ratio of the effective time range 20 to the length of the time range of the entire content. ..

By moving the cursor 14 to any of the effective time ranges 20a and 20b displayed by the time indicator 13, the user sets the time corresponding to that position as the time for generating and displaying the virtual viewpoint image. Can be specified. The image generation unit 6 acquires material data corresponding to the designated time from the storage unit 4, generates a virtual viewpoint image, and outputs the virtual viewpoint image to the image output unit 7 and the operation unit 10. The cursor 14 may be configured so that when the time is specified, it can be moved only in the effective time ranges 20a and 20b on the seek bar 13a, that is, a time other than the effective time range cannot be selected. An example in which such a configuration is made will be described with reference to FIG. 6 (c). FIG. 6C shows an example in which an interface for dragging and moving the cursor 14 on the seek bar 13a by the pointer 15 is used. As shown in FIG. 6C, the user drags the cursor 14 and moves the pointer 15 to the positions 15a and 15b within the first effective time range 20a to move the cursor 14 to the positions 14a and 14b. Can be moved. On the other hand, even if the user moves the pointer 15 to the position 15c while dragging the cursor 14, the cursor 14 is fixed at the position 14c which is the end time of the first effective time range 20a. That is, the time indicator 13 does not allow the cursor 14 to be moved outside the first effective time range 20a. On the other hand, when the user moves the pointer 15 within the second effective time range 20b or from the second effective time range 20b to the position 15d within a predetermined range while dragging the cursor 14, the cursor 14 moves to the second position. It is moved to the start time 14d of the effective time range 20b of. In this way, by preventing the cursor 14 from moving to the time range between the first effective time range 20a and the second effective time range 20b, the time at which the virtual viewpoint image cannot be generated is selected. Can be prevented. The time indicating unit 13 may be configured so that a portion other than the effective time range 20 can be selected or specified. Further, instead of the configuration in which the user cannot specify the time other than the valid time range, even if the user specifies the time other than the valid time range, the configuration in which the designation is invalidated may be adopted.

When the user specifies one time within the effective time range 20, the video generation unit 6 may generate the virtual viewpoint video only for the designated time, but the virtual viewpoint video may be generated in consecutive frames after the designated time. May be generated and played back. At this time, when the reproduction in continuous frames from the time within the effective time range 20a is started, if the reproduction is continued as it is after the end of the effective time range 20a, the effective time range between the effective time range 20b and the effective time range 20b is continued. Reproduction may continue even in an outside time range. Therefore, when the video generation and playback are completed until the end time of a certain effective time range (first effective time range 20a), the video continues from the start time of the next effective time range (second effective time range 20b). It is possible to generate and play back a virtual viewpoint image. According to this, since the generation and output of the virtual viewpoint video are not stopped, the continuous virtual viewpoint video can be viewed without requiring the user to operate. Further, when the virtual viewpoint video is reproduced across a plurality of effective time ranges, as shown in FIG. 7A, the time during which the virtual viewpoint video to be reproduced is discontinuous when the effective time range is switched is related. A display screen 22 may be inserted to indicate that it is a thing. Alternatively, as shown in FIG. 7B, at the start of playback in the effective time range after switching, information 23 indicating that the virtual viewpoint video to be played back is related to a discontinuous time is superimposed and displayed. You may. With these, it is possible to show the user that the frame being played is related to a time discontinuous with the frame being played so far. If the reproduction is completed up to the end time of the last effective time range among the plurality of effective time ranges, the reproduction may be stopped at that point. It should be noted that a setting for repeated reproduction may be provided, and reproduction may be started again from the start time of the first effective time range according to the selection of the setting.

Further, when real-time video should be generated, for example, when "live distribution" is selected, the video generation unit 6 acquires the latest time information 21 and the material data of the time corresponding to the latest time information 21. Can be obtained from the storage unit 4 to generate a virtual viewpoint image. In this case, the time indicated by the latest time information 21 does not necessarily have to be displayed on the time indicator 13. Also, in the case of live distribution, it is only necessary to have the latest time video, and the time information does not need to be specified. For example, when a type such as live distribution is selected as the video type, the video generation unit 6 requests the time information management unit 5 to acquire the "latest time" data group without specifying a specific time. conduct. The time information management unit 5 identifies the latest time when the requested data group is already stored, reads the data group at that time from the storage unit 4, and transmits the data group to the video generation unit 6. The time information management unit 5 may transmit an instruction to the storage unit 4 so as to transmit the data group to the video generation unit 6 without reading the data group from the storage unit 4. The image generation unit 6 generates a virtual viewpoint image using the obtained material data group. At this time, the video generation unit 6 does not need to explicitly acquire the latest time information, but when it is desired to display the time information on the display unit 11 of the operation unit 10, for example, the time information is input together with the material data group. It may be configured to acquire.

(Modification example)
The above configuration and processing are merely examples, and various types of device configurations may be used, and similar processing may be performed with different procedures and display contents. Further, one functional block shown in FIG. 1 may be shared and realized by a plurality of electronic devices, or one or more functional blocks may be realized by one electronic device. For example, in FIG. 1, the material data generation unit 3 and the storage unit 4 are shown as independent functional blocks, but these may be contained inside one and the same processing device, and these functional units and the video The generation unit 6 may be included in one device to serve as a device such as a virtual viewpoint video distribution server. It should be noted that at least a part of the functional blocks related to this system can be realized by executing a program stored in a memory such as ROM or RAM by a processor such as a CPU or MPU. For example, the operation unit 10 can be realized by a graphical user interface (GUI) that executes and displays a predetermined application program on a processor in a personal computer (PC), a tablet, a smartphone, or the like. Further, the video generation unit 6 and the operation unit 10 may be realized by executing an application program on a processor in a PC, a tablet, and / or a smartphone.

Further, in the system of FIG. 1, there is a configuration example of a system in which a series of processes from imaging by the imaging unit 1 to generation of virtual viewpoint contents by the video generation unit 6 under the operation of the operation unit 10 are collectively performed. It is shown, but not limited to this. For example, a system in which imaging, storage of material data, and generation of time range management information are performed, and a system in which virtual viewpoint video is generated may be prepared separately. That is, when generating the virtual viewpoint video, it is sufficient that the material data and the time range management information are stored and there is an information providing device capable of providing such information, and there is a mechanism for generating such information in the system. You don't have to.

In the above description, the time information management unit 5 generates or updates the time range management information when the material data is saved in the storage unit 4, but the time range management information is not always generated. good. For example, each time the time information management unit 5 receives an inquiry for an effective time range from the video generation unit 6, the time information management unit 5 extracts the time information of the material data stored in the storage unit 4 and generates information on the effective time range. Then, it may be transmitted to the image generation unit 6. According to this, accurate time information can be extracted based on the material data stored at that time. Further, the time information management unit 5 is effective regarding the material data group for generating the virtual viewpoint image of the image type specified by the inquiry in response to the inquiry of the effective time range from the image generation unit 6. The time range is specified, but it is not limited to this. For example, the time information management unit 5 generates data in the effective time range for each of the plurality of data groups in advance, and responds to an inquiry from the video generation unit 6 to generate the valid data in advance for the corresponding type. Information in the time range can be transmitted to the video generation unit 6. Further, the time information management unit 5 does not generate the time range management information for each material data, but generates the time range in which all the data included in the material data group for each type is stored as the time range management information. You may. Further, the time information management unit 5 may generate time range management information in a format other than this. Further, the time information management unit 5 notifies the video generation unit 6 of the logical product of the time range indicated by the time range management information for the designated material data group as the effective time range 20, but this is not necessarily limited to this. do not have. For example, the time information management unit 5 transmits the time range management information for each material data to the video generation unit 6, and the video generation unit 6 or the operation unit 10 calculates the logical product of the time range indicated by the information. The calculation result can be specified as an effective time range. Further, the video generation unit 6 or the operation unit 10 specifies a plurality of effective time ranges for each of the plurality of video types, and displays the plurality of effective time ranges in parallel on the time indicator unit 13 of the operation unit 10. The process may be executed. Further, although the operation unit 10 acquires the time range management information via the video generation unit 6, the time range management information may be directly acquired from the time information management unit 5.

Further, in the above description, the operation unit 10 includes the video type selection unit 12, and the video type selection unit 12 selects the video type, but the configuration is not necessarily such. For example, the video type may be automatically determined according to the type and performance of the video generation unit 6 and / or the operation unit 10. For example, the video type is automatically determined based on the authority of the user who operates the operation unit 10. The system may be configured to be selected for. When the video type is determined by the type and performance of the video generation unit 6, the data group selection unit 8 does not have to exist. For example, the video generation unit 6 retains information on a fixed combination of data, so that it is not necessary to make an inquiry to the data group selection unit 8. Further, when the video type is determined based on the performance of the video generation unit 6, the data group selection unit 8 may be configured to be connected to the storage unit 4 or the time information management unit 5. In this case, for example, the video generation unit 6 does not specify the data group, but for example, provides information such as the type and performance of the own device or information capable of specifying the type and performance such as model information and identification information of the own device. The accompanying inquiry can be sent to the time information management unit 5. The time information management unit 5 identifies the corresponding material data group based on the information included in this inquiry, and notifies the video generation unit 6 of the information of the effective time range corresponding to the specified data group.

Further, in the above description, it is explained that all the material data is stored in association with the time information, but the present invention is not necessarily limited to this. For example, the storage unit 4 stores only a single piece of data if there is no change in the background three-dimensional model, etc., and makes it valid at all times for inquiries in the valid time range, and each time an acquisition request is made. The same data may be output to the video generation unit 6. Further, such material data may not be the target of the inquiry of the effective time range, and the video generation unit 6 may continuously use the material data once the material data is acquired.

Further, although the configuration in which the video generation unit 6 is connected to the storage unit 4 and the time information management unit 5 to use the material data has been described, the material data may be used by a functional unit other than the video generation unit 6. For example, the system may be configured so that the subject analysis device that measures the position and speed of the subject without generating an image uses the subject position information that is the material data.

Further, in the above description, the video type selection unit 12 of the operation unit 10 has five types of "live distribution", "low image quality replay", "high image quality replay", "for mobile terminal", and "player position analysis". An example of selecting one from the video types has been described, but the video types are not limited to these. For example, further video types such as "Full HD", "4K", "for broadcasting", "for Internet distribution", and "for mobile terminals" may be set, and some of these video types may be omitted. You may. Further, at the time of video generation, the type of material data to be acquired may be directly specified without specifying the video type.

Further, the video type selection unit 12 of the operation unit 10 can be configured so that a video type having a priority such as "high image quality priority replay" can be selected. In the above-mentioned "high-quality replay", only the effective time range in which the high-quality virtual viewpoint image can be generated is acquired, and only the high-quality virtual viewpoint image is generated. On the other hand, in the "high image quality priority replay", the virtual viewpoint image is generated with high image quality in the time range in which the high quality virtual viewpoint image can be generated, and the virtual viewpoint image is generated in the other time range although the image quality is low. In the time range that can be achieved, a virtual viewpoint image is generated with low image quality. According to this, it is possible to prevent the viewer from feeling uncomfortable due to the interruption of the video. In this case, the data group selection unit 8 specifies, for example, a data group having a plurality of priorities such as an essential data group, a high priority data group, and a low priority data group based on a designated video type. , Notify the video generation unit 6 of the specified data group. In the example of "high quality priority replay", the high priority data group can be, for example, a high resolution subject 3D model, a high resolution subject texture, or a high resolution background texture for generating a high resolution image. .. On the other hand, the low priority data group may be a low resolution subject model, a low resolution subject texture, or a low resolution background texture capable of generating a low resolution image. Also, the essential data group can be a background model. It should be noted that what kind of data should be given high priority or low priority is not limited to this. For example, when the capacity of the communication line is small, the low resolution data has high priority and the high resolution data has high priority. Settings may be made that have low priority.

The video generation unit 6 inquires the time information management unit 5 about the effective time range based on these data groups. On the other hand, the time information management unit 5 first obtains a time range in which both the essential data group and the high priority data group are stored, as shown by 20c in FIG. Specified as a high priority range. Further, the time information management unit 5 sets the time range in which the essential data group and the low priority data group can be obtained as the low priority range as shown by 20d in FIG. 8 among the time ranges other than the high priority range 20c. Identify. Then, the time information management unit 5 transmits the specified high priority range 20c and low priority range 20d information to the video generation unit 6. The operation unit 10 displays the received effective time range 20 so as to be distinguishable between the high priority range 20c and the low priority range 20d (for example, in different colors). With such a display, the user mainly specifies a time within the high priority range 20c to generate a high-resolution virtual viewpoint image, but if he / she wants to generate an image of a certain time even if the resolution is low, the low priority range 20d You will be able to make decisions such as specifying the time within. The low priority range may be specified regardless of the high priority range 20c. In this case, in the video generation unit 6 or the operation unit 10, the range in which the high priority range and the low priority range overlap can be specified as the high priority range. Further, the operation unit 10 may display the high priority range and the low priority range in parallel. Further, for example, for a time range that is neither the high priority range 20c nor the low priority range 20d, the display as in the example of FIG. 6A or FIG. 6B can be performed.

The video generation unit 6 makes an inquiry to the time information management unit 5 in which the priority is set for each data type, but the present invention is not limited to this. For example, the video generation unit 6 makes an inquiry to the time information management unit 5 for each priority, and the operation unit 10 or the video generation unit 6 integrates the effective time range 20 obtained for each priority, and the time indicator unit 13 The valid time range 20 integrated into may be displayed.

Further, the effective time range 20 is specified by classifying the high-priority data group for generating a high-resolution video and the low-priority data group capable of generating a low-resolution video, and the video generation unit 6 is based on this. I explained that I will acquire material data, but it is not limited to this. For example, when the model and texture of the subject can be acquired with high resolution, but the background texture can be acquired only with low resolution, it is not necessary to use the one with low resolution even for the subject according to the background texture. In such a case, the image generation unit 6 may be configured to use a background texture having a low resolution. For such a configuration, a combination of priorities is created for each classification of data types such as a three-dimensional model and texture, and the video generation unit 6 gives priority to high-priority material data for each classification of data types. It may be configured to acquire. Further, in the time indicator 13 in this case, the effective time range in which the high-priority ones can be acquired in all types and the effective time range in which at least a part of the low-priority ones can be acquired can be distinguished from each other. It may be displayed.

Further, in the above example, the images captured by the plurality of imaging units 1 are input to the material data generation unit 3, and the storage unit 4 stores the material data generated by the material data generation unit 3. Not limited to this. For example, all the images captured by the plurality of imaging units 1 may be temporarily stored in the storage unit 4, and the material data generation unit 3 may generate material data using the stored images. An example of this configuration is shown in FIG. In the configuration of FIG. 9, the material data generation unit 3 generates material data using a plurality of images stored in the storage unit 4, and generates the generated material data and the time information corresponding to the material data. Output to the storage unit 4. According to this, for example, when the material data generation unit 3 generates high-quality material data, the material data that has been edited / corrected such as manual correction, or the material data that has been edited / corrected has been edited / corrected. Material data based on images can be generated.

With such a configuration, the images captured by the plurality of imaging units 1 are stored in the storage unit 4, so that it is possible to create higher quality material data using a lot of time based on the images captured by the storage unit 4. become. As a result, it is possible to generate a higher quality virtual viewpoint image. Since it costs money to create high-quality material data, high-quality material data may not be created in the entire time range. In such a case, by using the above-mentioned priority, a high-quality virtual viewpoint image can be generated within a time range in which a high-quality virtual viewpoint image can be generated, and in other parts, a high-quality virtual viewpoint image can be generated. It is possible to generate low quality virtual viewpoint video.

In the above-described embodiment, the system for generating the virtual viewpoint video has been described. However, the above-mentioned technology is applied to an information processing system for generating contents other than the virtual viewpoint video, such as free listening point sound. Can be applied. That is, the above discussion can be applied to any information processing apparatus that acquires one or more material data and generates arbitrary contents from the material data. That is, when one or more material data for generating the content exists, the information processing device acquires the information regarding the time corresponding to the material data. Then, the information processing device specifies an effective time range in which the content can be generated based on the information, acquires material data for the time within the range, and generates the content. At this time, the information processing apparatus can display the effective time range so as to be distinguishable from the time range in which the content cannot be generated. In the above-described embodiment, a method of displaying, for example, a time range in which high-priority content can be generated can be distinguished from other time ranges by coloring according to the priority has been described, but this is extended. can do. For example, a method of identifiablely displaying the types / combinations of content that can be generated can be used. For example, a time range in which both virtual viewpoint video and free listening point sound can be generated, a time range in which only virtual viewpoint video can be generated, a time range in which only free listening point sound can be generated, and a time range in which neither can be generated. Can be displayed in an identifiable manner by coloring or the like. Further, the contents may be further classified based on the resolution or the like so that the contents can be displayed in an identifiable manner for each classification. The effective time range for each type of content may be displayed in parallel. In this way, by showing the types of content that can be generated so that the user can identify them, it is possible to improve the convenience of the user when generating the content.

<< Other Embodiments >>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1: Imaging unit, 2: Synchronization unit, 3: Material data generation unit, 4: Storage unit, 5: Time information management unit, 6: Video generation unit, 7: Video output unit, 8: Data group selection unit, 10: Operation unit, 11: Display unit, 12: Video type selection unit, 13: Time indicator unit

Claims (19)

A determination means for determining the time when the virtual viewpoint content can be generated based on the state of the material data used to generate the virtual viewpoint content, and
Based on the time determined by the determination means, data for displaying information that can distinguish between the time range in which the virtual viewpoint content can be generated and the time range in which the virtual viewpoint content cannot be generated is output. Output means to
An information processing device characterized by having. The information processing apparatus according to claim 1, wherein the state of the material data is whether or not the material data is stored in the storage means. The information processing according to claim 2, wherein the determination means determines a time at which the virtual viewpoint content can be generated based on a time associated with the material data stored in the storage means. Device. The determination means specifies the time when the material data necessary for generating the virtual viewpoint content is stored in the storage means based on the time associated with the material data, thereby specifying the virtual viewpoint content. The information processing apparatus according to claim 3, further comprising determining a time at which the data can be generated. The determination means according to any one of claims 2 to 4, wherein the determination means determines a time at which the virtual viewpoint content can be generated based on the type of the material data stored in the storage means. The information processing device described. The determination means determines the time when the virtual viewpoint content can be generated by specifying that a predetermined type of material data necessary for generating the virtual viewpoint content is stored in the storage means. The information processing apparatus according to claim 5. The material data according to any one of claims 2 to 6, wherein the material data is stored in the storage means by being generated based on a plurality of captured images obtained from the plurality of imaging devices. Information processing device. The determination means generates a first time at which the first virtual viewpoint content can be generated and a second virtual viewpoint content different from the first virtual viewpoint content as a time at which the virtual viewpoint content can be generated. Judging that it is a possible second time,
The output means has a time range in which the first virtual viewpoint content can be generated based on the first time and the second time determined by the determination means, and the second virtual viewpoint content. Outputs data for displaying information that identifiablely represents a time range that cannot be generated.
The information processing apparatus according to any one of claims 1 to 7. The information processing apparatus according to claim 8, wherein the quality of the first virtual viewpoint content is different from the quality of the second virtual viewpoint content. The information processing apparatus according to claim 8 or 9, wherein the use of the first virtual viewpoint content is different from the use of the second virtual viewpoint content. The determination means determines, as the time when the virtual viewpoint content can be generated, a time when the virtual viewpoint content having a quality higher than a predetermined quality can be generated.
The output means has a time range in which virtual viewpoint content having a quality higher than the predetermined quality can be generated based on the time determined by the determination means, and a virtual viewpoint having a quality higher than the predetermined quality. Outputs data for displaying information that distinguishes between time ranges in which content cannot be generated.
The information processing apparatus according to any one of claims 1 to 10. Any one of claims 1 to 11, wherein the material data includes at least one of a three-dimensional shape model representing the three-dimensional shape of the object and a texture representing the color of the three-dimensional shape model. The information processing device described in. The material data includes at least one of a high-resolution three-dimensional shape model representing the three-dimensional shape of an object with high resolution and the low-resolution three-dimensional shape model representing the three-dimensional shape of an object with low resolution. The information processing apparatus according to any one of claims 1 to 12, further comprising. Based on the data output by the output means, the display means is made to display a seek bar that identifiable the time range in which the virtual viewpoint content can be generated and the time range in which the virtual viewpoint content cannot be generated. The information processing apparatus according to any one of claims 1 to 13, further comprising a display control means for controlling. It has a selection means for selecting at least one type from a plurality of types of virtual viewpoint contents based on at least one of user operations and user attributes.
The determination means determines the time at which the virtual viewpoint content corresponding to the type selected by the selection means can be generated.
The information processing apparatus according to any one of claims 1 to 14. The data output by the output means can generate virtual viewpoint content corresponding to any of a plurality of types of virtual viewpoint contents at at least one time point in the time range in which the virtual viewpoint contents can be generated. The information processing apparatus according to any one of claims 1 to 15, wherein the data represents the above. Claims 1 to 16 are characterized in that the output means outputs data for displaying information representing a time range in which virtual viewpoint contents corresponding to each type can be generated for a plurality of types of virtual viewpoint contents. The information processing apparatus according to any one of the above items. It is a control method performed by an information processing device.
A determination process for determining the time when the virtual viewpoint content can be generated based on the state of the material data used to generate the virtual viewpoint content, and a determination step.
Based on the time determined in the determination step, data for displaying information that can distinguish between the time range in which the virtual viewpoint content can be generated and the time range in which the virtual viewpoint content cannot be generated is output. Output process and
A control method characterized by having. A program for causing a computer to function as the information processing device according to any one of claims 1 to 17.

Images