JP7465227B2 - 3D model generation device, method and program - Google Patents

Description

The present invention relates to an apparatus, method, and program for quickly generating a 3D model of a subject from images captured by multiple cameras.

The volume intersection method disclosed in Non-Patent Document 1 is a widely known approach to generating a 3D model of a subject from images captured by multiple cameras. As shown in Figure 14, the volume intersection method is a technique for generating a 3D model by projecting binary silhouette images, which are obtained by extracting the subject's area from each camera image, into a 3D space and retaining the intersection of these images.

Visual volume intersection is used as one of the elemental technologies for restoring the shape of a 3D model in free viewpoint imaging technology as described in Non-Patent Document 2. Free viewpoint imaging technology reconstructs 3D space from images captured by multiple cameras, making it possible to view images from angles where there are no cameras.

The smallest unit that makes up a 3D model generated by the volume intersection method is called a voxel. A voxel is a cube with a small volume, and when generating a 3D model using the volume intersection method, a voxel grid is defined that fills the entire 3D area where the 3D model is to be created with these cubes, and a determination is made for each voxel grid as to whether or not a model will be generated.

When the length of one side of this cube (unit voxel size) is M, the larger the unit voxel size M is set, the more discrete the 3D space is treated, so the processing time for the visual volume intersection method is shortened, but the model is discretized, resulting in a 3D model with a coarse shape.

On the other hand, the smaller the unit voxel size M, the more detailed the shape can be restored, but the increase in calculation units results in an explosive increase in processing time. In particular, when considering applications to free viewpoint video, voxels must be generated for large spaces such as sports stadiums, which means that calculation times tend to increase.

To solve these technical problems, Non-Patent Document 3 and Patent Document 1 disclose techniques for accelerating the processing of the volume intersection method. In Non-Patent Document 3, when generating a 3D voxel model using the volume intersection method, first, as shown in Figure 15, a model is generated with a coarse unit voxel size Mb, and a bounding box in 3D space is obtained with the connected region of voxels as a single subject.

Then, we successfully significantly reduced processing time by modeling the inside of each 3D bounding box with a fine unit voxel size Ma (< Mb) using a volume intersection method.

Patent Document 1 also discloses a similar technology that speeds up 3D model generation by estimating a coarse 3D voxel model for the 3D space in which the 3D model is to be created, placing a finer voxel grid at the generation position of the coarse 3D voxel model, and repeating the process of generating a 3D model using the volume intersection method again for this voxel grid.

Patent Document 2 discloses a two-stage model generation method that improves model quality by estimating object positions using the volume intersection method in the first stage to obtain object candidates, and then determining whether the candidates are subjects or virtual objects in the second stage and deleting the virtual objects, in order to prevent the generation of virtual objects in the volume intersection method. It also discloses a mechanism that refers to the subject generation position in a previous frame at this time, and makes it less likely to be determined as a virtual object if it is close to the reference position.

Non-Patent Document 4 discloses an invention that speeds up the generation of 3D models by balancing the calculation load when calculating them on a PC cluster.

JP 2018-063635 A Patent No. 5454573 Patent Application No. 2019-153696

A. Laurentini, "The visual hull concept for silhouette based image understanding.", IEEE Transactions on Pattern Analysis and Machine Intelligence, 16, 150-162, (1994). J. Kilner, J. Starck, A. Hilton and O. Grau, "Dual-Mode Deformable Models for Free-Viewpoint Video of Sports Events," Sixth International Conference on 3-D Digital Imaging and Modeling (3DIM 2007), pp. 177-184, (2007). J. Chen, R. Watanabe, K. Nonaka, T. Konno, H. Sankoh, S. Naito, "A Fast Free-viewpoint Video Synthesis Algorithm for Sports Scenes", 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019), WeAT17.2, (2019). Tomomi Iwashita, Ryo Kurazume, Kenji Hara, Seiichi Uchida, Kenichi Morooka, Tsutomu Hasegawa, "Fast 3D Shape Recovery of Real Objects Using Parallel Fast Level Set Method", Abstracts of the 2006 Symposium on Robotics and Mechatronics, 2P1-C13, (2006). C. Stauffer and W. E. L. Grimson, "Adaptive background mixture models for real-time tracking," 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 246-252, Vol. 2, (1999). J. Ruttle, M. Manzke and R. Dahyot, "Estimating 3D Scene Flow from Multiple 2D Optical Flows," 2009 13th International Machine Vision and Image Processing Conference, Dublin, 2009, pp. 1-6, doi: 10.1109/IMVIP.2009.8. Arun, Somani; Thomas S. Huang; Steven D. Blostein, "Least-square fitting of two 3-D point sets". IEEE Pattern Analysis and Machine Intelligence, (1987).

Although the high-speed techniques described in Non-Patent Document 3 and Patent Document 1 are effective, the experiment described in Non-Patent Document 3 only demonstrated the real-time generation of 3D models up to the size of a volleyball court, and did not verify whether real-time calculations while maintaining quality were possible in a wide space such as a soccer stadium.

Considering the applications of free viewpoint video technology, possible use cases include turning a sports stadium into a free viewpoint, generating and viewing replay videos with any camerawork, or allowing users to interactively enjoy free viewpoint video from the viewpoint they want to view according to their own operations. In other words, real-time generation of 3D models of large spaces such as an entire soccer or baseball stadium will be essential.

It may be possible to ensure real-time performance by increasing the unit voxel sizes Ma and Mb, but increasing Ma and Mb will result in a deterioration in quality.

In addition, when using a server with high computational processing power, the cost of the server increases, which is an obstacle to practical use. In addition, in a use case such as creating a 3D model of an entire stadium, the generation time not only for fine voxels generated with a unit voxel size Ma, but also for coarse voxels generated with a unit voxel size Mb tends to increase due to the large area.

In addition, although the high-speed method using a PC cluster as described in Non-Patent Document 4 is effective in increasing speed, there is an issue of increased costs due to the need to prepare a large number of PCs.

The objective of the present invention is to solve the above technical problems and to speed up 3D model generation by determining the generation position of the 3D model in the input frame by estimating the movement direction and amount of movement of the target object between successive frames based on the generation position of the 3D model in the previous frame, and narrowing down the area where the 3D model calculation should be performed.

To achieve the above object, the present invention is a 3D model generation device that generates a 3D model of a subject based on silhouette images extracted frame by frame from video captured by multiple cameras with different viewpoints, and is characterized by having the following configuration.

(1) The system includes a means for storing a model generation history including the generation position of each 3D model in a past frame, a means for determining a model calculation area in an input frame based on the model generation history, and a means for performing a 3D model calculation on the model calculation area to generate a 3D model.

(2) A means for classifying input frames as key frames or non-key frames is included, and the means for generating the 3D model includes a means for generating a relatively high-resolution 3D model and a means for generating a low-resolution 3D model, and in the case of key frames, a high-resolution 3D model is generated in an area where a low-resolution 3D model is generated based on a silhouette image, and in the case of non-key frames, a 3D model calculation is performed for a model calculation area to generate a high-resolution 3D model.

(3) In non-keyframes, 3D model calculations are performed on the model calculation area to generate a low-resolution 3D model, and then 3D model calculations are performed on the generation area to generate a high-resolution 3D model.

(4) The method for determining the model calculation area is to determine the area that is an extension of the 3D model generation position in the previous frame as the model calculation area.

(5) A means for classifying each 3D model into a class is provided, and the means for determining the model calculation area is adapted to expand the 3D model generation position in the previous frame by an expansion amount according to the class.

(6) The classification method classifies each 3D model into classes based on the assumed moving speed of the subject, and the method for determining the model calculation area is set so that the amount of expansion is larger for classes of subjects with faster moving speeds.

(7) A means for estimating a velocity field of each 3D model based on the generation history of each 3D model in multiple past frames is provided, and the means for determining the model calculation area determines the model calculation area for each 3D model based on the velocity field.

(8) The number of silhouette images used when generating a low-resolution 3D model in the non-keyframe is made smaller than the number of silhouette images used when generating a low-resolution 3D model in the keyframe.

The present invention achieves the following effects:

(1) The model calculation area in the input frame is determined based on the generation position of each 3D model in the previous frame, and 3D model calculation is performed only for that model calculation area, which enables high-speed generation of 3D models.

(2) In keyframes, a high-resolution 3D model is generated only in the area where a low-resolution 3D model is generated based on the silhouette image, whereas in non-keyframes, a high-resolution 3D model is generated by performing 3D model calculations on the model calculation area. This makes it possible to speed up 3D model generation, especially in non-keyframes, while maintaining its quality.

(3) In non-keyframes, 3D model calculations are performed on the model calculation area, and then a high-resolution 3D model is generated by performing 3D model calculations on the generation area of the low-resolution 3D model that is generated by performing 3D model calculations on the model calculation area. This allows the model calculation area in non-keyframes to be determined more accurately.

(4) The 3D model generation position in the previous frame is expanded to determine the model calculation area, so the model calculation area can be determined with calculations that have a light processing load, enabling the 3D model to be generated quickly.

(5) Each 3D model is classified into a class, and the 3D model generation position in the previous frame is expanded by an amount corresponding to the class, so that the model calculation area can be determined to be an appropriate range for each 3D model, without being excessive or insufficient.

(6) Each 3D model is classified into a class based on the assumed moving speed of the subject, and the amount of expansion is increased for classes of subjects with faster moving speeds. This makes it possible to determine an appropriate range for the model calculation area, neither too much nor too little, even if there are differences in the moving speeds of the 3D models.

(7) The velocity field of each 3D model is estimated based on the generation history of each 3D model in multiple past frames, and the model calculation region is determined for each 3D model based on that velocity field. This makes it possible to determine the model calculation region to be an appropriate range that is neither excessive nor insufficient, even if there is a difference in the movement speed or direction of the 3D models.

(8) The number of silhouette images used when generating a low-resolution 3D model in a non-keyframe is reduced compared to the number of silhouette images used when generating a low-resolution 3D model in a keyframe. This makes it possible to more accurately determine the model calculation area while minimizing the increase in processing load in non-keyframes.

1 is a functional block diagram showing a configuration of a 3D model generating device according to a first embodiment of the present invention. 13 is a diagram showing an example of determining a 3D model calculation area by expanding a generation area of a 3D model in a past frame. FIG. FIG. 11 is a functional block diagram showing the configuration of a 3D model generating device according to a second embodiment of the present invention. FIG. 13 is a diagram showing an example of dividing each 3D model into 3D bounding boxes. 13A and 13B are diagrams illustrating an example in which the amount of expansion is changed depending on the class of a 3D model. FIG. 11 is a functional block diagram showing the configuration of a 3D model generating device according to a third embodiment of the present invention. FIG. 13 is a diagram showing an example of determining a model calculation domain based on a velocity field. FIG. 13 is a functional block diagram showing the configuration of a 3D model generating device according to a fourth embodiment of the present invention. FIG. 1 is a diagram showing an example of how a 3D model is generated in a different way depending on the frame. FIG. 2 is a diagram (part 2) showing an example of changing the method of generating a 3D model depending on the frame. 13A and 13B are diagrams illustrating an example in which the model calculation area of each 3D model in a non-key frame is determined based on an expansion amount according to the class. 13A and 13B are diagrams showing examples of determining the model calculation region of each 3D model in a non-key frame based on a velocity field. FIG. 13 is a functional block diagram showing the configuration of a 3D model generating device according to a fifth embodiment of the present invention. FIG. 13 is a diagram showing a method for generating a 3D model using the volume intersection method. FIG. 13 shows an example of generating a 3D model in two stages.

The following describes in detail an embodiment of the present invention with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a 3D model generation device 100 according to a first embodiment of the present invention, and the main components are a silhouette image acquisition unit 10, a 3D model generation unit 20, a history database (DB) 30, a calculation area determination unit 40, and a history registration unit 50.

Such a 3D model generating device 100 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software. In this embodiment, an example will be described in which a sports scene is photographed by N cameras Cam1 to CamN, and a 3D model is generated for each subject.

The silhouette image acquisition unit 10 acquires silhouette images to be used in the volume intersection method on a frame-by-frame basis from multiple camera images (multi-viewpoint images) in which multiple subjects are captured from different viewpoints. To form a 3D model using the volume intersection method, it is desirable to acquire silhouette images from two or more cameras. The silhouette image is acquired in the form of a binary mask image in which the subject area from which the 3D model is to be generated is shown in white and the other areas in black. Such silhouette images can be calculated using conventional techniques such as the background subtraction method disclosed in Non-Patent Document 5.

The history DB30 stores model generation history including information on the position where a 3D model was generated in a past frame prior to the current input frame. The calculation area determination unit 40 determines the model calculation area in the input frame based on the model generation history of the past frames.

In this embodiment, the calculation region determination unit 40 includes an expansion unit 41, and as shown in an example in FIG. 2, for example, a region obtained by expanding the model generation region r in the previous frame by a predetermined distance or number of voxels (hereinafter collectively referred to as the expansion amount P) around it is determined as the model calculation region R in the input frame. Note that while FIG. 2 is depicted in two dimensions for convenience, in reality the expansion based on the expansion amount P is performed in three-dimensional space.

In the 3D model generation unit 20, the voxel model calculation unit 21 places a voxel grid with a unit voxel size Ma suitable for generating a 3D model that satisfies the required quality in the three-dimensional model calculation area determined by the calculation area determination unit 40, and generates a high-resolution 3D voxel model by a volume intersection method using the silhouette image acquired by the silhouette image acquisition unit 10.

Note that while a voxel model is volume data formed from a large number of voxels, it is generally more convenient to handle 3D model data as a polygon model in many cases. Therefore, in this embodiment, a 3D model output unit 22 is provided in the 3D model generation unit 20, and the voxel model is converted into a polygon model using a method for converting a voxel model into a polygon model, such as the marching cubes method, and the 3D model is output as a polygon model.

The history registration unit 50 registers the 3D model generation history obtained for each frame in the history DB 50 as history information to be referenced when determining the model calculation area in the next frame and thereafter.

According to this embodiment, the model calculation area in the input frame is determined based on the generation position of each 3D model in the previous frame, and the 3D model calculation is performed only for that model calculation area, so that the 3D model can be generated quickly.

In addition, according to this embodiment, the 3D model generation position in the previous frame is expanded and determined as the model calculation area, so the model calculation area can be determined with calculations that impose a light processing load, enabling the 3D model to be generated even faster.

Figure 3 is a functional block diagram showing the configuration of the second embodiment of the present invention. The same reference numerals as those described above represent the same or equivalent parts, and therefore their explanation will be omitted.

In this embodiment, the history registration unit 50 includes a classification unit 51 that classifies the 3D models generated based on the silhouette images, for example, based on their general moving speed, and the expansion unit 41 adaptively determines the expansion amount P when expanding the model generation area r to determine the model calculation area R, based on the results of the classification for each 3D model.

The classification unit 51 classifies each 3D model generated for each frame into a class such as "person" or "ball," and registers the classification result together with the generation position of each 3D model in the history DB 30 as a model generation history. In order to perform the classification process, the 3D models must be input separately for each subject, and this is done by obtaining the connected areas as a single block when the input 3D models are connected. Alternatively, as shown in Figure 4, classification may be performed based on the 3D bounding box that contains each 3D model.

Classification can be done using (1) classification based on the size of the 3D model, (2) classification based on the position of the 3D model, or (3) classification based on deep learning, etc.

(1) Classification based on the size of the 3D model Each subject can be classified based on the size and shape (overall size, length, width, and height) of its 3D voxel model. For example, when classifying subjects into "people" and "balls," which are often seen in sports scenes, if the voxel model is larger than a certain threshold, it can be classified as a "people" and if it is smaller, it can be classified as a "ball." Alternatively, if the shape of the 3D bounding box is a cube, it can be classified as a "ball," and if it is a rectangular prism, it can be classified as a "people."

(2) Classification based on the position of the 3D model. For example, a 3D model formed at a height of more than 10 m can be classified as a ball, and the probability that it is a person or equipment can be estimated to be low.

(3) Classification based on deep learning etc.
Taking advantage of the fact that the shape of a 3D model is characteristic of each subject, a predictive model can be constructed by learning in advance the relationship between the model shape and the subject using deep learning or other methods, and class classification can be performed by applying each 3D model to the predictive model.

Each of the above classification methods may be used alone, or multiple classification methods may be combined as appropriate.

Figure 5 is a schematic diagram showing an example in which the expansion unit 41 adaptively determines the expansion amount P when determining the model calculation area R according to the class of the 3D model. In this embodiment, when a 3D model is generated, the classification unit 51 classifies each 3D model into a "person" or a "ball." Since the movement speed of a "ball" is generally faster than the movement speed of a "person," the expansion unit 41 sets the expansion amount when determining the model calculation area of a ball to be larger than the expansion amount when determining the model calculation area of a person.

According to this embodiment, each 3D model is classified into a class, and the 3D model generation position in the previous frame is expanded by an expansion amount according to the class, so that the model calculation area can be determined to be an appropriate range for each 3D model, neither too much nor too little.

In addition, according to this embodiment, each 3D model is classified into classes based on the expected moving speed of the subject, and the amount of expansion is increased for classes of subjects with faster moving speeds, so that even if there are differences in the moving speeds of the 3D models, the model calculation area can be determined to be an appropriate range that is neither excessive nor insufficient.

Figure 6 is a functional block diagram showing the configuration of the third embodiment of the present invention. The same reference numerals as those described above represent the same or equivalent parts, and therefore their explanation will be omitted.

In this embodiment, the calculation area determination unit 40 is provided with a velocity field estimation unit 42, and as shown in Figure 7, the movement field is estimated for each 3D model based on the change in the 3D model generation position between past frames, and the model calculation area in the input frame is determined based on the estimation result.

The velocity field estimation unit 42 estimates the 3D velocity field ( _vx , _vy , _vz) of each voxel grid in the 3D space to be the free viewpoint production target based on the change in the generation position of each 3D model between past frames. For the estimation of the 3D velocity field, a technique for reconstructing a 3D optical flow from the 2D optical flow of each camera can be used, as disclosed in, for example, Non-Patent Document 6.

In addition, 3D point cloud data may be formed by regarding the voxel formation positions between past frames as points, and a point cloud registration method such as the ICP (Iterative Closest Point) method disclosed in Non-Patent Document 7 may be used for each 3D model of the subject, and the subject may be moved to the registered position, and the velocity field of the voxel may be calculated as (position after movement) - (position before movement).

Alternatively, the 3D bounding boxes of each 3D model can be tracked between adjacent frames, for example by matching the 3D bounding boxes that are closest to each other, and the motion vector from the center of gravity of each 3D bounding box in the previous frame to the center of gravity of the corresponding 3D bounding box in the subsequent frame can be estimated as the velocity field of all voxels within the 3D bounding box in the subsequent frame.

The expansion unit 41 increases the expansion amount P when determining the model calculation area for a 3D model with a larger velocity field.

In this case, similar to the second embodiment, a classification unit 51 may be provided in the history registration unit 50 to classify each 3D model or its 3D bounding box into a class such as "person" or "ball," and a different velocity field may be calculated for each class by associating 3D models or their 3D bounding boxes that are in the same class and in the closest position between successive frames.

According to this embodiment, the velocity field of each 3D model is estimated based on the generation history of each 3D model in multiple past frames, and the model calculation area is determined for each 3D model based on that velocity field. This makes it possible to determine the model calculation area to be an appropriate range that is neither excessive nor insufficient, even if there is a difference in the movement speed or movement direction of the 3D models.

Figure 8 is a functional block diagram showing the configuration of the fourth embodiment of the present invention. The same reference numerals as those described above represent the same or equivalent parts, and therefore their explanation will be omitted.

In this embodiment, the 3D model generation unit 20 is characterized in that it includes, as the voxel model calculation unit 21, a high-resolution model generation unit 21a that generates a relatively high-resolution 3D voxel model and a low-resolution model generation unit 21b that generates a low-resolution 3D voxel model, and generates a 3D voxel model by selectively using or combining each of the model generation units 21a and 21b depending on the period and type of the input frames.

The high-resolution model generation unit 21a, like the voxel model calculation unit 21 in the first embodiment, generates a high-resolution 3D voxel model by a volume intersection method using the silhouette image acquired by the silhouette image acquisition unit 10 in a three-dimensional space in which a voxel grid with a unit voxel size of Ma is arranged.

The low-resolution model generation unit 21b generates a low-resolution 3D voxel model by a volume intersection method using the silhouette image acquired by the silhouette image acquisition unit 10 in a three-dimensional space in which a voxel grid with a unit voxel size of Mb (>Ma) is arranged.

The input frame identification unit 23 identifies whether the current input frame is a key frame or a non-key frame.

In this embodiment, as shown in FIG. 9, in the key frame, the low-resolution model generation unit 21b places a voxel grid with a unit voxel size of Mb in the 3D space of the free viewpoint production target, and generates a low-resolution 3D voxel model by the visual volume intersection method using the silhouette image.

Next, the high-resolution model generation unit 21a places a voxel grid with a unit voxel size Ma only within the generation area of the low-resolution 3D voxel model or its 3D bounding box, and again generates a high-resolution 3D voxel model by the volume intersection method using the silhouette image.

In contrast, for non-key frames, the high-resolution model generation unit 21a places a voxel grid with a unit voxel size Ma only in the model calculation area determined by the calculation area determination unit 40 based on the generation history of the 3D model in past frames, and generates a high-resolution 3D voxel model by the volume intersection method using the silhouette image.

The input frame identification unit 23 defines each frame image, for example in the order of input, as a key frame, at a ratio of one frame per multiple frames, and defines the rest as non-key frames. Then, as shown in FIG. 10, after generating a low-resolution 3D voxel model for a key frame, a model is generated at high resolution only for the generation area of that model, while for non-key frames, a model is generated at high resolution only for the model calculation area determined based on the model generation history, as in the first embodiment.

According to this embodiment, in keyframes, a high-resolution 3D model is generated only in the area where a low-resolution 3D model is generated based on a silhouette image, while in non-keyframes, a high-resolution 3D model is generated by performing 3D model calculations on the model calculation area. This not only speeds up the two-stage 3D model generation in keyframes, but also speeds up the generation of 3D models in non-keyframes while maintaining their quality.

When the history registration unit 50 is provided with a classification unit 51 superimposed thereon as in the second embodiment, as shown in FIG. 11, the high-resolution model generation unit 21a generates a 3D model in a model calculation area that is adaptively expanded in accordance with the results of class classification in non-key frames.

In addition, when the calculation domain determination unit 40 is equipped with a velocity field estimation unit 42 superimposed thereon as in the third embodiment, the estimated results of the velocity field may be used for model calculation domain prediction in non-key frames, as shown in FIG. 12.

Furthermore, as in the fifth embodiment shown in FIG. 13, a low-resolution 3D model may be generated for the model calculation area predicted by the calculation area determination unit 40 in a non-key frame, and the high-resolution model generation unit 21a may generate a high-resolution 3D model for only the generation area of the low-resolution 3D voxel model or its 3D bounding box.

At this time, if the low-resolution model generation unit 21b generates a low-resolution 3D model with a unit voxel size of Mb, as in the case of generating a low-resolution 3D model in a key frame, the calculation load may increase. Therefore, a voxel grid with a unit voxel size of Mc (>Mb) may be placed in the predicted model calculation area, and a 3D model with even lower resolution may be generated.

Alternatively, the unit voxel size can be kept at Mb, but the number of silhouette images (number of cameras) used to generate the model can be reduced compared to when generating a low-resolution 3D model in non-keyframes.

According to this embodiment, in non-keyframes, 3D model calculations are performed on the model calculation area to generate a low-resolution 3D model, and then 3D model calculations are performed on the generation area to generate a high-resolution 3D model, so the model calculation area in non-keyframes can be determined more accurately.

In this embodiment, when generating a low-resolution 3D model in a model calculation area in a non-key frame, it is possible to determine for each voxel how many cameras used silhouette images in which the silhouette images were in the foreground, and calculate the model generation likelihood. For example, for a voxel that was in the foreground in the silhouette images of seven cameras out of a total of eight cameras, the model generation likelihood L _model is recorded as 7/8 (=0.875).

Furthermore, for each voxel of the low-resolution 3D model generated in the model calculation area in a non-key frame, the distance to the 3D model generation area in a past frame (e.g., the previous frame) is measured as, for example, the distance from the center of gravity position or the nearest voxel of the 3D model generation area, and a history-based generation likelihood L _history is calculated based on the inverse of the distance. At this time, the maximum likelihood (=1) can be given to voxels that overlap with the 3D model generation area.

Then, for each voxel of the low-resolution 3D model, only the region of the voxel where the model generation likelihood L _model and/or the history-based generation likelihood L _history exceeds a preset threshold T1, or only the region of the voxel where the likelihood obtained by adding the model generation likelihood L _model and the history-based generation likelihood L _history exceeds a preset threshold T2, as shown in the following equation (1), may be determined as the high-resolution model calculation region.

L _model + L _history ＞T2 (1)

Furthermore, according to each of the above embodiments, it becomes possible to provide high-quality 3D models of subjects in real time even via communication infrastructure, making it possible to provide a variety of entertainment to many people regardless of geographic or economic disparities. As a result, it becomes possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

10...Silhouette image acquisition unit, 20...3D model generation unit, 21...Voxel model calculation unit, 21a...High resolution model generation unit, 21b...Low resolution model generation unit, 22...3D model output unit, 23...Input frame identification unit, 30...History database, 40...Calculation area determination unit, 41...Expansion unit, 42...Velocity field estimation unit, 50...History registration unit, 51...Classification unit, 100...3D model generation device

Claims (11)

A 3D model generation device that generates a 3D model of a subject based on silhouette images extracted frame by frame from a video sequence captured by a plurality of cameras with different viewpoints,
A means for storing a model generation history including a generation position of each 3D model in a past frame;
means for determining a model calculation region in an input frame based on the model generation history;
A means for performing 3D model calculation on the model calculation area to generate a 3D model;
means for classifying the input frames as key frames or non-key frames;
The means for generating the 3D model includes:
means for generating a relatively high resolution 3D model and means for generating a relatively low resolution 3D model;
In the keyframe, a high-resolution 3D model is generated in the area where a low-resolution 3D model was generated based on the silhouette image,
In the non-key frame, a 3D model generation device performs 3D model calculation on the model calculation area to generate a high-resolution 3D model. The 3D model generating device according to claim 1, characterized in that, in the non-key frames, a 3D model calculation is performed on a generation area of a low-resolution 3D model generated by performing 3D model calculation on the model calculation area to generate a high -resolution 3D model. 3. The 3D model generating device according to claim 1, wherein the means for determining the model calculation area determines an area obtained by expanding a 3D model generation position in a previous frame as the model calculation area. means for classifying each 3D model into a class;
4. The 3D model generating device according to claim 3 , wherein the means for determining the model calculation area expands the 3D model generation position in the previous frame by an expansion amount according to the class. The classifying means classifies each 3D model into a class based on an expected moving speed of the subject,
5. The 3D model generating device according to claim 4 , wherein the means for determining the model calculation region increases the amount of expansion for a subject class having a faster moving speed. A means for estimating a velocity field of each 3D model based on a generation history of each 3D model in a plurality of past frames is provided;
6. The 3D model generating device according to claim 1, wherein the means for determining the model calculation region determines the model calculation region for each 3D model on the basis of its velocity field. 3. The 3D model generation device according to claim 2, wherein in the non-key frames, the number of silhouette images used when generating a low-resolution 3D model for the model calculation area is smaller than the number of silhouette images used when generating a low-resolution 3D model in the key frames. 3. The 3D model generating device according to claim 2, wherein in the non-key frames, a unit voxel size when generating a low-resolution 3D model for the model calculation region is larger than a unit voxel size when generating a low-resolution 3D model in the key frames . In the non-key frame,
The 3D model generation device according to claim 2, characterized in that for each voxel of a low-resolution 3D model generated for the model calculation area, at least one of a calculation of a model generation likelihood based on how many cameras it was in the foreground in, and a calculation of a history-based generation likelihood based on the inverse of the distance from the model generation position in a past frame is performed, and a calculation area for a high-resolution 3D model is determined based on at least one of the model generation likelihood and the history-based generation likelihood. A 3D model generation method in which a computer generates a 3D model of a subject based on silhouette images extracted frame by frame from a video sequence captured by a plurality of cameras with different viewpoints, comprising:
storing a model generation history including the generation position of each 3D model in past frames;
determining a model calculation domain in an input frame based on the model generation history;
A 3D model is generated by performing a 3D model calculation on the model calculation area;
classifying the input frames as key frames or non-key frames;
In the key frame, a high-resolution 3D model is generated in an area where a low-resolution 3D model is generated based on a silhouette image;
a 3D model generating method for generating a high-resolution 3D model by performing 3D model calculation on the model calculation area in the non-key frame; A 3D model generation program that generates a 3D model of a subject based on silhouette images extracted frame by frame from a video sequence captured by multiple cameras with different viewpoints,
storing a model generation history including a generation position of each 3D model in a past frame;
determining a model calculation region in an input frame based on the model generation history;
performing a 3D model calculation on the model calculation area to generate a 3D model;
classifying said input frames as key frames or non-key frames ;
Run the following on your computer:
The step of generating the 3D model comprises:
In the key frame, a high-resolution 3D model is generated in an area where a low-resolution 3D model is generated based on a silhouette image;
A 3D model generation program, characterized in that in the non-keyframe, a 3D model calculation is performed on the model calculation area to generate a high-resolution 3D model.

Images