JP7574569B2 - Spectral texture image generating system, spectral texture image generating method and program - Google Patents

Description

The present invention relates to a spectral texture image generation system, a spectral texture image generation method, and a program for generating texture images for performing texture mapping on a three-dimensional shape model.

3D shape reconstruction technology, which generates a 3D shape model of a target object based on multi-view images, is attracting attention not only from the computer vision research community but also from a wide range of fields, such as digital archiving of cultural assets and the entertainment industry.

The three-dimensional shape of a target object is reconstructed based on differences in how the silhouette and surface texture of the target object appear in each of the captured images in the multi-viewpoint images (see, for example, Patent Documents 1 and 2).
Generally, in restoring the three-dimensional shape of a target object from such multi-viewpoint images, a texture image for the restored three-dimensional shape model is generated from the RGB values (pixel values of each channel of Red, Green, and Blue) of corresponding pixels in the viewpoint images in the multi-viewpoint image, or the average value of the RGB values in each of multiple corresponding viewpoint images in the multi-viewpoint image.

However, the RGB values of typical imaging devices (e.g., digital cameras) available on the market are affected by the spectral distribution of the light source in the imaging environment when capturing an image of the target object and the spectral sensitivity characteristics of the imaging device used for capturing the image.
Therefore, when generating a texture image of a three-dimensional shape model using viewpoint images of the same target object captured under different light sources or with different imaging devices, even though the target object is the same, the texture image of the three-dimensional shape model will be generated in different colors depending on the light source or digital camera, because the spectral reflectance of the target object cannot be obtained.

On the other hand, there is a method that uses a hyperspectral camera to obtain color information of a target object for three-dimensional shape reconstruction without being affected by the spectral distribution of the light source in the imaging environment between viewpoint images in a multi-viewpoint image or differences in spectral sensitivity during imaging.
A hyperspectral camera can capture a hyperspectral image, which is an image in which wavelengths are split into tens to hundreds of bands and spectral information indicating the intensity of each band is stored for each pixel.
By using the above-mentioned hyperspectral camera to acquire the spectral reflectance (reflectance of light for each wavelength) of a target object, it is possible to represent colors that are not dependent on the spectral distribution of the light source in the imaging environment between the viewpoint images of a multi-view image or on differences in the spectral sensitivity of the digital cameras used for imaging.

JP 2018-181047 A Patent No. 5068732

However, a problem with hyperspectral cameras is that they are expensive compared to general digital cameras that capture RGB images.
In addition, a hyperspectral camera requires several tens of seconds (or in some cases, more than one minute) to capture one hyperspectral image, which is much longer than a digital camera that can capture one image in less than one second.
For this reason, a hyperspectral camera takes more time to capture a target object than a digital camera because many images must be captured to generate a texture image. For example, a digital camera can capture 100 or more multi-viewpoint images of a target object in just a few minutes, but a hyperspectral camera requires several hours of work.

Furthermore, the hyperspectral camera has a problem that the resolution of the captured image is lower than that of a general digital camera that captures RGB images. That is, when generating a texture image of a three-dimensional shape model from a multi-view image captured by a hyperspectral camera, the resolution of the viewpoint images in the multi-view image significantly affects the resolution of the texture image.
For this reason, a hyperspectral camera with low resolution cannot generate a texture image of a three-dimensional shape model of a target object with the same resolution as when it is generated from an image captured by a digital camera.

For the reasons mentioned above, it is difficult to use hyperspectral images captured by a hyperspectral camera as multi-viewpoint images for generating texture images of a three-dimensional shape model that reconstructs a target object. For this reason, there is a demand for technology that can easily obtain the shape and spectral reflectance of a target object using an imaging device that is less expensive than a hyperspectral camera and can capture many multi-viewpoint images in a short period of time.

The present invention has been made in consideration of the above-mentioned circumstances, and provides a spectral texture image generation system, a spectral texture image generation method, and a program that can easily generate a texture image that reflects the spectral reflectance of a target object using images captured by an imaging device that is less expensive than the expensive hyperspectral cameras available on the market to general users.

The spectral texture image generation system of the present invention is a spectral texture image generation system that generates a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object as a texture image of a three-dimensional shape model of the target object from a multi-viewpoint image, and is characterized by comprising: a multi-viewpoint image classification unit that classifies each of the multi-viewpoint images captured by imaging devices with a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity; a virtual multi-viewpoint image generation unit that uses the multi-viewpoint image at each of the viewpoints as a reference image and converts other multi-viewpoint images that are in the vicinity of the viewpoint of the reference image and belong to a different spectral sensitivity group into an image captured from the viewpoint of the reference image to generate a virtual multi-viewpoint image; a texture image generation unit that generates a texture image corresponding to the spectral sensitivity of the spectral sensitivity group from each of the reference image and virtual multi-viewpoint images in the spectral sensitivity group; and a spectral texture image generation unit that estimates a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities to generate the spectral texture image of the three-dimensional shape model.

The spectral texture image generation system of the present invention is characterized in that, when the texture image generation unit generates the texture image for any of the spectral sensitivities, it generates a lookup table in which texture coordinates indicating the positions of each vertex of a texture region in a two-dimensional coordinate system of the texture image correspond to vertex coordinates indicating the positions of the vertices of polygons of a three-dimensional shape model to which each of the texture regions is attached, and references the lookup table to generate texture images for other spectral sensitivities in which the correspondence between the vertex coordinates of the polygons and the texture coordinates is similar.

The spectral texture image generation system of the present invention further includes a texture image reconstruction unit that reconstructs the texture images, and the texture image generation unit generates texture images corresponding to each of the spectral sensitivities, and when generating each of the texture images, generates lookup tables in which texture coordinates indicating the positions of each vertex of a texture region in the texture image correspond to vertex coordinates indicating the positions of the vertices of a polygon of a three-dimensional shape model to which the texture image is applied, and the texture image reconstruction unit uses one of the texture images as a reference texture image and reconstructs the correspondence between the vertex coordinates of the polygons and the texture coordinates in the other texture images to match the correspondence between the vertex coordinates of the polygons and the texture coordinates in the lookup table of the reference texture image.

The spectral texture image generating system of the present invention is characterized in that the texture image generating unit generates texture images corresponding to each of the spectral sensitivities, and when generating each of the texture images, generates lookup tables in which texture coordinates indicating each vertex of a texture region in the texture image correspond to vertex coordinates indicating the positions of vertices of a polygon of a three-dimensional shape model to which the texture image is applied, the spectral texture image generating unit refers to each of the lookup tables, associates pixels in the texture region of each of the texture images with the vertex coordinates in each of the lookup tables, estimates a spectral reflectance spectrum from each of the associated pixels, and generates the spectral texture image.

The spectral texture image generation method of the present invention is a spectral texture image generation method for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from a multi-viewpoint image as a texture image of a three-dimensional shape model of the target object, the method including a multi-viewpoint image classification process in which a multi-viewpoint image classification unit classifies each of the multi-viewpoint images of the target object captured by an imaging device having a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity, and a virtual multi-viewpoint image generation unit classifies the multi-viewpoint images at each of the viewpoints as a reference image, and classifies a virtual multi-viewpoint image that is in the vicinity of the viewpoint of the reference image and is in a different spectral sensitivity group. The method includes a virtual multi-viewpoint image generation process for generating a virtual multi-viewpoint image by converting the other multi-viewpoint images belonging to the reference image into images captured from the viewpoint of the reference image, a texture image generation process in which a texture image generation unit generates a texture image corresponding to the spectral sensitivity of the spectral sensitivity group from each of the reference images and virtual multi-viewpoint images in the spectral sensitivity group, and a spectral texture image generation process in which a spectral texture image generation unit estimates a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities, and generates the spectral texture image of the three-dimensional shape model.

The program of the present invention is a program for causing a computer to function as a spectral texture image generation system that generates, as a texture image of a three-dimensional shape model of a target object from a multi-viewpoint image, a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of the target object, and causes the computer to function as a multi-viewpoint image classification means that classifies each of the multi-viewpoint images captured by an imaging device having a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity, a virtual multi-viewpoint image generation means that uses the multi-viewpoint image at each of the viewpoints as a reference image and converts other multi-viewpoint images that are in the vicinity of the viewpoint of the reference image and belong to different spectral sensitivity groups into images captured from the viewpoint of the reference image to generate a virtual multi-viewpoint image, a texture image generation means that generates a texture image corresponding to the spectral sensitivity of the spectral sensitivity group from each of the reference image and virtual multi-viewpoint images in the spectral sensitivity group, and a spectral texture image generation means that estimates a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities and generates the spectral texture image of the three-dimensional shape model.

As described above, the present invention provides a spectral texture image generation system, a spectral texture image generation method, and a program that can easily generate a texture image that reflects the spectral reflectance of a target object using images captured by an imaging device that is less expensive than expensive hyperspectral cameras available on the market to general users.

1 is a block diagram showing an example of the configuration of a spectral texture image generating system according to a first embodiment of the present invention. FIG. 11 is a diagram showing an example of the configuration of a multi-viewpoint image information table written and stored in a multi-viewpoint image storage unit 107. FIG. 13 is a diagram showing an example of the configuration of a spectral sensitivity group data table in the multi-viewpoint image storage unit 107. FIG. 1 is a conceptual diagram illustrating generation of a spectral texture image by a spectral texture image generation system according to a first embodiment. FIG. 11A and 11B are diagrams illustrating a process of selecting a target multi-viewpoint image that can be converted into a virtual multi-viewpoint image. 11 is a diagram showing an example of the configuration of a same-viewpoint multi-viewpoint image table in a multi-viewpoint image storage unit 107. FIG. FIG. 13 is a conceptual diagram showing how spectral reflectance is expressed by combining spectral reflectance basis vectors. 4 is a diagram showing an example of the configuration of a spectroscopic image table stored in a spectroscopic image storage unit 109 in this embodiment. FIG. 1A and 1B are conceptual diagrams showing the generation of a development view of a polygon mesh in a three-dimensional shape model. 11 is a diagram showing an example of the configuration of a correspondence coordinate lookup table indicating the correspondence between polygon mesh vertex coordinates in a three-dimensional shape model and area vertex coordinates in a development view. FIG. FIG. 11 is a diagram showing an example of the configuration of a spectral texture image table in the spectral texture image storage unit 110. As shown in FIG. FIG. 13 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #1 being rendered onto a three-dimensional shape model. FIG. 13 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #2 being rendered onto a three-dimensional shape model. FIG. 13 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #3 being rendered onto a three-dimensional shape model. 10 is a flowchart showing an example of the operation of a process for generating spectral texture images for a three-dimensional shape model from multi-viewpoint images according to the present embodiment. FIG. 11 is a block diagram showing an example of the configuration of a spectral texture image generating system according to a second embodiment of the present invention. 11 is a conceptual diagram illustrating generation of a spectral texture image by a spectral texture image generation system according to a second embodiment. FIG. FIG. 11 is a diagram illustrating an example of the configuration of a texture image table of a texture image storage unit 113 according to the second embodiment. FIG. 13 is a diagram illustrating an example of the configuration of a texture image table in a texture image storage unit 113 according to the third embodiment. FIG. 13 is a block diagram showing an example of the configuration of a spectral texture image generating system according to a fourth embodiment of the present invention. FIG. 13 is a block diagram showing an example of the configuration of a spectral texture image generating system according to a fifth embodiment of the present invention.

First Embodiment
Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing an example of the configuration of a spectral texture image generating system according to a first embodiment of the present invention. In this embodiment, captured images of a target object to be restored as a three-dimensional shape model are supplied as viewpoint image data from an external device (not shown). Each of these viewpoint images is a multi-view image captured from different viewpoints. Each of the viewpoint images includes captured image data of the same and different viewpoints captured by each of a plurality of imaging devices with different spectral sensitivities. The imaging device used in this embodiment is not a hyperspectral camera, but is, for example, a digital camera used by a general user. In this embodiment, a multi-view image is described as a collection of multiple viewpoint images (captured images captured by imaging devices having different viewpoints and different spectral sensitivities).

In FIG. 1, the spectral texture image generation system 100 includes a data input/output unit 101, a multi-view image classification unit 102, a virtual multi-view image generation unit 103, a spectral image generation unit 104, a spectral texture image generation unit 105, a visualization texture image generation unit 106, a multi-view image storage unit 107, a three-dimensional shape model storage unit 108, a spectral image storage unit 109, and a spectral texture image storage unit 110.

The data input/output unit 101 writes and stores, in the multi-viewpoint image storage unit 107, each of the pieces of viewpoint image data obtained by capturing an object from each viewpoint, supplied from an external device, as a multi-viewpoint image.
Furthermore, the data input/output unit 101 writes and stores data of the three-dimensional shape model of the target object, which is reconstructed using the multi-viewpoint images, in the three-dimensional shape model storage unit 108. The three-dimensional shape model is a surface model formed of a polygon mesh reconstructed from the three-dimensional point cloud.
In addition, the data input/output unit 101 writes and stores in the multi-viewpoint image storage unit 107 the viewpoint information indicating the viewpoint coordinates and viewpoint direction at which the multi-viewpoint image was captured, the camera parameters of the imaging device that captured the multi-viewpoint image, and data on the spectral distribution of the light source (light source spectral distribution) in the environment in which the multi-viewpoint image was captured, which are supplied from the external device that restored the three-dimensional shape model.
Furthermore, when the data input/output unit 101 is accessed by an external device and requested to transmit spectral texture images, the data input/output unit 101 reads each of the spectral texture images (described later) from the spectral texture image storage unit 110 and outputs them to the external device.

Fig. 2 is a diagram showing an example of the configuration of a multi-viewpoint image information table written and stored in the multi-viewpoint image storage unit 107. The multi-viewpoint image information table shown in Fig. 2 has columns for each record, corresponding to multi-viewpoint image identification information, for an image capture device type, viewpoint information, camera parameters, an image data index, and a spectral distribution data index.
The multi-viewpoint image identification information is identification information that uniquely identifies each of the individual multi-viewpoint images (e.g., a number indicating the order in which the images were supplied). The viewpoint position is, for example, information indicating an external parameter of the imaging device, and indicating the center coordinates of the lens of the imaging device (hereinafter, viewpoint position) in the three-dimensional spatial coordinate system (world coordinate system) of the three-dimensional shape model, and the optical axis direction of the lens (optical axis vector, hereinafter, optical axis vector).

The camera parameters indicate, for example, internal parameters of the imaging device, such as the focal length of the imaging device when the viewpoint image was captured, the image center, the image resolution (number of pixels), and the distortion aberration coefficient. The image data index is an address indicating an area in the multi-viewpoint image storage unit 107 where data of the multi-viewpoint image is written. The spectral sensitivity data index is an address indicating an area in the multi-viewpoint image storage unit 107 where data of the spectral sensitivity of the imaging device is written. The spectral distribution data index is an address indicating an area in the multi-viewpoint image storage unit 107 where data of the spectral distribution of the light source (light source spectral distribution) of the environment in which the multi-viewpoint image of the target object supplied from an external device was captured is written.

Returning to FIG. 1, the multi-viewpoint image classification unit 102 classifies each of the multi-viewpoint images stored in the multi-viewpoint image storage unit 107 into a spectral sensitivity group that is provided for each different spectral sensitivity. Then, the multi-viewpoint image classification unit 102 writes and stores each of the multi-viewpoint images classified into the respective spectral sensitivity groups in a spectral sensitivity group data table for each spectral sensitivity group in the multi-viewpoint image storage unit 107.

Figure 3 is a diagram showing an example of the configuration of a spectral sensitivity group data table in the multi-viewpoint image storage unit 107. The spectral sensitivity group data table is provided for each spectral sensitivity group, and shows the spectral sensitivity data of the spectral sensitivity group. Furthermore, the spectral sensitivity group data table has columns for multi-viewpoint image identification information, image index, viewpoint information, camera parameters, and spectral distribution data index for each record.

For example, when capturing images using three imaging devices each with a different spectral sensitivity, the viewpoint images each have three different types of spectral sensitivity and are therefore classified into three spectral sensitivity groups. Each of the three spectral sensitivity groups may have the same number of multi-view images or a different number of multi-view images. In addition, a spectral sensitivity data index is added to the spectral sensitivity group data table. The spectral sensitivity data index is an address that indicates the area where the spectral sensitivity data of the imaging devices is written.

The viewpoint image identification information is identification information for identifying each multi-view image (e.g., a number indicating the order in which they were supplied). The image index is an address indicating an area in the multi-view image storage unit 107 in which the data of the multi-view image has been written. The viewpoint position indicates, for example, external parameters of the imaging device, and is information indicating the viewpoint position and optical axis vector in the world coordinate system. The camera parameters indicate, for example, internal parameters of the imaging device, and are the focal length, image center, image resolution, and distortion aberration coefficient of the imaging device when the viewpoint image was captured. The spectral distribution data index is an address indicating an area in the multi-view image storage unit 107 in which data of the light source spectral distribution of the environment in which the multi-view image of the target object supplied from an external device was captured has been written.

4 is a conceptual diagram for explaining generation of a spectral texture image by the spectral texture image generation system of the first embodiment. As the imaging devices, for example, three imaging devices A1, A2, and A3 having different spectral sensitivities are used.
4A shows each of the multi-viewpoint images (such as multi-viewpoint images 601 to 607) captured of the target object 500 at different viewpoint positions and optical axis vectors. Here, the multi-viewpoint images 601, 604, and 607 are captured by the imaging device A3. The multi-viewpoint images 602 and 605 are captured by the imaging device A1. The multi-viewpoint images 603 and 606 are captured by the imaging device A2.

FIG. 4B shows a process for generating a virtual multi-viewpoint image from a target multi-viewpoint image.
Multi-viewpoint image 601 is used as a reference multi-viewpoint image, and multi-viewpoint images 602 and 603 captured by imaging devices A1 and A2 having different spectral sensitivities from the imaging device A3 that captured multi-viewpoint image 601 are used as target multi-viewpoint images, and virtual viewpoint images having viewpoint information similar to the viewpoint information of multi-viewpoint image 601 are generated from multi-viewpoint images 602 and 603, which are the target multi-viewpoint images.
Similarly, multi-viewpoint image 602 is used as a reference multi-viewpoint image, and multi-viewpoint images 601 and 603 captured by imaging devices A3 and A2, which have different spectral sensitivities from the imaging device A1 that captured multi-viewpoint image 601, are used as target multi-viewpoint images, and the generation of virtual viewpoint images having viewpoint information similar to the viewpoint information of multi-viewpoint image 602 is shown from multi-viewpoint images 601 and 603, which are the target multi-viewpoint images.

Multi-viewpoint image 603 is used as a reference multi-viewpoint image, and multi-viewpoint images 602 and 604 captured by imaging devices A1 and A3 having different spectral sensitivity from the imaging device A2 that captured multi-viewpoint image 603 are used as target multi-viewpoint images.The figure shows the generation of virtual viewpoint images having viewpoint information similar to the viewpoint information of multi-viewpoint image 602 from multi-viewpoint images 602 and 604, which are the target multi-viewpoint images.
In addition, each of the multi-viewpoint images 604 to 607 is also used as a reference multi-viewpoint image, and multi-viewpoint images captured by imaging devices with different spectral sensitivities are used as target multi-viewpoint images, and a process of generating (converting) a virtual multi-viewpoint image is performed from the target multi-viewpoint images.

4C shows a process of generating a spectral image (hyperspectral image) from each of the target multi-view image and the virtual multi-view image in each viewpoint information. The spectral image is generated, for example, at 10 nm intervals from wavelength 400 nm to 700 nm (described later).
4D shows a process of using the spectral images of each viewpoint information to generate spectral texture images for a three-dimensional shape model of the target object 500. The spectral texture images are generated, for example, at intervals of 10 nm from wavelength 400 nm to 700 nm (described later).

Returning to FIG. 1, the virtual multi-viewpoint image generation unit 103 sets the multi-viewpoint image at each viewpoint position as a reference multi-viewpoint image, and sets the multi-viewpoint images at other viewpoint positions near this reference multi-viewpoint image and belonging to a different spectral sensitivity group from the reference multi-viewpoint image as target multi-viewpoint images. The virtual multi-viewpoint image generation unit 103 then converts each of the target multi-viewpoint images into an image captured from the same viewpoint position and optical axis vector as the reference multi-viewpoint image, thereby generating a virtual multi-viewpoint image.

At this time, when selecting a target multi-viewpoint image to be converted into a virtual multi-viewpoint image, if all three of the following conditions, the first condition, the second condition, and the third condition, are satisfied, the virtual multi-viewpoint image generation unit 103 determines that a portion of the three-dimensional shape model in the reference multi-viewpoint image can be reproduced in the virtual multi-viewpoint image converted from the target multi-viewpoint image with a different viewpoint position and optical axis vector.

5 is a diagram for explaining a process for selecting a target multi-viewpoint image that can be converted into a virtual multi-viewpoint image. When selecting a target multi-viewpoint image in which a portion 201 in a three-dimensional shape model 200 captured in a reference multi-viewpoint image exists when converted into a virtual multi-viewpoint image, the virtual multi-viewpoint image generating unit 103 determines the visibility of the portion 201 at each viewpoint position of the target multi-viewpoint image for each spectral sensitivity group.
If all three conditions, the first condition to the third condition below, are satisfied, the virtual multi-viewpoint image generator 103 determines that a part of the three-dimensional shape model has visibility from the viewpoint position of each viewpoint image.

Here, if a portion of the three-dimensional shape model captured in the reference multi-view image is projected onto a viewpoint image (first condition), the dot product of the normal vector of that portion and the optical axis vector of the target multi-view image is negative (second condition), and the portion is a target multi-view image at a viewpoint position that is not obstructed by other portions in the optical axis vector direction (third condition), then if the viewpoint position satisfies the above three conditions, it is determined that the target multi-view image can be converted into a virtual multi-view image, and the target multi-view image at the viewpoint position that satisfies these conditions is selected as the target multi-view image to be converted into the virtual multi-view image.

Here, the second condition is that the normal vector and the optical axis vector are opposite to each other, i.e., the normal vector points outward from the part of the three-dimensional shape model and the optical axis vector points from the viewpoint position to the part of the three-dimensional shape model, which is one of the conditions for whether or not a part of the three-dimensional shape model can be seen from the viewpoint position.
On the other hand, if the viewpoint position does not satisfy any one of the first condition, the second condition, or the third condition, the virtual multi-viewpoint image generation unit 103 does not select the target multi-viewpoint image for that viewpoint position as the target multi-viewpoint image to be converted into a virtual multi-viewpoint image.

5, viewpoint position 301 does not satisfy the second and third conditions, and viewpoint position 302 satisfies the second condition but does not satisfy the third condition. Viewpoint positions 303 to 306 satisfy the first, second and third conditions, where the second condition is that the inner product of the normal vector of the part and the line of sight vector of the viewpoint image is negative. For this reason, all may be selected as having visibility that satisfies all of the first, second and third conditions, or viewpoint images of viewpoint positions within a range of a predetermined inner product under the second condition may be selected from viewpoint positions that satisfy all of the first, second and third conditions.
On the other hand, the viewpoint position 307 satisfies the second and third conditions, but does not satisfy the first condition because the part 201 is not projected onto the viewpoint image at the viewpoint position.

The virtual multi-viewpoint image generator 103 may be configured to use data of a portion of the three-dimensional shape model and camera parameters (external parameters) of the viewpoint position to determine an aptitude score corresponding to the visibility of each viewpoint for each target multi-viewpoint image determined to be convertible into a virtual multi-viewpoint image. In this configuration, the virtual multi-viewpoint image generator 103 selects target multi-viewpoint images of a predetermined number of viewpoint positions in descending order of aptitude score (selects target viewpoint images of some viewpoint positions as target viewpoint images convertible into a virtual multi-viewpoint image).
In addition, the virtual multi-viewpoint image generating unit 103 may select one target multi-viewpoint image from each spectral sensitivity group, or may select one or more target multi-viewpoint images from each spectral sensitivity group. For example, one may be selected from the multi-viewpoint images that satisfy the first condition, the second condition, and the third condition, or all of the multi-viewpoint images that satisfy the first condition, the second condition, and the third condition may be selected as the target multi-viewpoint image. When multiple target multi-viewpoint images are selected, the virtual multi-viewpoint image generating unit 103 generates a virtual multi-viewpoint image using the average value or the median value of corresponding pixels of the multiple target multi-viewpoint images.

Returning to Figure 1, the virtual multi-viewpoint image generation unit 103 generates a coordinate transformation matrix for each target multi-viewpoint image (a matrix that transforms the two-dimensional coordinate system of the target multi-viewpoint image into the two-dimensional coordinate system of the reference multi-viewpoint image, and includes matrices for zoom, rotation, translation, affine transformation, and projective transformation) that converts an image of a portion of a specified range of the three-dimensional shape model in the target multi-viewpoint image into a two-dimensional plane with the same viewpoint position and optical axis vector as the reference multi-viewpoint image.
The virtual multi-viewpoint image generation unit 103 determines the correspondence between each of the vertex coordinate values (coordinate values in the world coordinate system, hereafter referred to as polygon mesh vertex coordinate values) of the polygon mesh that constitutes the surface of a three-dimensional shape model in a specified range of the three-dimensional shape model, and the coordinates A of a pixel in the target multi-viewpoint image (the two-dimensional coordinate system of the target multi-viewpoint image).
In addition, the virtual multi-viewpoint image generation unit 103 determines the correspondence between each of the polygon mesh vertex coordinates that constitute the surface of the three-dimensional shape model in a specified range of the three-dimensional shape model and pixel coordinates B in the reference multi-viewpoint image (the two-dimensional coordinate system of the reference multi-viewpoint image).

As a result, the virtual multi-viewpoint image generation unit 103 can obtain the coordinates A, B of each corresponding pixel in each of the two-dimensional coordinate systems of the target multi-viewpoint image and the reference multi-viewpoint image using the polygon mesh vertex coordinates, and generates a coordinate transformation matrix using the coordinates A, B of each corresponding pixel in the target multi-viewpoint image and the reference multi-viewpoint image.
Then, the virtual multi-viewpoint image generation unit 103 converts the coordinates A of each pixel in the two-dimensional coordinate system of the target multi-viewpoint image into the coordinates B of each pixel in the two-dimensional coordinate system of the reference multi-viewpoint image using the coordinate transformation matrix obtained, thereby generating a virtual multi-viewpoint image.
Here, the virtual multi-viewpoint image generating unit 103 may change the target multi-viewpoint image for each polygon mesh of the three-dimensional model projected onto the reference multi-viewpoint image. This is because the viewpoints that satisfy the first condition, the second condition, and the third condition for selecting the target multi-viewpoint image are different for different polygon meshes. In other words, the virtual multi-viewpoint image generating unit 103 may select an image of a different viewpoint as the target viewpoint image for each pixel of the reference multi-viewpoint image to generate a virtual multi-viewpoint image.

As shown in FIG. 4B, for each piece of viewpoint information, multi-viewpoint images (reference multi-viewpoint images and virtual multi-viewpoint images) captured by the imaging devices A1 to A3 having different spectral sensitivities are generated.
At this time, for pixels other than those corresponding to the polygon mesh vertex coordinates, virtual multi-viewpoint image generation unit 103 performs processing such as pixel integration and complementation so that pixels in the target multi-viewpoint image that have been subjected to coordinate transformation correspond to each of the pixels in the reference multi-viewpoint image. In other words, virtual multi-viewpoint image generation unit 103 performs correction processing to place each of the pixels in the virtual multi-viewpoint image to be generated in one-to-one correspondence with each of the pixels in the reference multi-viewpoint image.

In addition, the virtual multi-viewpoint image generation unit 103 generates a same-viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 107 corresponding to each piece of viewpoint information, and writes and stores each piece of information about the reference multi-viewpoint image and the virtual multi-viewpoint image in the same-viewpoint multi-viewpoint image table.
6 is a diagram showing an example of the configuration of a same-viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 107. The same-viewpoint multi-viewpoint image table is generated for each piece of viewpoint information, and therefore has a column for describing the viewpoint information. In addition, the same-viewpoint multi-viewpoint image table has columns for reference viewpoint image identification information, image data index, camera parameters, spectral sensitivity data index, and spectral distribution data index in a record corresponding to a reference multi-viewpoint image.

The reference viewpoint image identification information is the multi-viewpoint image identification information of the multi-viewpoint image that is set as the reference multi-viewpoint image. The image data index indicates the address of an area in which data of the multi-viewpoint image that is set as the reference multi-viewpoint image is written. The camera parameters indicate internal parameters of the imaging device that captured the multi-viewpoint image that is set as the reference multi-viewpoint image, and include the focal length of the imaging device when the viewpoint image was captured, the image center, the image resolution, and the distortion aberration coefficient. The spectral sensitivity index indicates the address of an area in which spectral sensitivity data of the imaging device that captured the multi-viewpoint image that is set as the reference multi-viewpoint image is written. The spectral sensitivity index indicates the address of an area in which spectral distribution data of the light source of the environment in which the multi-viewpoint image that is set as the reference multi-viewpoint image is captured is written.

The same-viewpoint multi-viewpoint image table also has columns for target viewpoint image identification information, image data index, camera parameters, spectral sensitivity data index, and spectral distribution data index in the records corresponding to the target multi-viewpoint image. The target viewpoint image identification information is the multi-viewpoint image identification information of the multi-viewpoint image that has been determined as the target multi-viewpoint image. The image data index indicates the address of the area where the data of the multi-viewpoint image that has been determined as the target multi-viewpoint image has been written.

The camera parameters indicate the internal parameters of the imaging device that captured the multi-view image that is the target multi-view image, and include the focal length, image center, image resolution, and distortion aberration coefficient of the imaging device when the viewpoint image was captured. The spectral sensitivity index indicates the address of an area in the multi-view image storage unit 107 into which the spectral sensitivity data of the imaging device that captured the multi-view image that is the target multi-view image is written. The spectral sensitivity index indicates the address of an area in the multi-view image storage unit 107 into which the spectral distribution data of the light source of the environment in which the multi-view image that is the target multi-view image was captured is written.

Returning to Figure 1, the spectral image generation unit 104 generates spectral images for each of multiple wavelength bands (e.g., bands every 10 nm from wavelengths 400 nm to 700 nm) from the multi-viewpoint images (multi-viewpoint images and virtual multi-viewpoint images) for each viewpoint information (Figure 4 (c)).
Here, the spectral image generation unit 104 refers to a same-viewpoint multi-viewpoint image table corresponding to each viewpoint information in the multi-viewpoint image storage unit 107, sequentially reads out each of the image data, spectral sensitivity data and spectral distribution data of the reference multi-viewpoint image and the virtual multi-viewpoint image, and estimates the spectral reflectance spectrum in a specified wavelength band for each pixel.

In this embodiment, the spectral image generating unit 104 reads in a plurality of spectral reflectance basis vectors from, for example, the spectral image storage unit 109. The spectral image generating unit 104 then expresses the spectral reflectance of the pixel of the multi-viewpoint image from each viewpoint as a weighted sum of the plurality of spectral reflectance basis vectors read in. The spectral image generating unit 104 estimates the weighting coefficients that weight each spectral reflectance basis vector, thereby estimating the spectral reflectance spectrum at each pixel of the reference multi-viewpoint image and the virtual multi-viewpoint image captured of the target object when restoring the three-dimensional shape model.

Here, the spectral reflectance basis vectors are derived in advance from a spectral reflectance data group, which is a collection of a plurality of pieces of spectral reflectance data, by using principal component analysis as spectral reflectance basis vectors for expressing the spectral reflectance data group.
In this principal component analysis, it becomes possible to express the spectral reflectance by a combination of a smaller number of spectral reflectance basis vectors than the actual dimensions of the spectral reflectance. In other words, by using the spectral reflectance basis vectors, the dimensions of the spectral reflectance to be estimated are compressed, and the calculation load is reduced.

That is, the spectral image generating unit 104 estimates the spectral reflectance spectrum s by expressing it as a weighted sum of a plurality of spectral reflectance basis vectors B = [ _b1 , _b2 , ...] ^T as shown in the following formula (1), and in order to perform this weighting, a weighting coefficient r = [r1, r2, ...] ^T for each of the spectral reflectance basis vectors is calculated to estimate the spectral reflectance spectrum s. Here, the spectral reflectance basis vectors are stored in advance in the spectral image storage unit 109, for example, as a set. In addition, the spectral reflectance basis vectors are stored in the spectral image storage unit 109 in a plurality of arrangements in descending order of principal component scores in principal component analysis, and a configuration may be adopted in which a predetermined number of spectral reflectance basis vectors are read and used as necessary.

Fig. 7 is a conceptual diagram showing how spectral reflectance is expressed by combining spectral reflectance basis vectors. As shown in Fig. 7, the spectral reflectance spectrum s is obtained by multiplying each of the spectral reflectance basis vectors _b1 to _bNr obtained by principal component analysis using the spectral reflectances of each of the predetermined objects by a weighting coefficient and combining the multiplication results.
That is, the spectral image generating unit 104 extracts a required number of spectral reflectance basis vectors b ₁ to b _Nr from the spectral image storage unit 109 .
Then, the spectral image generation unit 104 estimates the weighting coefficients _r1 to _rNr of the extracted spectral reflectance basis vectors _b1 to _bNr so as to obtain a spectral reflectance spectrum s that is the pixel value of each pixel of the reference multi-view image, and obtains the spectral reflectance spectrum s from the estimation result (described later).

Returning to FIG. 1, the spectral image generating unit 104 estimates each weighting coefficient r for the spectral reflectance spectrum expressed by the above formula (1) by solving the following formula (2). The spectral image generating unit 104 then estimates the spectral reflectance spectrum s by formula (1) using the estimated weighting coefficients. Here, the spectral reflectance spectrum si is represented as a 31-dimensional vector obtained by sampling the wavelength range from 400 nm to 700 nm in 10 nm increments, for example.

In the above formula (2), p _m,k is the pixel value of the k-th channel of the m-th multi-view image. That is, in p _m,k , m is a number indicating a multi-view image in the multi-view image group for generating a spectral image (reference multi-view image and virtual multi-view image) of the same-view multi-view image table. Also, k indicates the channel of the color component in a part in the multi-view image. For example, if a part is indicated by pixel values of color components R, G, and B, for example, channel (band) 1 is color component R, channel 2 is color component G, and channel 3 is color component B. Also, the pixels whose pixel values are read from each multi-view image are extracted from the image data of each virtual multi-view image as pixels corresponding to each of the reference multi-view images in the multi-view image group for generating a spectral image.

In addition, in formula (2), the matrix C _m,k indicates the spectral sensitivity of the k-th channel (wavelength band of color components) of the m-th multi-view image. The element c _m,k (λ) in C _m,k indicates the spectral sensitivity of the wavelength λ in the k-th channel of the m-th multi-view image in the multi-view image group for generating a spectral image.
The spectral distribution l of the light source is written as already measured spectral distribution data in the spectral image storage unit 109. The spectral image generation unit 104 reads out the spectral distribution of the light source in the environment in which each of the multi-viewpoint images in the group of multi-viewpoint images for generating spectral images was captured from the same-viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 107.

Here, the spectral image generating unit 104 may be configured to select and read out a set of spectral reflectance basis vectors from the spectral image storage unit 109 in accordance with the number of spectral sensitivities in the group of multi-viewpoint images for generating a spectral image.
Furthermore, when solving equation (2), the upper limit of the number of spectral reflectance basis vectors (principal components) used to represent the spectral reflectance data (hereinafter, the number of spectral reflectance basis vectors) _Nr is determined based on, for example, the number of spectral sensitivity groups that capture multi-viewpoint images (the number of imaging devices, the number of spectral sensitivity adjustments using color-changing filters, etc.).

When using multi-viewpoint images in q spectral sensitivity groups, it is determined by the relational expression " _Nr (number of spectral reflectance basis vectors) = q (number of spectral sensitivity types, number of spectral sensitivity groups) x k (number of channels, hereinafter referred to as channel number)." In other words, if the number _Nr of spectral reflectance basis vectors indicating the spectral reflectance data is equal to or less than k times the number of channels of the number q of spectral sensitivity groups, it is possible to solve expression (3).
As an example, when three imaging devices with different spectral sensitivities are used and there are three channels, R, G, and B, the upper limit of the basis vectors is 9 (=3×3).

However, in order to suppress the influence of noise when solving equation (3), it is desirable to operate the number of spectral reflectance basis vectors _Nr as less than k times the number of channels of the number of spectral sensitivity groups q, for example, as "Nr = q x k - 1." In this case, when three imaging devices with different spectral sensitivities are used and there are three channels, R, G, and B, the upper limit of the spectral reflectance basis vectors is 8 (= 3 x 3 - 1).

In addition, a group of multi-viewpoint images for generating a spectral image may be obtained for each piece of viewpoint information, and the number of pixels in the virtual multi-viewpoint image for each pixel in the reference multi-viewpoint image, i.e., the number of spectral sensitivity groups q, may differ. In this case, the set of spectral reflectance basis vectors (the number of spectral reflectance basis vectors _Nr ) that the spectral image generating unit 104 reads out from the spectral image storage unit 109 differs for each pixel in the reference multi-viewpoint image. Each pixel of the spectral images is configured to correspond to each pixel in the reference multi-viewpoint image.

Through the above-described processing, the spectral image generation unit 104 estimates the spectral reflectance spectrum of each pixel of the spectral image corresponding to each pixel of the reference multi-viewpoint image in the same-viewpoint multi-viewpoint image table from the pixel values of corresponding pixels between the reference multi-viewpoint image and the virtual multi-viewpoint image, generates a spectral image for each piece of viewpoint information (a spectral image with a predetermined number of wavelength bands), and writes and stores the spectral image table for each piece of viewpoint information in the spectral image storage unit 109.
Fig. 8 is a diagram showing an example of the configuration of a spectral image table stored in the spectral image storage unit 109 in this embodiment. The spectral image table shown in Fig. 8 has a column for viewpoint information, and also has columns for band wavelengths and spectral image data indexes corresponding to the spectral image identification information for each record.

Here, the viewpoint information is the viewpoint information of the spectroscopic image in the spectroscopic image table. The spectroscopic image identification information is identification information that identifies each spectroscopic image (e.g., a number indicating the order in which they were supplied). The wavelength of the band indicates the wavelength band of the spectral reflectance spectrum of the corresponding spectroscopic image. The spectroscopic image data index is an address that indicates the area in the spectroscopic image storage unit 109 where the image data of the spectroscopic image is written.

Returning to Figure 1, the spectral texture image generation unit 105 reads out data of the polygon mesh that forms the surface of the three-dimensional shape model from the three-dimensional shape model storage unit 108, and generates an unfolded view of the polygon mesh (e.g., an unfolded view obtained by performing UV unfolding) based on the brightness values of each of the spectral images at each viewpoint information.
9A and 9B are conceptual diagrams showing the generation of development views of polygon meshes in a three-dimensional shape model. Fig. 9A shows a three-dimensional shape model in which surfaces are configured with polygon meshes. Fig. 9B shows development views (UV development views as an example) in which each of the polygon meshes in the above surfaces is developed in a two-dimensional coordinate system.

For example, polygon meshes 651 and 602 in Fig. 9A correspond to regions 801 and 802 in Fig. 9B, respectively. That is, polygon mesh 651 is developed in region 801 of the development view, and polygon mesh 652 is developed in region 802 of the development view.
Here, each of the vertices (hereinafter referred to as polygon mesh vertices) 701 to 704 of the polygon mesh 651 of the three-dimensional shape model in the world coordinate system corresponds to each of the vertices (hereinafter referred to as area vertices) 951, 952, 953, and 954 of the area 801 of the development drawing in the two-dimensional coordinate system.
Similarly, each of the polygon mesh vertices 703 to 706 of the polygon mesh 652 of the three-dimensional shape model in the world coordinate system corresponds to each of the area vertices 953, 954, 955, and 956 of the area 801 of the development in the two-dimensional coordinate system.

The spectral texture image generating unit 105 generates a correspondence coordinate lookup table indicating the correspondence between the vertex coordinate values of the polygon mesh in the above-mentioned three-dimensional shape model and the area vertex coordinate values (coordinates of the area vertices) in the development.
FIG. 10 is a diagram showing an example of the configuration of a correspondence coordinate lookup table indicating the correspondence between the vertex coordinate values of a polygon mesh in a three-dimensional shape model and the vertex coordinate values of an area in a development.
The corresponding coordinate lookup table indicates the correspondence between polygon mesh vertex coordinate values and region vertex coordinate values on a record-by-record basis in response to region identification information. Region identification information is identification information that individually identifies each region in the development. Polygon mesh vertex coordinate values are the coordinate values of each vertex of a polygon mesh in the world coordinate system. Region vertex coordinate values are the coordinate values of each vertex of a region (texture region, described below) in the development.

For example, in FIG. 10, if the texture region identification information P10001 indicates the polygon mesh 651 in FIG. 9(a), the coordinate values of polygon mesh vertex 701 are polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), the coordinate values of polygon mesh vertex 702 are polygon mesh vertex coordinate values (x1_2, y1_2, z1_2), the coordinate values of polygon mesh vertex 703 are polygon mesh vertex coordinate values (x1_3, y1_3, z1_3), and the coordinate values of polygon mesh vertex 704 are polygon mesh vertex coordinate values (x1_4, y1_4, z1_4).

Also, in Figure 10, when showing area 801 in Figure 9 (b), the coordinate values of area vertex 951 are area vertex coordinate values (U1_1, V1_1), the coordinate values of area vertex 952 are area vertex coordinate values (U1_2, V1_2), the coordinate values of area vertex 953 are area vertex coordinate values (U1_3, V1_3), and the coordinate values of area vertex 954 are area vertex coordinate values (U1_4, V1_4).
According to this corresponding coordinate lookup table, it can be seen that the polygon mesh formed by the polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), (x1_2, y1_2, z1_2), (x1_3, y1_3, z1_3) and (x1_4, y1_4, z1_4) is deployed in an area whose vertices are the area vertex coordinate values (U1_1, V1_1), (U1_2, V1_2), (U1_3, V1_3) and (U1_4, V1_4) in the development drawing. Furthermore, the corresponding coordinate lookup table indicates that each of the polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), (x1_2, y1_2, z1_2), (x1_3, y1_3, z1_3) and (x1_4, y1_4, z1_4) correspond to the area vertex coordinate values (U1_1, V1_1), (U1_2, V1_2), (U1_3, V1_3) and (U1_4, V1_4), respectively.

The spectral texture image generating unit 105 reads out each of the spectral images of the same wavelength band from each of the same-viewpoint multi-viewpoint image tables provided for each piece of viewpoint information in the spectral image storage unit 109 .
Then, the spectral texture image generating unit 105 extracts correspondence between pixels in each of the spectral images and polygon mesh vertex coordinate values of the polygon mesh of the three-dimensional shape model. Here, the spectral texture image generating unit 105 projects the polygon mesh of the three-dimensional shape model onto the spectral images in correspondence with, for example, each of the viewpoint information and camera parameters, and obtains correspondence between the polygon mesh vertex coordinate values (i.e., each vertex of the polygon mesh) and pixels (coordinate values of pixels in the coordinate system of the spectral images).

The spectral texture image generation unit 105 assigns pixel values (spectral reflectance spectrum) of pixels in a pixel region surrounded by pixel coordinate values of the spectral image corresponding to the polygon mesh vertex coordinate values as pixel values of pixels in each region of the development diagram obtained by developing the polygon mesh.
At this time, the spectral texture image generating unit 105 preferentially assigns to the pixels of the spectral texture image the pixel values of the spectral image corresponding to the viewpoint information having the optical axis vector that is most parallel to the normal of each face of the polygon mesh in the three-dimensional shape model.

The spectral texture image generation unit 105 then performs the above-mentioned processing on all of the spectral images, thereby assigning pixel values to pixels in all areas of the development view, and generating a spectral texture image of a wavelength band corresponding to the wavelength band of the used spectral image.
The spectral texture image generation unit 105 performs the above-mentioned process of generating a spectral texture image on each of the spectral images of all wavelength bands. For example, if there are 31 wavelength bands, then 31 spectral texture images corresponding to each wavelength band are generated (Figure 4(d)).
Here, the spectral texture image generating unit 105 may perform the above-mentioned processing one band at a time, or may perform the above-mentioned processing on multiple bands at once. For example, considering that many image formats are expressed in three bands of RGB, processing can be performed efficiently by processing three bands at a time as a single image.

The spectral texture image generating unit 105 writes and stores each of the generated spectral texture images in the spectral texture image storage unit 110 in combination with the respective wavelength bands.
FIG. 11 is a diagram showing an example of the configuration of a spectral texture image table in the spectral texture image storage unit 110. As shown in FIG.
The spectral texture image table has columns for spectral texture image identification information, band wavelength, and spectral texture image data index for each record.

The spectral texture image identification information is identification information for individually identifying each of the spectral texture images. The band wavelength indicates the wavelength of the band (channel) of the spectral texture image indicated by the spectral texture image identification information. The spectral texture image data index indicates the address of the area in the spectral texture image storage unit 110 where the image data of the spectral texture image is written.
The spectral texture image table in FIG. 11 corresponds to, for example, each of the spectral texture images of 31 bands of wavelengths in 10 nm increments from a wavelength of 400 nm to a wavelength of 700 nm.

The visualization texture image generating unit 106 renders a visualized spectral texture image on the display unit under predetermined conditions for the three-dimensional shape model read out from the three-dimensional shape model storage unit 108 .
The visualization texture image generation unit 106 converts the spectral reflectance of the spectral texture image (the spectral reflectance spectrum of each of all the spectral images) into pixel values of wavelength bands that can be displayed on the display screen based on the following three predetermined conditions, and displays them on the display screen.

FIG. 12 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #1 being rendered onto a three-dimensional shape model.
Pattern #1 shows an image from a predetermined viewpoint position and viewpoint direction (corresponding to the optical axis vector) of a three-dimensional shape model in which a texture image of RGB values is rendered using arbitrary spectral sensitivities corresponding to three bands of RGB (red, green, blue) color components that can be displayed on a display screen and the spectral distribution of an arbitrary light source, as the spectral reflectance of each pixel in the spectral texture image. Here, the arbitrary spectral sensitivity may be, for example, the spectral sensitivity of an imaging device or a color matching function. The color matching function may be, for example, the CIE (Commission Internationale de Illumination) colorimetric observer color matching function. The spectral distribution of the light source may be, for example, the spectral distribution of a standard light source D65, or the spectral distribution of sunlight, an LED (light emitting diode), a fluorescent lamp, an incandescent lamp, or the like. The data of the spectral sensitivity and the spectral distribution are written and stored in a storage unit (not shown) in advance, and the visualization texture image generating unit 106 reads them out as necessary.

In processing this pattern #1, the visualization texture image generating unit 106 reads out the three-dimensional shape model 901 from the three-dimensional shape model storage unit 108.
Then, the visualization texture image generation unit 106 refers to the spectral image storage unit 109 and generates a texture image 902 in which the spectral reflectance based on the pixel values of each pixel of the spectral texture image is expressed in RGB values using an arbitrary spectral sensitivity and the spectral distribution of an arbitrary light source.
The visualization texture image generating unit 106 displays a three-dimensional shape model 903 obtained by rendering the texture image 902 onto the three-dimensional shape model 901 as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 13 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #2 being rendered onto a three-dimensional shape model.
Pattern #2 shows an image in which the gradations of three arbitrary wavelength bands are displayed as gradations of each band of RGB values that can be displayed on the display screen.
13 , the 440 nm wavelength band is replaced with RGB color component B, and the pixel values (spectral reflectance spectrum) of the spectral texture image at wavelength 440 nm are set as pixel values of color component B. Similarly, the 500 nm wavelength band is replaced with RGB color component G, and the pixel values (spectral reflectance spectrum) of the spectral texture image at wavelength 500 nm are set as pixel values of color component G. Furthermore, the 660 nm wavelength band is replaced with RGB color component R, and the pixel values (spectral reflectance spectrum) of the spectral texture image at wavelength 660 nm are set as pixel values of color component R.

In processing this pattern # 2 , the visualization texture image generating unit 106 reads out the three-dimensional shape model 913 from the three-dimensional shape model storage unit 108 .
Then, the visualization texture image generation unit 106 refers to the spectral image storage unit 109, and sets the pixel values of each pixel of the spectral texture image of any three wavelength bands to the pixel values of the color components R, G, and B respectively (graph 911), thereby generating a texture image 912 expressed in RGB values.
The visualization texture image generating unit 106 renders the texture image 912 onto a three-dimensional shape model 913 to generate a three-dimensional shape model 914, which is then displayed as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 14 is a conceptual diagram showing an example of a spectral texture image processed under the conditions of pattern #3 being rendered onto a three-dimensional shape model.
FIG. 14A illustrates, as pattern #3, a process of allocating pixel values in a spectral texture image to pixel values of each of the RGB color components that can be displayed on a display device.
14A, the range T of the spectral reflectance spectrum R(λi) of each pixel in any one of the spectral texture images is divided into three ranges, for example, a high value range H, a medium value range M, and a low value range L. λi indicates the wavelength of each band of the spectral texture image (for example, in the case of 31 bands, 1≦i≦31).
The conversion rule for pattern #3 is shown, in which the spectral reflectance spectrum intensity in the high numerical range H is allocated to the gradient range of color component R, the spectral reflectance spectrum intensity in the medium numerical range M is allocated to the gradient range of color component G, and the spectral reflectance spectrum intensity in the low numerical range L is allocated to the gradient range of color component B.

In FIG. 14B, the visualization texture image generating unit 106 reads out the three-dimensional shape model 921 from the three-dimensional shape model storage unit 108 .
Then, the visualization texture image generation unit 106 refers to the spectral image storage unit 109, and assigns the spectral reflectance spectrum intensity, which is the pixel value of each pixel of the spectral texture image of any one wavelength band, to pixel values in the gradient ranges of each of the color components R, G, and B in accordance with the conversion rule described above ( FIG. 4( a) ), thereby generating a texture image 922 expressed in RGB values.
The visualization texture image generating unit 106 displays a three-dimensional shape model 923 obtained by rendering the texture image 922 onto the three-dimensional shape model 921 as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 15 is a flowchart showing an example of the operation of a process for generating a spectral texture image for a three-dimensional shape model from multi-viewpoint images according to this embodiment.
Step S101:
The data input/output unit 101 adds information on the spectral sensitivity at the time when each of the viewpoint images in the multi-viewpoint image supplied from the external device to the multi-viewpoint image storage unit 107 for storage.
In addition, the data input/output unit 101 writes and stores in the multi-viewpoint image storage unit 107 data on a three-dimensional shape model of a surface model (the surface is composed of a polygon mesh) restored from the multi-viewpoint image supplied from an external device.

Step S102:
The multi-viewpoint image classification unit 102 refers to the multi-viewpoint image storage unit 107 and classifies each of the multi-viewpoint images into a spectral sensitivity group in accordance with the spectral sensitivity added to the multi-viewpoint image.
Then, the multi-viewpoint image classification unit 102 writes and stores each of the multi-viewpoint images classified into the spectral sensitivity groups in the corresponding spectral sensitivity group data table.

Step S103:
The virtual multi-viewpoint image generation unit 103 sets a multi-viewpoint image at each viewpoint position as a reference multi-viewpoint image, and sets a multi-viewpoint image at another viewpoint position near this reference multi-viewpoint image, which belongs to a different spectral sensitivity group from the reference multi-viewpoint image, as a target multi-viewpoint image.
Then, the virtual multi-viewpoint image generator 103 selects, from each of the target multi-viewpoint images, a target multi-viewpoint image that can reproduce a virtual viewpoint image that is assumed to have been captured with the same viewpoint information as the reference multi-viewpoint image.
Then, the virtual multi-viewpoint image generation unit 103 converts each of the selected target multi-viewpoint images into an image captured from the same viewpoint position and optical axis vector as the reference multi-viewpoint image, generates a virtual multi-viewpoint image, and writes and stores it in the same multi-viewpoint image table in the multi-viewpoint image storage unit 107.

Step S104:
The spectral image generation unit 104 uses, for each piece of viewpoint information, each pixel of the multi-viewpoint images (reference multi-viewpoint image and virtual multi-viewpoint image) in the same multi-viewpoint image table, the spectral sensitivity of the imaging device that captured each multi-viewpoint image, and the spectral distribution of the light source to calculate the spectral reflectance spectrum of each pixel of the spectral images (corresponding to a pixel of the reference multi-viewpoint image) corresponding to each wavelength band, using each of equations (1) and (2).
Then, the spectral image generating unit 104 generates spectral images (31 images) of predetermined wavelength bands (31 types) for each piece of viewpoint information, and writes and stores them in a spectral image table in the spectral image storage unit 109 .

Step S105:
The spectral texture image generation unit 105 creates an unfolded view by unfolding the polygon mesh of the three-dimensional shape model, and assigns the intensity of the spectral reflectance spectrum as the pixel values of the pixels in each region in this unfolded view, which are surrounded by the pixel coordinate values of the spectral image corresponding to the vertex coordinate values of the polygon mesh.

Then, the spectral texture image generation unit 105 generates a spectral texture image corresponding to one wavelength band by assigning the intensity of the spectral reflectance spectrum of each spectral image of the viewpoint information as a pixel value to each pixel in all areas of the development drawing.
The spectral texture image generating unit 105 writes and stores the generated spectral texture image in a spectral texture image table in the spectral texture image storage unit 110 in association with the wavelength band of the spectral image used for generation.

Step S106:
The spectral texture image generating unit 105 determines whether or not spectral texture images for all wavelength bands have been generated.
At this time, if the spectral texture image generating unit 105 determines that the spectral texture images for all wavelength bands have been generated, the processing ends.
On the other hand, if the spectral texture image generating unit 105 determines that spectral texture images for all wavelength bands have not been generated, the process proceeds to step S105.

As described above, according to this embodiment, a virtual multi-viewpoint image with different spectral sensitivity for each viewpoint information is generated from a multi-viewpoint captured image captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a digital camera available on the market that is available to general users. This makes it possible to obtain a multi-viewpoint image corresponding to multiple spectral sensitivities for each viewpoint information unit, and generate a spectral image similar to a hyperspectral image for each viewpoint information. Compared to an image captured with normal RGB color components, the spectral reflectance of the target object can be obtained as a more detailed spectral reflectance spectrum, and a spectral texture image for a three-dimensional shape model can be easily generated as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of a specified number of wavelength bands (a number of bands exceeding the three bands of RGB color components (e.g., 31 bands of different wavelengths)).

In addition, according to this embodiment, a spectral texture image of multiple wavelength bands (e.g., 31 bands) is generated, and the pixel values of the spectral texture images of each wavelength are made to correspond to the pixel values of the respective RGB color components so that the image can be observed on a display device that displays the RGB color components. This allows the texture mapping onto the restored three-dimensional shape model, and the texture-mapped three-dimensional shape model to be observed on the display device, thereby allowing the state of the spectral reflectance of the target object to be viewed in more detail.

Second Embodiment
A second embodiment of the present invention will now be described with reference to the drawings.
FIG. 16 is a block diagram showing an example of the configuration of a spectral texture image generating system according to the second embodiment of the present invention.
16, the spectral texture image generation system 100A includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a virtual multi-viewpoint image generation unit 103, a texture image generation unit 111, a spectral texture image generation unit 112, a visualization texture image generation unit 106, a multi-viewpoint image storage unit 107, a three-dimensional shape model storage unit 108, a texture image storage unit 113, and a spectral texture image storage unit 110. Below, configurations and operations that differ from the first embodiment will be described.

17 is a conceptual diagram for explaining generation of a spectral texture image by the spectral texture image generation system of the second embodiment. As the imaging devices, for example, three imaging devices A1, A2, and A3 having different spectral sensitivities are used, as in the case of FIG.
In Figure 17, the process of capturing multi-viewpoint images of a target object 500 at different viewpoint positions and optical axis vectors (viewpoint information conditions) in Figure 17(a), and the process of generating a virtual multi-viewpoint image from the target multi-viewpoint image in Figure 17(b) are similar to those in Figures 4(a) and 4(b), respectively.

17C shows a process of generating texture images by extracting multi-viewpoint images of the same spectral sensitivity (reference multi-viewpoint image and virtual multi-viewpoint image) with reference to each of the same multi-viewpoint image tables for each viewpoint information in the multi-viewpoint image storage unit 107. Texture images 951, 952, and 953 of spectral sensitivity B1 are generated from multi-viewpoint images captured by different imaging devices having spectral sensitivities B1, B2, and B3, respectively.
17D shows a process for generating, from texture images with different spectral sensitivities, spectral texture images (hyperspectral images) for a three-dimensional shape model of a target object 500. The spectral texture images are generated, for example, at wavelengths from 400 nm to 700 nm, every 10 nm (described later).

Returning to Figure 16, the texture image generation unit 111 refers to each of the same multi-viewpoint image tables for each viewpoint information in the multi-viewpoint image storage unit 107, extracts multi-viewpoint images with the same spectral sensitivity (reference multi-viewpoint image and virtual multi-viewpoint image), and generates a texture image.
Here, the texture image generating unit 111 reads out the data of the three-dimensional shape model from the three-dimensional shape model storage unit 108 .
The texture image generator 111, like the spectral texture image generator 105 in the first embodiment, generates a development of a polygon mesh of a three-dimensional shape model by the process described in Fig. 9. Here, for the development corresponding to each texture image, a corresponding coordinate lookup table that differs for each spectral sensitivity of the multi-viewpoint image is generated.

Then, the texture image generation unit 111 assigns pixel values to pixels in the areas of the multi-viewpoint images (reference multi-viewpoint image and virtual multi-viewpoint image) in each piece of viewpoint information onto which a polygon mesh corresponding to each area in the development drawing is projected.
The texture image generating unit 111 writes and stores each of the texture images generated corresponding to each of the spectral sensitivities in a texture image table in the texture image storage unit 113 (FIG. 17(c)).
At this time, when generating each texture image, the texture image generating unit 111 generates a corresponding coordinate lookup table shown in FIG. 10 corresponding to each texture image.

FIG. 18 is a diagram showing an example of the configuration of a texture image table in the texture image storage unit 113 in the second embodiment.
The texture image table has columns for texture image identification information, a spectral sensitivity data index, a texture image data index, and a lookup table index for each record.
The texture image identification information is identification information that individually identifies each of the texture images generated for each spectral sensitivity.

The spectral sensitivity data index is an address indicating an area in the texture image storage unit 113 into which the spectral sensitivity data corresponding to the texture image has been written. The texture image index is an address indicating an area in the texture image storage unit 113 into which the texture image data has been written. The lookup table index is an address indicating an area in the texture image storage unit 113 into which the corresponding coordinate lookup table data has been written.

The spectral texture image generator 112 refers to a texture image table in the texture image storage unit 113, and generates a spectral texture image for each wavelength band using the spectral sensitivity data of each texture image, the pixel values of the pixels in the texture image, and a corresponding coordinate lookup table of the texture image.
Here, the spectral texture image generating unit 112 estimates the spectral reflectance spectrum of each pixel in the spectral texture image of each wavelength band using the already explained equations (1) and (2).

At this time, the spectral texture image generator 112 detects texture regions that correspond to the same polygon mesh in each texture image by extracting combinations of the same polygon mesh vertex coordinate values in the corresponding coordinate lookup table of each texture image.
The spectral texture image generator 112 uses the pixel values (RGB color components) of each pixel in the detected texture region to estimate the spectral reflectance spectrum of each wavelength band of the pixel in the spectral texture image corresponding to that pixel position, using equations (1) and (2).

Here, the spectral texture image generating unit 112 selects one of the texture images, and generates a spectral texture image by assigning the acquired spectral reflectance spectrum intensity as the pixel value of each pixel in the texture region of the development view of this texture image (Figure 17 (d)).
Then, each of the generated spectral texture images is combined with the corresponding wavelength band and written and stored in the spectral texture image storage unit 110 shown in FIG.

As described above, according to this embodiment, a virtual multi-viewpoint image with different spectral sensitivities for each viewpoint is generated from a multi-viewpoint captured image captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a digital camera available on the market that is available to general users. By preparing multi-viewpoint images corresponding to multiple spectral sensitivities for each viewpoint information, it is possible to generate texture images for each spectral sensitivity. As with the case of generating texture images with different spectral sensitivities from hyperspectral images, a spectral texture image for a three-dimensional shape model can be easily generated as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of multiple finer wavelength bands (a number of bands exceeding the three bands of the color components RGB (e.g., 31 bands of different wavelengths)) compared to an image captured with the normal color components RGB.

Third Embodiment
A third embodiment of the present invention will be described below with reference to the drawings. The configuration of the spectral texture image generation system according to the third embodiment is similar to that of the second embodiment (the spectral texture image is generated by the process of FIG. 17). In the following description, operations different from those of the second embodiment will be described.
The texture image generation unit 111 generates a texture image using all of the multi-viewpoint images included in the same-viewpoint multi-viewpoint image table, corresponding to the spectral sensitivity of any of the multi-viewpoint images (reference multi-viewpoint image and virtual multi-viewpoint image), for example, the reference multi-viewpoint image in the same-viewpoint multi-viewpoint image table of the viewpoint information listed first in the multi-viewpoint image storage unit 107.
At this time, the texture image generating unit 111 writes and stores the corresponding coordinate lookup table (similar to the table in FIG. 10) created when generating the texture image in the texture image storage unit 113 as a reference corresponding coordinate lookup table.

Then, the texture image generation unit 111 refers to the reference correspondence coordinate lookup table, and generates a texture image in which the polygon mesh vertex coordinate values and the region vertex coordinate values each have a similar correspondence relationship for each correspondence between the polygon mesh vertex coordinate values and the region vertex coordinate values in the reference correspondence coordinate lookup table.
That is, the texture image generating unit 111 refers to the reference corresponding coordinate lookup table, and generates each of the texture images of the spectral sensitivities other than the spectral sensitivity for which this reference corresponding coordinate lookup table was generated.

FIG. 19 is a diagram showing an example of the configuration of a texture image table in the texture image storage unit 113 in the third embodiment.
The texture image table has columns for texture image identification information, a spectral sensitivity data index, a texture image data index, and a lookup table index for each record.
The texture image identification information is identification information that individually identifies each of the texture images generated for each spectral sensitivity.

The spectral sensitivity data index is an address indicating an area in the texture image storage unit 113 in which the spectral sensitivity data corresponding to the texture image has been written. The texture image index is an address indicating an area in the texture image storage unit 113 in which the data of the texture image has been written.
In addition, the spectral texture image generation unit 112 refers to the texture image table in the texture image storage unit 113, and uses each of the pixels in the texture region based on the region vertex coordinate values of each texture image, and uses equations (1) and (2) to determine the spectral reflectance spectrum of each pixel in the texture region in the spectral texture image of each wavelength band, thereby generating a spectral texture image.
Then, each of the generated spectral texture images is combined with the corresponding wavelength band and written and stored in the spectral texture image storage unit 110 shown in FIG.

Fourth Embodiment
Hereinafter, a third embodiment of the present invention will be described with reference to the drawings.
Fig. 20 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a fourth embodiment of the present invention. Similarly to the second embodiment, the spectral texture image generation system according to this embodiment generates a spectral texture image by the process shown in Fig. 17.
20, the spectral texture image generation system 100B includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a virtual multi-viewpoint image generation unit 103, a texture image generation unit 111, a texture image reconstruction unit 114, a spectral texture image generation unit 112, a visualization texture image generation unit 106, a multi-viewpoint image storage unit 107, a three-dimensional shape model storage unit 108, a texture image storage unit 113, and a spectral texture image storage unit 110. Below, configurations and operations that differ from the first embodiment will be described.

The texture image generating unit 111 generates a texture image for each spectral sensitivity, similarly to the second embodiment.
At this time, when generating texture images for each of the spectral sensitivities, the texture image generating unit 111 generates a corresponding coordinate lookup table corresponding to each of the spectral sensitivities.
Then, the texture image reconstruction unit 114 sets the corresponding coordinate lookup table corresponding to the spectral sensitivity of a multi-viewpoint image (reference multi-viewpoint image and virtual multi-viewpoint image) of any of the spectral sensitivity, for example, the reference multi-viewpoint image in the same-viewpoint multi-viewpoint image table of the viewpoint information listed first in the multi-viewpoint image storage unit 107, as the reference corresponding coordinate lookup table.

The texture image reconstruction unit 114 compares the reference correspondence coordinate lookup table in the texture image storage unit 113 with the correspondence coordinate lookup tables of texture images of other spectral sensitivities, matches them with the developments of the texture images that correspond to the reference correspondence coordinate lookup table, and changes the arrangement of the texture regions in the other texture images.
That is, the texture image reconstructing unit 114 compares the correspondence between the polygon mesh vertex coordinates and the region vertex coordinates in the correspondence coordinate lookup table with the correspondence between the polygon mesh vertex coordinates and the region vertex coordinates in the reference correspondence coordinate lookup table.

Then, the texture image reconstructing unit 114 extracts a polygon mesh formed from the polygon mesh vertex coordinates in each of the corresponding coordinate lookup table and the reference corresponding coordinate lookup table.
The texture image reconstruction unit 114 reconstructs the correspondence between the polygon mesh vertex coordinates and the region vertex coordinates so that the polygon mesh vertex coordinates and the region vertex coordinates in the reference correspondence coordinate lookup table are identical to the polygon mesh vertex coordinates and the region vertex coordinates in the correspondence coordinate lookup table, and changes the correspondence coordinate lookup table.

The spectral texture image generation unit 112 matches the region vertex coordinates in the correspondence coordinate lookup table and the reference correspondence coordinate lookup table, and uses each of the pixels in the texture region based on the region vertex coordinate values of each texture image to determine the spectral reflectance spectrum of each pixel in the texture region in the spectral texture image of each wavelength band using equations (1) and (2), thereby generating a spectral texture image.
Then, each of the generated spectral texture images is combined with the corresponding wavelength band and written and stored in the spectral texture image storage unit 110 shown in FIG.

Fifth embodiment
Hereinafter, a fifth embodiment of the present invention will be described with reference to the drawings.
Fig. 21 is a block diagram showing an example of the configuration of a spectral texture image generation system according to the fifth embodiment of the present invention. In Fig. 21, the spectral texture image generation system 100C includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a virtual multi-viewpoint image generation unit 103, a spectral distribution estimation unit 115, a spectral image generation unit 104, a spectral texture image generation unit 105, a visualization texture image generation unit 106, a multi-viewpoint image storage unit 107, a three-dimensional shape model storage unit 108, a spectral distribution basis vector storage unit 116, a spectral image storage unit 109, and a spectral texture image storage unit 110.

In this embodiment as well, captured image data of a target object to be restored as a three-dimensional shape model is supplied as a viewpoint image from an external device (not shown). Only configurations and operations that differ from the first embodiment will be described below.
In the first to fourth embodiments, the spectral distribution of the light source when each multi-viewpoint image is captured is already acquired as measured spectral distribution data.
In the fifth embodiment, when estimating the spectral reflectance, the spectral distribution is also estimated.

The spectral distribution estimation unit 115 estimates the spectral distribution of a light source in the environment in which a multi-viewpoint image is captured, using the multi-viewpoint image made up of viewpoint images captured by imaging devices with different spectral sensitivities.
The spectral distribution estimation unit 115 reads in a plurality of spectral distribution basis vectors, for example, from the spectral distribution basis vector storage unit 116. Then, similar to the case of the spectral reflectance, the spectral distribution estimation unit 115 expresses the spectral distribution of the light source in the environment in which the multi-viewpoint image is captured as a weighted sum of the plurality of read-in spectral distribution basis vectors. The spectral distribution estimation unit 115 estimates the spectral distribution of the light source by estimating a weighting coefficient that weights each of the spectral distribution basis vectors.

Here, the spectral distribution basis vectors are derived from a spectral distribution data group, which is a collection of the spectral distributions of a plurality of light sources, by using principal component analysis as spectral distribution basis vectors for expressing the spectral reflectance data group.
In this principal component analysis, as in the case of spectral reflectance, it becomes possible to express the spectral distribution by combining a smaller number of spectral distribution basis vectors than the actual dimensions of the spectral distribution. In other words, by using the spectral distribution basis vectors, the dimensions of the spectral distribution of the light source to be estimated are compressed, and the calculation load is reduced.

That is, the spectral distribution estimation unit 115 estimates the spectral distribution l by expressing it as a weighted sum of a plurality of spectral distribution basis vectors _ej = [ _ej(λ1) , _ej(λ2) , ...] ^T and finding the weighting coefficient a = [ _a1 , _a2 , ...] ^T of each of the spectral distribution basis vectors, as shown in the following formula (3). Here, the spectral distribution basis vectors are, for example, stored in advance as a set in the spectral distribution basis vector storage unit 116. Also, the spectral distribution basis vector storage unit 116 may be configured to store a plurality of spectral distribution basis vectors arranged in descending order of principal component scores in the principal component analysis, and to read and use the spectral distribution basis vectors up to a predetermined order as necessary (described later).
The synthesis of this spectral distribution basis vector is similar to the case where the spectral reflectance is expressed by synthesizing the spectral reflectance basis vectors shown in FIG.

Then, the spectral distribution estimation unit 115 estimates each weighting coefficient a by solving the following equation (4) for the spectral distribution expressed by the above equation (3). The spectral distribution estimation unit 115 then estimates the spectral distribution l by equation (3) using the estimated weighting coefficients. Here, the spectral distribution l is expressed as a 31-dimensional vector in which the wavelength range from 400 nm to 700 nm is sampled in 10 nm increments, for example, as in the case of the spectral reflectance.

In the above formula (4), p _m,k is the pixel value of the k-th channel of the m-th viewpoint image. That is, in p _m,k , m is a number indicating a viewpoint image in the viewpoint group for estimating spectral reflectance. Furthermore, k indicates the channel of a color component in a part in the viewpoint image. For example, if a part is indicated by pixel values of color components R, G, and B, for example, channel 1 is color component R, channel 2 is color component G, and channel 3 is color component B.

In addition, in formula (4), the matrix C _m,k indicates the spectral sensitivity of the k-th channel (wavelength band of color components, i.e., band) of the m-th viewpoint image. The element c _m,k (λ) in C _m,k indicates the spectral sensitivity of the wavelength λ in the k-th channel of the m-th viewpoint image.
The viewpoint images used in equation (4) are obtained by sampling several parts of a three-dimensional shape model, and are viewpoint images of a group of viewpoints for estimating the spectral reflectance of the sampled parts.
Then, the spectral distribution estimation unit 115 estimates the spectral distribution of the light source by solving equation (4) using the spectral reflectance of the sampled portion, the pixel value corresponding to the portion in each viewpoint image, and the spectral sensitivity of the imaging device that captured the viewpoint image. The portion to be sampled may be selected from the three-dimensional shape model, or may be selected from the pixels in the viewpoint image.

Here, similar to the case of estimating spectral reflectance, the spectral distribution estimation unit 115 may be configured to select and read out a set of spectral distribution base vectors from the spectral distribution base vector storage unit 116 in accordance with the number of spectral sensitivity groups in the group of viewpoints for estimating spectral reflectance.
Furthermore, when solving equation (3), the upper limit of the number of spectral distribution basis vectors (principal components) used to represent the spectral distribution data (hereinafter, the number of spectral distribution basis vectors) _Nχ is determined based on, for example, the number of spectral sensitivity groups that capture multi-viewpoint images (the number of image capture devices, the number of adjustments of spectral sensitivity using color-changing filters, etc.).

In addition, a group of viewpoints for estimating spectral reflectance is obtained for each part of the three-dimensional shape model, and the number m of spectral sensitivity groups in each part is different. Therefore, the set of spectral distribution basis vectors (the number of spectral distribution basis vectors N _χ ) that the spectral distribution estimation unit 115 reads out from the spectral distribution basis vector storage unit 116 is different for each part of the three-dimensional shape model.

When estimating the spectral reflectance of each portion of a three-dimensional shape model and the spectral distribution of a light source, the estimation of the spectral reflectance and the spectral distribution is alternately repeated multiple times.
When starting with estimating the spectral reflectance, the spectral texture image generation unit 112 reads the spectral distribution of a preset default light source from the spectral image storage unit 109, and estimates the spectral reflectance using equations (1) and (2).
Next, when estimating the spectral distribution, the spectral distribution estimating unit 115 uses the spectral reflectance obtained from the above-mentioned default light source spectral distribution to estimate the spectral distribution using equations (3) and (4).

On the other hand, when starting from estimating the spectral distribution, the spectral distribution estimation unit 115 estimates the spectral distribution using equations (3) and (4) by using a default spectral reflectance that is set in advance as the spectral reflectance of the portion sampled from the three-dimensional shape model.
Next, when estimating the spectral reflectance, the spectral texture image generating unit 112 uses the spectral distribution obtained from the default spectral reflectance and estimates the spectral reflectance using equations (1) and (2).

As described above, according to this embodiment, by repeatedly estimating the spectral reflectance of each voxel portion of a three-dimensional shape model and estimating the spectral distribution using a portion sampled from that portion, it is possible to determine with improved estimation accuracy the spectral reflectance of each portion of the three-dimensional shape model and the spectral distribution of the light source in the environment in which the multi-view image was captured.
In addition, at this time, the number of times to repeat the estimation of the spectral distribution and the estimation of the spectral reflectance may be set in advance, and the processing may be terminated when the number of repetitions reaches this set number, or the processing may be terminated when the difference between the obtained numerical value and the numerical value of the immediately preceding calculation result reaches a predetermined value.

Furthermore, when estimating the spectral reflectance, the spectral distribution of the light source may be estimated using either a viewpoint image in which a reference object (e.g., a color chart) with known spectral reflectance is captured in the same image as the target object, or a reference image in which the reference object is captured under the light source in which the target object is captured.
In this case, when using viewpoint images in which the reference object and the target object are captured as the same image, the spectral distribution estimation unit 115 selects viewpoint images in which the reference object is captured from at least two of the spectral sensitivity groups, and generates a group of viewpoints for light source estimation.

Then, the spectral distribution estimation unit 115 calculates the spectral distribution of the light source that captured the multi-viewpoint image using the spectral sensitivities corresponding to the viewpoint images in the group of viewpoints for light source estimation, the known spectral reflectances of specific parts of the reference object, and the pixel values at positions corresponding to the above-mentioned specific parts of the viewpoint images in the group of viewpoints for light source estimation, according to equations (3) and (4).
Furthermore, in the present embodiment, the first embodiment has been described by providing the spectral distribution estimation unit 115 that estimates the spectral distribution of the light source and the spectral distribution basis vector storage unit 116, but these may be provided in each of the second to fourth embodiments.

As described above, according to this embodiment, by repeating the process of estimating the three-dimensional shape and spectral reflectance of the target object and the process of generating a grayscale image from the viewpoint image corresponding to the spectral reflectance of the target object, the spectral distribution viewpoint image of the light source, and the spectral sensitivity, it is possible to improve the accuracy of estimating the three-dimensional shape of the target object as well as the accuracy of estimating the spectral reflectance.

Furthermore, the first to fifth embodiments described above may be configured to include a three-dimensional shape restoration unit (not shown) that reads each of the multi-viewpoint images of the target object stored in the multi-viewpoint image storage unit 107, estimates the three-dimensional shape of the target object, and generates a three-dimensional shape model. The three-dimensional shape restoration unit writes data of the three-dimensional shape model of the target object into the three-dimensional shape model storage unit 108 for storage.
Here, regarding the method of generating a 3D shape model, for example, a depth map may be generated from a viewpoint image, and a 3D shape model may be generated from this depth map as a 3D point cloud, and then generated as a 3D mesh model (e.g., a triangular mesh model) represented by a polygonal mesh (see, for example, Shuji Sakai, Takashi Watanabe, Tomohito Masuda, Koichi Ito, and Takafumi Aoki: Proposal of a High-Precision Multi-View 3D Restoration Algorithm Based on Phase-Only Correlation Method, 20th Image Recognition and Understanding Symposium (2017)).
Furthermore, when reproducing the three-dimensional shape as a three-dimensional point cloud or mesh, the three-dimensional shape restoration unit also estimates the camera parameters of the image capturing device when each of the viewpoint images was captured.

Note that a program for implementing the functions of each of the spectral texture image generating systems of the present invention shown in Figs. 1, 16, 20, and 21 may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to generate a spectral texture image of each of the wavelength bands corresponding to a hyperspectral image of a three-dimensional shape model of a target object. Note that the term "computer system" here includes hardware such as an OS (Operating System) and peripheral devices. The term "computer system" also includes a WWW (World Wide Web) system equipped with a homepage providing environment (or display environment). The term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc - Read Only Memory), as well as storage devices such as hard disks built into a computer system. Furthermore, "computer-readable recording medium" also includes things that hold a program for a certain period of time, such as volatile memory (RAM (Random Access Memory)) inside a computer system that serves as a server or client when a program is transmitted over a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

The embodiment of the present invention has been described above in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present invention are also included.

Reference Signs List 100, 100A, 100B...Spectral texture image generation system 101...Data input/output unit 102...Multi-viewpoint image classification unit 103...Virtual multi-viewpoint image generation unit 104...Spectral image generation unit 105, 112...Spectral texture image generation unit 106...Visualization texture image generation unit 107...Multi-viewpoint image storage unit 108...Three-dimensional shape model storage unit 109...Spectral image storage unit 110...Spectral texture image storage unit 111...Texture image generation unit 113...Texture image storage unit 114...Texture image reconstruction unit 115...Spectral distribution estimation unit 116...Spectral distribution basis vector storage unit

Claims (6)

a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object as a texture image of a three-dimensional shape model of the target object from multi-viewpoint images,
a multi-viewpoint image classifying unit that classifies each of the multi-viewpoint images obtained by capturing the object from a plurality of different viewpoints and using an imaging device with a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a virtual multi-viewpoint image generating unit that generates a virtual multi-viewpoint image by using the multi-viewpoint image at each of the viewpoints as a reference image and converting other multi-viewpoint images that are in the vicinity of the viewpoint of the reference image and belong to different spectral sensitivity groups into images captured from the viewpoint of the reference image;
a texture image generating unit that generates a texture image corresponding to the spectral sensitivity of the spectral sensitivity group from each of the reference image and the virtual multi-viewpoint image in the spectral sensitivity group;
and a spectral texture image generation unit that estimates a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities, and generates the spectral texture image of the three-dimensional shape model. The texture image generating unit
2. The spectral texture image generation system according to claim 1, wherein, when generating the texture image for any of the spectral sensitivities, a lookup table is generated in which texture coordinates indicating the positions of each vertex of a texture region in a two-dimensional coordinate system of the texture image correspond to vertex coordinates indicating the positions of vertices of polygons of a three - dimensional shape model to which each of the texture regions is attached, and by referring to the lookup table, texture images for other spectral sensitivities having a similar correspondence between the vertex coordinates of the polygons and the texture coordinates are generated. A texture image reconstruction unit that reconstructs the texture image,
The texture image generating unit
generating texture images corresponding to the respective spectral sensitivities, and when generating the respective texture images, generating respective lookup tables in which texture coordinates indicating the positions of the vertices of the texture regions in the respective texture images correspond to vertex coordinates indicating the positions of the vertices of the polygons of the three-dimensional shape model to which the respective texture images are applied;
2. The spectral texture image generating system according to claim 1, wherein the texture image reconstructing unit sets one of the texture images as a reference texture image, and reconstructs the correspondence between the vertex coordinates of the polygons and the texture coordinates in the other texture images to match the correspondence between the vertex coordinates of the polygons and the texture coordinates in a lookup table of the reference texture image. The texture image generating unit
generating texture images corresponding to the respective spectral sensitivities, and when generating the respective texture images, generating respective lookup tables in which texture coordinates indicating vertices of texture regions in the respective texture images correspond to vertex coordinates indicating positions of vertices of polygons of a three-dimensional shape model to which the respective texture images are applied;
A spectral texture image generating unit
2. The spectral texture image generating system according to claim 1, further comprising: referencing each of the lookup tables; associating pixels in each texture region of the texture image with the vertex coordinates in each of the lookup tables; estimating a spectral reflectance spectrum based on each of the associated pixels; and generating the spectral texture image. A method for generating spectral texture images, comprising: generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images as a texture image of a three-dimensional shape model of the target object;
a multi-viewpoint image classification step in which a multi-viewpoint image classification unit classifies each of the multi-viewpoint images obtained by capturing the target object using a plurality of different viewpoints and imaging devices having a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a virtual multi-viewpoint image generating process in which a virtual multi-viewpoint image generating unit uses the multi-viewpoint images at each of the viewpoints as a reference image, and converts other multi-viewpoint images that are in the vicinity of the viewpoint of the reference image and belong to different spectral sensitivity groups into images captured from the viewpoint of the reference image to generate a virtual multi-viewpoint image;
a texture image generating step of generating texture images corresponding to the spectral sensitivities of the spectral sensitivity groups from the reference images and the virtual multi-viewpoint images in the spectral sensitivity groups by a texture image generating unit;
a spectral texture image generation step of: a spectral texture image generation unit estimating a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities, and generating the spectral texture image of the three-dimensional shape model. a program for causing a computer to function as a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object as a texture image of a three-dimensional shape model of the target object from multi-viewpoint images,
The computer,
a multi-viewpoint image classification means for classifying each of the multi-viewpoint images obtained by capturing the object from a plurality of different viewpoints and using an imaging device with a plurality of different spectral sensitivities into a group having the same spectral sensitivity;
a virtual multi-viewpoint image generating means for generating a virtual multi-viewpoint image by using the multi-viewpoint image at each of the viewpoints as a reference image and converting other multi-viewpoint images that are in the vicinity of the viewpoint of the reference image and belong to different spectral sensitivity groups into images captured from the viewpoint of the reference image;
a texture image generating means for generating a texture image corresponding to the spectral sensitivity of the spectral sensitivity group from each of the reference image and the virtual multi-viewpoint image in the spectral sensitivity group;
a program for causing the program to function as a spectral texture image generating means for estimating a spectral reflectance spectrum of four or more bands of the target object from each of the texture images corresponding to each of the spectral sensitivities, and generating the spectral texture image of the three-dimensional shape model.

Images