JP7779046B2 - Spectral texture image generation system, spectral texture image generation method and program - Google Patents

Description

The present invention relates to a spectral texture image generation system, a spectral texture image generation method, and a program for generating texture images for texture mapping onto three-dimensional shape models.

3D shape reconstruction technology, which generates a 3D shape model of a target object based on multi-view images, is attracting attention not only from the computer vision research community but also from a wide range of fields, such as digital archiving of cultural assets and the entertainment industry.
The three-dimensional shape of the target object is reconstructed based on the differences in how the silhouette and surface texture of the target object appear in each of the captured images in the multi-viewpoint images (see, for example, Patent Documents 1 and 2).

Generally, when restoring the three-dimensional shape of a target object from such multi-viewpoint images, a texture image for the restored three-dimensional shape model is generated using the RGB values (pixel values for each channel of Red, Green, and Blue) of corresponding pixels in the multi-viewpoint images, or the average RGB values of each of the corresponding multiple viewpoint images in the multi-viewpoint image.

However, the RGB values of common imaging devices (e.g., digital cameras) available on the market are affected by the spectral distribution of the light source in the imaging environment when capturing an image of the target object and the spectral sensitivity characteristics of the imaging device used for capturing the image.
Therefore, when generating a texture image of a three-dimensional shape model using viewpoint images of the same target object captured under different light sources or with different imaging devices, the spectral reflectance of the target object cannot be obtained, even though the target object is the same, and the texture image of the three-dimensional shape model will be generated in different colors depending on the light source or digital camera.

On the other hand, there is a method that uses a hyperspectral camera to acquire color information of the target object for three-dimensional shape reconstruction without being affected by the spectral distribution of the light source in the imaging environment between viewpoint images in a multi-viewpoint image or differences in spectral sensitivity during imaging.
A hyperspectral camera can capture a hyperspectral image, which is an image that separates wavelengths into tens to hundreds of bands and stores spectral information indicating the intensity of each band for each pixel.
By using the above-mentioned hyperspectral camera to acquire the spectral reflectance (reflectance of light at each wavelength) of a target object, it is possible to express colors that are not dependent on the spectral distribution of the light source in the imaging environment between viewpoint images of a multi-view image or differences in the spectral sensitivity of the digital camera used for imaging.

Japanese Patent Application Laid-Open No. 2018-181047 Patent No. 5068732

However, a problem with hyperspectral cameras is that they are expensive compared to general digital cameras that capture RGB images.
Furthermore, a hyperspectral camera takes several tens of seconds (or even more than one minute in some cases) to capture one hyperspectral image, which is much longer than a digital camera, which can capture one image in less than one second.
For this reason, hyperspectral cameras take many images to generate a texture image, and therefore take longer to capture an object than digital cameras. For example, while a digital camera can complete the process in a few minutes to capture 100 or more multi-viewpoint images of an object, a hyperspectral camera requires several hours of work.

Furthermore, hyperspectral cameras have the problem that the resolution of captured images is lower than that of general digital cameras that capture RGB images. That is, when generating texture images of a 3D shape model from multi-viewpoint images captured by a hyperspectral camera, the resolution of the viewpoint images in the multi-viewpoint images significantly affects the resolution of the texture image.
For this reason, a hyperspectral camera with low resolution cannot generate a texture image of a three-dimensional shape model of a target object with the same resolution as when it is generated from an image captured by a digital camera.

For the reasons mentioned above, it is difficult to use hyperspectral images captured by a hyperspectral camera as multi-viewpoint images for generating texture images of a reconstructed 3D shape model of a target object. Therefore, there is a demand for technology that can easily acquire the shape and spectral reflectance of a target object using an imaging device that is less expensive than a hyperspectral camera and can capture many multi-viewpoint images in a short period of time.

The present invention was made in light of these circumstances, and provides a spectral texture image generation system, a spectral texture image generation method, and a program that can easily generate texture images that reflect the spectral reflectance of a target object using images captured by an imaging device that is less expensive than the expensive hyperspectral cameras available on the market for general users.

In order to solve the above-mentioned problems, the spectral texture image generation system of the present invention is a spectral texture image generation system that generates, from multi-viewpoint images, spectral texture images corresponding to the spectral reflectance spectrum of four or more bands of the spectral reflectance of a target object as texture images of a three-dimensional shape model of the target object. It is characterized by comprising: a multi-viewpoint image classification unit that classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; a correspondence coordinate information generation unit that generates, for each of the spectral sensitivity groups, correspondence coordinate information indicating the correspondence between first coordinates indicating the coordinates of a first pixel that is a pixel constituting the multi-viewpoint image and second coordinates indicating the coordinates of a second pixel that is a pixel constituting the texture image; a texture image generation unit that generates the texture image from the multi-viewpoint images using the correspondence coordinate information, for each of the spectral sensitivity groups; and a spectral texture image generation unit that references the correspondence coordinate information to estimate the spectral reflectance spectrum of the target object from the texture image and generate spectral texture images corresponding to each of the spectral reflectance spectra.

The spectral texture image generation system of the present invention is characterized in that the corresponding coordinate information generation unit generates the corresponding coordinate information by using the same coordinate system for the second coordinates of each of the texture images in each of the spectral sensitivity groups, and the spectral texture image generation unit generates each of the spectral texture images from each of the texture images by referring to the corresponding coordinate information.

The spectral texture image generation system of the present invention is characterized in that the corresponding coordinate information generation unit generates the corresponding coordinate information using the second coordinates of each of the texture images in each of the spectral sensitivity groups as different coordinate systems, and the spectral texture image generation unit references the corresponding coordinate information and generates each of the spectral texture images from each of the texture images using the third coordinate of a third pixel, which is a coordinate point in the three-dimensional shape model, as a reference.

The spectral texture image generation system of the present invention is characterized in that the corresponding coordinate information generation unit generates corresponding coordinate information for each of the spectral sensitivity groups by treating the second coordinates of each of the texture images in each of the spectral sensitivity groups as different coordinate systems, and generates integrated corresponding coordinate information that integrates the corresponding coordinate information for each of the spectral sensitivity groups, the texture image generation unit generates each of the texture images for each of the spectral sensitivity groups from each of the multi-viewpoint images by referencing the corresponding coordinate information, and reconstructs the texture images by making the coordinate systems of each of the texture images identical using the integrated corresponding coordinate information, and the spectral texture image generation unit generates each of the spectral texture images from each of the texture images.

The spectral texture image generation system of the present invention is a system for generating, from multi-viewpoint images, spectral texture images corresponding to the spectral reflectance spectrum of four or more bands of the spectral reflectance of a target object as texture images of a three-dimensional shape model of the target object. The system is characterized by comprising: a multi-viewpoint image classification unit that classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; a correspondence coordinate information generation unit that generates, for each of the spectral sensitivity groups, correspondence coordinate information that indicates a correspondence relationship between first coordinates indicating the coordinates of a first pixel, which is a pixel that constitutes the multi-viewpoint image, and second coordinates indicating the coordinates of a second pixel, which is a pixel that constitutes the spectral texture image; and a spectral texture image generation unit that references the correspondence coordinate information to estimate the spectral reflectance spectrum of the target object from the multi-viewpoint images , and generates spectral texture images corresponding to each of the spectral reflectance spectra.

The spectral texture image generation system of the present invention is characterized by further comprising a texture image correction unit that detects the positional deviation of the pixels between the texture images of each of the spectral sensitivity groups and corrects the positional deviation of the texture images.

The spectral texture image generation system of the present invention is characterized by further comprising a texture image correction unit that detects the positional deviation of the pixels between the texture images of each of the spectral sensitivity groups and corrects the positional deviation in the corresponding coordinate information.

The spectral texture image generation system of the present invention is characterized by further comprising a texture image correction unit that detects the positional shift of the pixels between the multi-viewpoint images of each of the spectral sensitivity groups and corrects the positional shift in the corresponding coordinate information.

The spectral texture image generation system of the present invention is characterized in that the correspondence coordinate information generation unit generates the correspondence coordinate information from second correspondence coordinate information indicating the correspondence between the second coordinates and third coordinates, which are the coordinates of pixels in the three-dimensional shape model, and third correspondence coordinate information indicating the correspondence between the third coordinates and first coordinates of the multi-viewpoint image for each spectral sensitivity group.

The spectral texture image generation system of the present invention is characterized in that the correspondence coordinate information generation unit generates the correspondence coordinate information from third correspondence coordinate information, which is generated for each spectral sensitivity group and indicates the correspondence between third coordinates, which are the coordinates of pixels in the three-dimensional shape model, and the first coordinates, which are the coordinates of pixels in the multi-view image.

The spectral texture image generation method of the present invention is a spectral texture image generation method for generating, from multi-viewpoint images, a spectral texture image corresponding to the spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object as a texture image of a three-dimensional shape model of the target object. The method includes a multi-viewpoint image classification process in which a multi-viewpoint image classifier classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; and a corresponding coordinate information generation unit generates, corresponding to each of the spectral sensitivity groups, first coordinates indicating the coordinates of a first pixel that constitutes the multi-viewpoint image; and a corresponding coordinate information generation unit generating a corresponding coordinate information for each of the spectral sensitivity groups. The method includes a corresponding coordinate information generation process that generates corresponding coordinate information, which is information indicating a correspondence relationship between the first coordinate and a second coordinate that indicates the coordinate of a second pixel that is a pixel constituting the texture image; a texture image generation process in which a texture image generation unit generates the texture image from the multi-viewpoint image using the corresponding coordinate information, corresponding to each of the spectral sensitivity groups; and a spectral texture image generation process in which a spectral texture image generation unit references the corresponding coordinate information, estimates a spectral reflectance spectrum of four or more bands of the target object from the texture image, and generates spectral texture images corresponding to each of the spectral reflectance spectra.

The spectral texture image generation method of the present invention is a spectral texture image generation method for generating, from multi-viewpoint images, spectral texture images corresponding to four or more bands of spectral reflectance of a target object as texture images of a three-dimensional shape model of the target object, and is characterized by including: a multi-viewpoint image classification step in which a multi-viewpoint image classifying unit classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; a correspondence coordinate information generation step in which a correspondence coordinate information generation unit generates, for each of the spectral sensitivity groups, correspondence coordinate information that indicates a correspondence relationship between first coordinates indicating the coordinates of a first pixel, which is a pixel that constitutes the multi-viewpoint image, and second coordinates indicating the coordinates of a second pixel, which is a pixel that constitutes the spectral texture image; and a spectral texture image generation step in which the spectral texture image generation unit references the correspondence coordinate information and estimates the four or more bands of the spectral reflectance spectrum of the target object from the multi-viewpoint images, and generates spectral texture images corresponding to each of the spectral reflectance spectra.

The program of the present invention is a program for causing a computer to function as a spectral texture image generation system that generates, from multi-viewpoint images, spectral texture images corresponding to the spectral reflectance spectrum of four or more bands of the target object's spectral reflectance as texture images of a three-dimensional shape model of the target object. The program causes the computer to function as: multi-viewpoint image classification means that classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; correspondence coordinate information generation means that generates, for each of the spectral sensitivity groups, correspondence coordinate information indicating the correspondence between first coordinates indicating the coordinates of a first pixel that constitutes the multi-viewpoint image and second coordinates indicating the coordinates of a second pixel that constitutes the texture image; texture image generation means that generates the texture image from the multi-viewpoint images using the correspondence coordinate information, for each of the spectral sensitivity groups; and spectral texture image generation means that references the correspondence coordinate information to estimate the spectral reflectance spectrum of the target object from the texture image and generate spectral texture images corresponding to each of the spectral reflectance spectra.

A program of the present invention is a program for causing a computer to function as a spectral texture image generation system that generates, from multi-viewpoint images, spectral texture images corresponding to four or more bands of spectral reflectance spectra of a target object as texture images of a three-dimensional shape model of the target object. The program causes the computer to function as: multi-viewpoint image classification means that classifies each of the multi-viewpoint images, obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into spectral sensitivity groups having the same spectral sensitivity; correspondence coordinate information generation means that generates, for each of the spectral sensitivity groups, correspondence coordinate information that indicates a correspondence relationship between first coordinates that indicate the coordinates of a first pixel that is a pixel constituting the multi -viewpoint image and second coordinates that indicate the coordinates of a second pixel that is a pixel constituting the spectral texture image; and spectral texture image generation means that references the correspondence coordinate information to estimate the four or more bands of the spectral reflectance spectrum of the target object from the multi-viewpoint images and generate spectral texture images corresponding to each of the spectral reflectance spectra.

As described above, the present invention provides a spectral texture image generation system, a spectral texture image generation method, and a program that can easily generate a texture image that reflects the spectral reflectance of a target object using images captured by an imaging device that is less expensive than the expensive hyperspectral cameras available on the market for general users.

1 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a first embodiment of the present invention. 10 is a diagram showing an example of the configuration of a multi-viewpoint image information table written and stored in a multi-viewpoint image storage unit 109. FIG. 10 is a diagram showing an example of the configuration of a spectral sensitivity group data table in a multi-viewpoint image storage unit 109. FIG. 1A and 1B are conceptual diagrams illustrating generation of a spectral texture image by the spectral texture image generation system of the first embodiment. 10 is a diagram showing an example of the configuration of a corresponding coordinate information table stored in a corresponding coordinate information storage unit 111. FIG. FIG. 10 is a conceptual diagram illustrating generation of a development view of a polygon mesh in a three-dimensional shape model. 10 is a diagram showing an example of the configuration of a correspondence coordinate lookup table showing the correspondence between polygon mesh vertex coordinate values in a three-dimensional shape model and area vertex coordinate values in a development view. FIG. FIG. 10 is a diagram showing an example of the configuration of a second correspondence coordinate information table showing the correspondence between the coordinate values (third coordinates) of pixels in each polygon mesh in a three-dimensional shape model and the coordinate values (second coordinates) of pixels in a development view. FIG. 10 is a conceptual diagram showing how spectral reflectance is expressed by combining spectral reflectance basis vectors. FIG. 10 is a diagram showing an example of the configuration of a spectral texture image table in the spectral texture image storage unit 113. FIG. 10 is a conceptual diagram showing an example of rendering a spectral texture image processed under the conditions of pattern #1 onto a three-dimensional shape model. FIG. 10 is a conceptual diagram showing an example of rendering a spectral texture image processed under the conditions of pattern #2 onto a three-dimensional shape model. FIG. 10 is a conceptual diagram showing an example of rendering a spectral texture image processed under the conditions of pattern #3 onto a three-dimensional shape model. 10 is a flowchart illustrating an example of the operation of a process for generating spectral texture images for a three-dimensional shape model from multi-viewpoint images according to the first embodiment. FIG. 11 is a diagram showing an example of the configuration of a fourth corresponding coordinate information table written and stored in the corresponding coordinate information storage unit 111 in the second embodiment. FIG. 10 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a fourth embodiment of the present invention. FIG. 10 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a fifth embodiment of the present invention.

First Embodiment
A first embodiment of the present invention will be described below with reference to the drawings.
1 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a first embodiment of the present invention. In this embodiment, captured images of a target object to be reconstructed as a three-dimensional shape model are supplied as viewpoint image data from an external device (not shown).

Each of these viewpoint images is a multi-view image captured from a different viewpoint. Each viewpoint image contains image data captured from the same and different viewpoints, each captured by multiple imaging devices with different spectral sensitivities. The imaging device used in this embodiment is not a hyperspectral camera, but, for example, a digital camera used by a general user. In this embodiment, a multi-view image is described as a collection of multiple viewpoint images (images captured from different viewpoints and by imaging devices with different spectral sensitivities).

In FIG. 1, the spectral texture image generation system 100 includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a corresponding coordinate information generation unit 104, a texture image generation unit 105, a texture image correction unit 106, a spectral texture image generation unit 107, a visualization texture image generation unit 108, a multi-viewpoint image storage unit 109, a three-dimensional shape model storage unit 110, a corresponding coordinate information storage unit 111, a texture image storage unit 112, and a spectral texture image storage unit 113.

The data input/output unit 101 writes and stores, in the multi-viewpoint image storage unit 109, each of the data of viewpoint images obtained by capturing an object from each viewpoint, which are supplied from an external device.
Furthermore, the data input/output unit 101 writes and stores data of the three-dimensional shape model of the target object, which is reconstructed using the multi-viewpoint images, in the three-dimensional shape model storage unit 110. The three-dimensional shape model is a surface model formed of a polygon mesh reconstructed from the three-dimensional point cloud.

In addition, the data input/output unit 101 writes and stores in the multi-viewpoint image storage unit 109 the viewpoint information, which indicates the viewpoint coordinates and viewpoint direction at which the multi-viewpoint image was captured, the camera parameters of the imaging device that captured the multi-viewpoint image, and the spectral distribution of the light source (light source spectral distribution) in the environment in which the multi-viewpoint image was captured, which are supplied from the external device that restored the three-dimensional shape model.
Furthermore, when the data input/output unit 101 is accessed by an external device and is requested to transmit spectral texture images, it reads each of the spectral texture images (described later) from the spectral texture image storage unit 113 and outputs them to the external device.

Fig. 2 is a diagram showing an example of the configuration of a multi-viewpoint image information table written and stored in the multi-viewpoint image storage unit 109. The multi-viewpoint image information table shown in Fig. 2 has columns for each record, corresponding to the multi-viewpoint image identification information, for the image capture device type, viewpoint information, camera parameters, image data index, spectral sensitivity data index, and spectral distribution data index.
The multi-viewpoint image identification information is identification information that uniquely identifies each individual multi-viewpoint image (e.g., a number indicating the order in which the images were supplied). The viewpoint position is, for example, information indicating an external parameter of the imaging device, and indicating the center coordinates of the lens of the imaging device (hereinafter referred to as the viewpoint position) in the three-dimensional spatial coordinate system (world coordinate system) of the three-dimensional shape model, and the optical axis direction of the lens (optical axis vector, hereinafter referred to as the optical axis vector).

The camera parameters indicate, for example, the internal parameters of the imaging device, such as the focal length of the imaging device when the viewpoint image was captured, the image center, the image resolution (number of pixels), and the distortion coefficient. The image data index is an address indicating an area in the multi-viewpoint image storage unit 109 where data for the multi-viewpoint image has been written. The spectral sensitivity data index is an address indicating an area where data for the spectral sensitivity of the imaging device has been written. The spectral distribution data index is an address indicating an area in the multi-viewpoint image storage unit 109 where data for the spectral distribution of the light source (light source spectral distribution) in the environment where the multi-viewpoint image of the target object, supplied from an external device, was captured has been written.

Returning to Figure 1, the multi-viewpoint image classification unit 102 classifies each of the multi-viewpoint images stored in the multi-viewpoint image storage unit 109 into spectral sensitivity groups each having a different spectral sensitivity. The multi-viewpoint image classification unit 102 then writes and stores each of the multi-viewpoint images classified into the corresponding spectral sensitivity group in a spectral sensitivity group data table for each spectral sensitivity group in the multi-viewpoint image storage unit 109.

Figure 3 shows an example of the configuration of a spectral sensitivity group data table in the multi-viewpoint image storage unit 109. A spectral sensitivity group data table is provided for each spectral sensitivity group, and shows the spectral sensitivity data of the spectral sensitivity group. The spectral sensitivity group data table also has columns for multi-viewpoint image identification information, image data index, viewpoint information, camera parameters, and spectral distribution data index for each record.

For example, if images are captured using three image capture devices with different spectral sensitivities, the viewpoint images will each have three different spectral sensitivities and will therefore be classified into three spectral sensitivity groups. Each of the three spectral sensitivity groups may contain the same number of multi-viewpoint images, or a different number of multi-viewpoint images. Furthermore, a spectral sensitivity data index is added to the spectral sensitivity group data table. The spectral sensitivity data index is an address that indicates the area where the spectral sensitivity data of the image capture devices is written.

The viewpoint image identification information is identification information that identifies each multi-viewpoint image (for example, a number indicating the order in which they were supplied). The image data index is an address that indicates the area in the multi-viewpoint image storage unit 109 where the data for the multi-viewpoint image has been written. The viewpoint position indicates, for example, the external parameters of the imaging device, and is information that indicates the viewpoint position and optical axis vector in the world coordinate system. The camera parameters indicate, for example, the internal parameters of the imaging device, and include the focal length of the imaging device when the viewpoint image was captured, the image center, the image resolution, and the distortion aberration coefficient. The spectral distribution data index is an address that indicates the area in the multi-viewpoint image storage unit 109 where the data for the light source spectral distribution of the environment in which the multi-viewpoint image of the target object, supplied from an external device, was captured has been written.

4 is a conceptual diagram illustrating the generation of a spectral texture image by the spectral texture image generation system of the first embodiment. As the imaging devices, for example, three imaging devices A1, A2, and A3 with different spectral sensitivities are used.
4A shows each of the multi-viewpoint images (such as multi-viewpoint images 601 to 607) of the target object 500 captured at different viewpoint positions and optical axis vectors. Here, the multi-viewpoint images 601, 604, and 607 were captured by the image capture device A3. The multi-viewpoint images 602 and 605 were captured by the image capture device A1. The multi-viewpoint images 603 and 606 were captured by the image capture device A2.

4B shows the process of generating a texture image from each of the multi-viewpoint images for each viewpoint information. The texture image is generated in three bands (color component R, color component G, color component B) for each spectral sensitivity group.
Although not shown in the figure, when generating each texture image for each spectral sensitivity group, corresponding coordinate information is generated that indicates the correspondence between the coordinates (first coordinates) of pixels in each multi-viewpoint image (pixels that make up the multi-viewpoint image) that are the target of generating this texture image and the coordinates of pixels in the texture image (i.e., second coordinates, which are coordinates in the development view) (described in detail later).

Returning to FIG. 1, the corresponding coordinate information generating unit 104 generates corresponding coordinate information used when generating a texture image from a three-dimensional shape model and a multi-viewpoint image.
At this time, the corresponding coordinate information generation unit 104 generates a corresponding coordinate information table that shows corresponding coordinate information indicating the correspondence between a pixel (second pixel) in the second coordinate system (UV coordinate system), which is the coordinate system of the texture image, and a pixel (first pixel) in the first coordinate system, which is the coordinate system of the multi-viewpoint image.

FIG. 5 is a diagram showing an example of the structure of a corresponding coordinate information table stored in the corresponding coordinate information storage unit 111. As shown in FIG.
The corresponding coordinate information table indicates, on a record-by-record basis, the correspondence between the coordinate values in the second coordinate system (UV coordinate system) of the pixels that make up the development view and the coordinates in the first coordinate system of the pixels that make up the multi-view image. The area identification information is identification information that individually identifies each area in the texture image. The second coordinates are the coordinate values of the pixels of the texture image in the second coordinate system. The multi-view image identification information is identification information that identifies each of the multi-view images. The first coordinates are the coordinate values of the pixels of the multi-view image in the first coordinate system.

Returning to FIG. 1 , the correspondence coordinate information generation unit 104 generates, for each spectral sensitivity group, correspondence coordinate information to be used when generating development diagrams (FIG. 6(b) to be described later) of each of the color components R, G, and B, using a multi-viewpoint image of each piece of viewpoint information and a three-dimensional shape model (represented by a polygon mesh, as shown in FIG. 6(a) to be described later).
Here, the corresponding coordinate information generation unit 104 reads out data of the polygon mesh that forms the three-dimensional shape model from the three-dimensional shape model storage unit 110, and generates an unfolded view of the polygon mesh (for example, the unfolded view of Figure 6(b) obtained by UV unfolding the polygon mesh of Figure 6(a)).

6A and 6B are conceptual diagrams showing the generation of a development of a polygon mesh in a three-dimensional shape model, and Fig. 6A shows a three-dimensional shape model formed of polygon meshes.
FIG. 6B shows a development diagram in which each of the polygon meshes is developed in a two-dimensional coordinate system (a UV development diagram in which each of the polygon meshes is developed in a UV coordinate system, as an example).
For example, polygon meshes 651 and 652 in Fig. 6A correspond to regions 801 and 802 in Fig. 6B, respectively. That is, polygon mesh 651 is developed in region 801 of the development view, and polygon mesh 652 is developed in region 802 of the development view.

Here, each of the vertices (hereinafter referred to as polygon mesh vertices) 701 to 704 of the polygon mesh 651 of the three-dimensional shape model in the world coordinate system corresponds to each of the vertices (hereinafter referred to as area vertices) 951, 952, 953, and 954 of the area 801 of the development drawing in the two-dimensional coordinate system.
Similarly, polygon mesh vertices 703 to 706 of the polygon mesh 652 of the three-dimensional shape model in the world coordinate system correspond to area vertices 953, 954, 955, and 956 of the area 801 of the development in the two-dimensional coordinate system, respectively.

The corresponding coordinate information generating unit 104 generates a corresponding coordinate lookup table indicating the correspondence between the polygon mesh vertex coordinate values in the above-mentioned three-dimensional shape model and the area vertex coordinate values (coordinates of the area vertices) in the development.
FIG. 7 is a diagram showing an example of the configuration of a correspondence coordinate lookup table showing the correspondence between the coordinate values of polygon mesh vertices in a three-dimensional shape model and the coordinate values of area vertices in a development view.
The corresponding coordinate lookup table indicates the correspondence between polygon mesh vertex coordinate values and region vertex coordinate values on a record-by-record basis, corresponding to region identification information. Region identification information is identification information that individually identifies each region in the development. Polygon mesh vertex coordinate values are the coordinate values of each vertex of the polygon mesh in the world coordinate system. Region vertex coordinate values are the coordinate values of each vertex of a region (texture region, described below) in the development.

For example, in Figure 7, if the texture region identification information P10001 indicates the polygon mesh 651 in Figure 6(a), the coordinate values of polygon mesh vertex 701 are polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), the coordinate values of polygon mesh vertex 702 are polygon mesh vertex coordinate values (x1_2, y1_2, z1_2), the coordinate values of polygon mesh vertex 703 are polygon mesh vertex coordinate values (x1_3, y1_3, z1_3), and the coordinate values of polygon mesh vertex 704 are polygon mesh vertex coordinate values (x1_4, y1_4, z1_4).

Also, in Figure 7, when showing area 801 in Figure 6 (b), the coordinate values of area vertex 951 are area vertex coordinate values (U1_1, V1_1), the coordinate values of area vertex 952 are area vertex coordinate values (U1_2, V1_2), the coordinate values of area vertex 953 are area vertex coordinate values (U1_3, V1_3), and the coordinate values of area vertex 954 are area vertex coordinate values (U1_4, V1_4).
According to this corresponding coordinate lookup table, it can be seen that the polygon mesh formed by the polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), (x1_2, y1_2, z1_2), (x1_3, y1_3, z1_3) and (x1_4, y1_4, z1_4) is deployed in an area whose vertices are the area vertex coordinate values (U1_1, V1_1), (U1_2, V1_2), (U1_3, V1_3) and (U1_4, V1_4) in the development drawing. Furthermore, the corresponding coordinate lookup table indicates that each of the polygon mesh vertex coordinate values (x1_1, y1_1, z1_1), (x1_2, y1_2, z1_2), (x1_3, y1_3, z1_3) and (x1_4, y1_4, z1_4) corresponds to each of the area vertex coordinate values (U1_1, V1_1), (U1_2, V1_2), (U1_3, V1_3) and (U1_4, V1_4).

FIG. 8 is a diagram showing an example of the configuration of a second correspondence coordinate information table showing the correspondence between the coordinate values (third coordinates) of pixels in each polygon mesh in a three-dimensional shape model and the coordinate values (second coordinates) of pixels in a development drawing.
The second corresponding coordinate information table indicates, for each record, the correspondence between the coordinate values of pixels in a polygon mesh in a three-dimensional shape model (pixels at coordinate points constituting the mesh) and the coordinate values of pixels in a development drawing, corresponding to area identification information. The area identification information is identification information that individually identifies each area in the development drawing. The three-dimensional shape model coordinate values (third coordinates) are the coordinate values of each pixel (coordinate point) constituting the three-dimensional shape in the world coordinate system. The texture image coordinate values (second coordinates) are the coordinate values of each pixel in the texture image.
Here, the second corresponding coordinate information table shown in FIG. 8 includes the information of the corresponding coordinate lookup table of FIG.

Furthermore, the corresponding coordinate information generating unit 104 obtains the coordinate values of pixels in the multi-viewpoint image that correspond to each pixel of the three-dimensional shape model in the second corresponding coordinate information table.
At this time, the corresponding coordinate information generation unit 104 determines the multi-viewpoint image corresponding to the pixels (coordinate points) that make up the three-dimensional shape model and the coordinate values of the pixels in the multi-viewpoint image, in accordance with the viewpoint information (and camera parameters if necessary).

Then, the corresponding coordinate information generating unit 104 refers to the second corresponding coordinate information table and, using the pixels (coordinate points) of the three-dimensional shape model as a reference, associates each of the pixels in the obtained multi-view image with each of the pixels in the development drawing.
At this time, the corresponding coordinate information generation unit 104 preferentially matches pixels of the multi-view image corresponding to viewpoint information having an optical axis vector that is parallel or closest to parallel to the normal of each surface of the mesh in the three-dimensional shape model with pixels of the unfolded view.

As a result, the corresponding coordinate information generation unit 104 generates corresponding coordinate information that indicates the correspondence between the coordinates (first coordinates) of pixels in the multi-viewpoint image and the coordinates (second coordinates) of pixels that make up the unfolded view, using the pixels (coordinate points) of the three-dimensional shape model as a reference.
Here, each of the multi-viewpoint images for each spectral sensitivity group is used to generate second corresponding coordinate information used to generate a development corresponding to each spectral sensitivity group, so the origin of each development (UV coordinate) generated for each spectral sensitivity group is different, and the coordinate values of the same pixel in each development of each spectral sensitivity group are not common (identical) but are different coordinate values.

Therefore, the correspondence coordinate information generation unit 104 generates a second correspondence coordinate information table of a development drawing corresponding to one of the spectral sensitivity groups, and then generates second correspondence coordinate information tables of other spectral sensitivity groups based on the second correspondence coordinate information table.
That is, the corresponding coordinate information generation unit 104 references the corresponding coordinate information of one of the generated spectral sensitivity groups, and associates the relationship between the pixels (coordinate points) of the three-dimensional shape model and the coordinate points (coordinate values) of the pixels in the development view, thereby generating a second corresponding coordinate information table corresponding to another spectral sensitivity group.

As a result, the corresponding coordinate information generating unit 104 makes the coordinate systems of the developments in the second corresponding coordinate information table for each spectral sensitivity group the same (the origins are the same), and therefore the coordinate systems of the developments in the corresponding coordinate information can be made the same between spectral sensitivity groups (i.e., the coordinate values of the same pixel in each development between spectral sensitivity groups can be made the same).
Each texture image generated based on the above-described corresponding coordinate information is generated in the same coordinate system between spectral sensitivity groups.
The corresponding coordinate information generating unit 104 then writes and stores the corresponding coordinate information table generated for each spectral sensitivity group in the corresponding coordinate information storage unit 111 .

The texture image generation unit 105 refers to the corresponding coordinate information table for each spectral sensitivity group, and assigns the pixel values (for example, three bands, color component R, color component G, and color component B) of the pixel at the corresponding coordinate in the multi-view image to each coordinate of the pixel in the unfolded view.
That is, the texture image generating unit 105 generates a texture image of three bands for each spectral sensitivity group.
Then, the texture image generating unit 105 writes and stores each of the generated texture images in the texture image storage unit 112 for each spectral sensitivity group.

The texture image correcting unit 106 corrects the deviation of the coordinate values of pixels in each texture image generated by the texture image generating unit 105 .
When the mesh of a three-dimensional shape model is developed into a development drawing, linear interpolation is performed when developing three-dimensional coordinate points into two-dimensional coordinates of the development drawing.
However, when generating a three-dimensional shape model, the pixels of the multi-view image may be misaligned with the coordinates of the three-dimensional shape model between spectral sensitivity groups, resulting in a deviation in the pixel values actually assigned to the coordinates of the development drawing, and in pixel positional deviations occurring in each texture image.

For this reason, for example, in each texture image, the texture image correction unit 106 compares each of the pixels surrounded by a detection frame of a × b pixels (a and b are odd integer values of 3 or greater) with pixels within similar detection frames in texture images of other spectral sensitivities, and sequentially determines the similarity of patterns made up of the pixel values of each of the pixels within the detection frame, with the pixel in the center of the detection frame being the target pixel.
The texture image correction unit 106 then detects coordinates at which the pattern similarity is high within a detection frame that is centered around a target pixel in the texture image, and corrects each texture image by aligning each target pixel with the coordinates at which the similarity between texture images with different spectral sensitivities is highest.
In this way, the texture image correcting unit 106 corrects the positional deviation of each pixel between the texture images with different spectral sensitivities generated by the texture image generating unit 105 for each spectral sensitivity group.

The spectral texture image generation unit 107 generates spectral texture images for each of a plurality of wavelength bands (e.g., 31 bands, each 10 nm apart from a wavelength of 400 nm to a wavelength of 700 nm) from the texture image generated by the texture image generation unit 105 (or the texture image corrected by the texture image correction unit 106).
Here, the spectral texture image generating unit 107 refers to the corresponding coordinate information table for each spectral sensitivity group in the corresponding coordinate information storage unit 111, and acquires information about the multi-viewpoint image.

Then, the spectral texture image generation unit 107 sequentially reads out the spectral sensitivity data and spectral distribution data of each of the multi-viewpoint images in the group of multi-viewpoint images for generating spectral images, which corresponds to each pixel in the texture image, from the same-viewpoint multi-viewpoint image table.
As a result, the spectral texture image generation unit 107 estimates the spectral reflectance spectrum (i.e., spectral reflectance) in a specified wavelength band for each pixel of the development drawing, using each of the additional spectral sensitivity data and spectral distribution data and the pixel values of each band for each pixel of the texture image.

In this embodiment, the spectral texture image generating unit 107 reads a plurality of spectral reflectance basis vectors that have been written and stored in advance from the spectral texture image storage unit 113, for example.
The spectral texture image generating unit 107 then expresses the spectral reflectance of the pixels of the texture image of each spectral sensitivity group as a weighted sum of the plurality of spectral reflectance basis vectors that have been read.
The spectral texture image generating unit 107 estimates the spectral reflectance spectrum at each pixel of the texture image by estimating weighting coefficients that weight each spectral reflectance basis vector.

Here, the spectral reflectance basis vectors are derived in advance from a spectral reflectance data group, which is a collection of multiple spectral reflectance data, using principal component analysis as spectral reflectance basis vectors for expressing the spectral reflectance data group.
In this principal component analysis, spectral reflectance can be expressed by a combination of a smaller number of spectral reflectance basis vectors than the actual dimensions of the spectral reflectance. In other words, by using spectral reflectance basis vectors, the dimensions of the spectral reflectance to be estimated are compressed, reducing the calculation load.

That is, as shown in the following equation (1), the spectral texture image generation unit 107 expresses the spectral reflectance spectrum s as a weighted sum of multiple spectral reflectance basis vectors B = [ _b1 , _b2 , ...] ^T , and to perform this weighting, the spectral reflectance spectrum s is estimated by calculating the weighting coefficient r = [r1, r2, ...] ^T for each of the spectral reflectance basis vectors. Here, the spectral reflectance basis vectors are, for example, pre-stored as a set in the spectral texture image storage unit 113. Furthermore, the spectral texture image storage unit 113 may store a plurality of spectral reflectance basis vectors arranged in descending order of principal component scores in principal component analysis, and may be configured to read and use as many spectral reflectance basis vectors as necessary up to a predetermined order.

9 is a conceptual diagram showing how spectral reflectance is expressed by combining spectral reflectance basis vectors. As shown in Fig. 9, the spectral reflectance spectrum s is obtained by multiplying each of the spectral reflectance basis vectors _b1 to _bNr obtained by principal component analysis using the spectral reflectances of a given object by a weighting coefficient and combining the multiplication results.
That is, the spectral texture image generating unit 107 extracts the required number of spectral reflectance basis vectors b ₁ to b _Nr from the spectral texture image storage unit 113 .
The spectral texture image generation unit 107 then estimates the weighting coefficients _r1 to _rNr of the extracted spectral reflectance basis vectors _b1 to _bNr so as to obtain a spectral reflectance spectrum s that becomes the pixel value of each pixel in the multi-viewpoint image, and calculates the spectral reflectance spectrum s from the estimation result (described below).

1 , the spectral texture image generation unit 107 estimates each weighting coefficient r by solving the following equation (2) for the spectral reflectance spectrum expressed by the above equation (1). Then, the spectral texture image generation unit 107 estimates the spectral reflectance spectrum s according to equation (1) using the estimated weighting coefficients.
Here, the spectral reflectance spectrum si is expressed as a 31-dimensional vector (corresponding to 31 bands) obtained by sampling the wavelength range from 400 nm to 700 nm in 10 nm increments, for example.

In the above formula (2), p _m,k is the pixel value of the k-th channel of the m-th texture image. That is, in p _m,k , m is a number indicating the type of spectral sensitivity group of the texture image stored in the spectral texture image storage unit 113. Furthermore, k indicates the channel of the color component of a portion in the multi-viewpoint image. For example, if a portion is represented by pixel values of the color components R, G, and B, then channel (band) 1 is the color component R, channel 2 is the color component G, and channel 3 is the color component B. Furthermore, the spectral texture image generation unit 107 reads pixel values from each pixel of the texture image in the texture image storage unit 112.

In addition, in equation (2), the matrix C _m,k represents the spectral sensitivity of the k-th channel (wavelength band of color component) of the m-th texture image, and the element c _m,k (λ) in this C _m,k represents the spectral sensitivity of the wavelength λ in the k-th channel of the m-th texture image in the group of multi-viewpoint images used to generate spectral images (the spectral sensitivity of the imaging device that captured the texture image).
The spectral distribution l of the light source has already been written as measured spectral distribution data in the spectral texture image storage unit 113. The spectral texture image generation unit 107 reads out, from the same-viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 109, the spectral distribution of the light source in the environment in which each of the multi-viewpoint images used to generate the texture image in the group of multi-viewpoint images for generating spectral images was captured.

Here, the spectral texture image generation unit 107 may be configured to select and read out a set of spectral reflectance basis vectors from the spectral texture image storage unit 113 in accordance with the number of spectral sensitivity groups in the texture image generated from the multi-viewpoint images of the group of multi-viewpoint images for generating spectral images.
Furthermore, when solving equation (2), the upper limit of the number of spectral reflectance basis vectors (principal components) _Nr used to represent the spectral reflectance data (hereinafter referred to as the number of spectral reflectance basis vectors) is determined based on, for example, the number of spectral sensitivity groups capturing the multi-viewpoint image (the number of imaging devices, the number of spectral sensitivity adjustments using color change filters, etc.).

When texture images for q spectral sensitivity groups are used, the number is determined by the relational expression " _Nr (number of spectral reflectance basis vectors) = q (number of spectral sensitivity types, number of spectral sensitivity groups) x k (number of channels, hereinafter referred to as the number of channels)." In other words, if the number of spectral reflectance basis vectors _Nr indicating the spectral reflectance data is equal to or less than k times the number of channels of the number of spectral sensitivity groups q, then it is possible to solve expression (3).
As an example, if three imaging devices with different spectral sensitivities are used to capture a multi-view image and there are three channels, R, G, and B, the upper limit of the basis vectors is 9 (= 3 × 3).

However, in order to suppress the influence of noise when solving equation (3), it is desirable to operate the system such that the number of spectral reflectance basis vectors _Nr is less than k times the number of channels of the number of spectral sensitivity groups q, for example, Nr = q × k - 1. In this case, if three image capture devices with different spectral sensitivities are used and there are three channels, R, G, and B, the upper limit of the spectral reflectance basis vectors is 8 (= 3 × 3 - 1).

Furthermore, a group of multi-viewpoint images for generating spectral images may be obtained for each piece of viewpoint information, and the number of pixels in the texture image obtained from the multi-viewpoint images of the viewpoint information, i.e., the number q of spectral sensitivity groups, may differ. In this case, the set of spectral reflectance basis vectors (the number N _r of spectral reflectance basis vectors) that the spectral texture image generator 107 reads from the spectral texture image storage unit 113 differs for each pixel in the texture image. Each pixel in the spectral image is configured to correspond to a corresponding pixel in the texture image.

Through the above-described processing, the spectral texture image generation unit 107 estimates the spectral reflectance spectrum of each pixel of the spectral texture image from the pixel values of corresponding pixels between the texture images, corresponding to each pixel of the texture images in the same spectral sensitivity group, generates a spectral image (a spectral image with a predetermined number of wavelength bands) for each piece of viewpoint information, and writes and stores it in the spectral texture image table of the spectral texture image storage unit 113.
FIG. 10 is a diagram showing an example of the structure of a spectral texture image table in the spectral texture image storage unit 113.
The spectral texture image table has columns for spectral texture image identification information, band wavelength, and spectral texture image data index in each record.

The spectral texture image identification information is identification information that individually identifies each spectral texture image. The band wavelength indicates the wavelength of the band (channel) of the spectral texture image indicated by the spectral texture image identification information. The spectral texture image data index indicates the address of the area in the spectral texture image storage unit 113 where the image data of the spectral texture image is written.
The spectral texture image table of FIG. 10 corresponds to spectral texture images of each of 31 bands of wavelengths in 10 nm increments, for example, from 400 nm to 700 nm.

The visualization texture image generation unit 108 renders a visualized spectral texture image on the display unit according to predetermined conditions for the three-dimensional shape model read out from the three-dimensional shape model storage unit 110 .
The visualization texture image generation unit 108 converts the spectral reflectance of the spectral texture image (the spectral reflectance spectrum of each of all spectral images) into pixel values of wavelength bands that can be displayed on the display screen based on the following three predetermined conditions, and displays them on the display screen.

FIG. 11 is a conceptual diagram showing an example in which a spectral texture image processed under the conditions of pattern #1 is rendered onto a three-dimensional shape model.
Pattern #1 shows an image from a predetermined viewpoint and viewpoint direction (corresponding to the optical axis vector) of a three-dimensional shape model in which the spectral reflectance of each pixel in the spectral texture image is rendered using arbitrary spectral sensitivities corresponding to the three color components R, G, and B (red, green, blue) that can be displayed on a display screen and the spectral distribution of an arbitrary light source. Here, the arbitrary spectral sensitivities may be, for example, the spectral sensitivities of an imaging device or color matching functions. Examples of the color matching functions include the CIE (Commission Internationale de l'Eclairage) colorimetric observer matching functions. The spectral distribution of the light source may be, for example, the spectral distribution of standard illuminant D65, sunlight, LED (light emitting diode), fluorescent lamp, incandescent lamp, or the like. The data for the spectral sensitivities and spectral distributions are written and stored in a storage unit (not shown) in advance, and are read by the visualization texture image generation unit 108 as needed.

In processing this pattern #1, the visualization texture image generating unit 108 reads out the three-dimensional shape model 901 from the three-dimensional shape model storage unit 110.
Then, the visualization texture image generation unit 108 refers to the spectral texture image storage unit 113 and generates a texture image 902 in which the spectral reflectance based on the pixel values of each pixel of the spectral texture image is expressed in RGB values using an arbitrary spectral sensitivity and the spectral distribution of an arbitrary light source.
The visualization texture image generation unit 108 renders the texture image 902 onto the three-dimensional shape model 901 to generate a three-dimensional shape model 903, which is then displayed as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 12 is a conceptual diagram showing an example in which a spectral texture image processed under the conditions of pattern #2 is rendered onto a three-dimensional shape model.
Pattern #2 shows an image in which the gradation of three arbitrary wavelength bands is displayed as the gradation of each band of RGB values that can be displayed on the display screen.
12 , the 440 nm wavelength band is replaced with the color component B of the R, G, and B color components, and the pixel values (spectral reflectance spectrum) of the spectral texture image at a wavelength of 440 nm are defined as the pixel values of the color component B. Similarly, the 500 nm wavelength band is replaced with the color component G of the R, G, and B color components, and the pixel values (spectral reflectance spectrum) of the spectral texture image at a wavelength of 500 nm are defined as the pixel values of the color component G. Furthermore, the 660 nm wavelength band is replaced with the color component R of the R, G, and B color components, and the pixel values (spectral reflectance spectrum) of the spectral texture image at a wavelength of 660 nm are defined as the pixel values of the color component R.

In processing this pattern # 2 , the visualization texture image generation unit 108 reads out the three-dimensional shape model 913 from the three-dimensional shape model storage unit 110 .
Then, the visualization texture image generation unit 108 refers to the spectral texture image storage unit 113, sets the pixel values of each pixel in the spectral texture image of any three wavelength bands to the pixel values of the color components R, G, and B (graph 911), and generates a texture image 912 expressed by the pixel values of the color components R, G, and B.
The visualization texture image generation unit 108 renders the texture image 912 onto a three-dimensional shape model 913 to generate a three-dimensional shape model 914, which is then displayed as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 13 is a conceptual diagram showing an example in which a spectral texture image processed under the conditions of pattern #3 is rendered onto a three-dimensional shape model.
FIG. 13A illustrates a process of allocating pixel values in a spectral texture image as pattern #3 to pixel values of each of the color components R, G, and B that can be displayed on a display device.
13A, the range T of the spectral reflectance spectrum R(λi) of each pixel in any one spectral texture image is divided into three ranges, for example, a high value range H, a medium value range M, and a low value range L. λi indicates the wavelength of each band of the spectral texture image (for example, in the case of 31 bands, 1≦i≦31).
The conversion rule for pattern #3 is shown, in which the spectral reflection spectrum intensity in the high numerical range H is allocated to the gradient range of color component R, the spectral reflection spectrum intensity in the medium numerical range M is allocated to the gradient range of color component G, and the spectral reflection spectrum intensity in the low numerical range L is allocated to the gradient range of color component B.

In FIG. 13B, the visualization texture image generating unit 108 reads out the three-dimensional shape model 921 from the three-dimensional shape model storage unit 110 .
Then, the visualization texture image generation unit 108 refers to the spectral texture image storage unit 113, and assigns the spectral reflectance spectrum intensity, which is the pixel value of each pixel in the spectral texture image of any one wavelength band, to pixel values in the gradient ranges of each of the color components R, G, and B in accordance with the conversion rule described above ( FIG. 4( a) ), thereby generating a texture image 922 expressed by the pixel values of the color components R, G, and B.
The visualization texture image generation unit 108 renders the texture image 922 onto the three-dimensional shape model 921 to generate a three-dimensional shape model 923, which is then displayed as an image from a predetermined viewpoint position and viewpoint direction.

FIG. 14 is a flowchart showing an example of the operation of a process for generating spectral texture images for a three-dimensional shape model from multi-viewpoint images according to this embodiment.
Step S101:
The data input/output unit 101 adds information about the spectral sensitivity at the time when each viewpoint image in the multi-viewpoint image supplied from the external device was captured, and writes and stores the information in the multi-viewpoint image storage unit 109.
In addition, the data input/output unit 101 writes and stores data of a three-dimensional shape model of a surface model (the surface is composed of a polygon mesh) reconstructed from the above multi-viewpoint image supplied from an external device into the multi-viewpoint image storage unit 109.

Step S102:
The multi-viewpoint image classification unit 102 refers to the multi-viewpoint image storage unit 109 and classifies each of the multi-viewpoint images into a spectral sensitivity group in accordance with the spectral sensitivity added to the multi-viewpoint image.
Then, the multi-viewpoint image classifying unit 102 writes and stores each of the multi-viewpoint images classified into the spectral sensitivity groups in the corresponding spectral sensitivity group data table.

Step S103:
The corresponding coordinate information generation unit 104 generates, for each spectral sensitivity group, corresponding coordinate information indicating the correspondence between the coordinates (first coordinates) of pixels in the multi-viewpoint image (group of multi-viewpoint images for creating spectral sensitivities) and the coordinates (second coordinates) of pixels in the development view (Figure 6 (b)) of the polygon mesh in the three-dimensional shape model, and writes and stores the generated information in a corresponding coordinate information table in the corresponding coordinate information storage unit 111.
Here, the corresponding coordinate information generation unit 104 generates the above-mentioned corresponding coordinate information from third corresponding coordinate information indicating the correspondence between the coordinates of the pixel of the multi-view image (first coordinates) and the coordinates of the pixel of the three-dimensional shape model (coordinate point, third coordinate in Figure 6 (a)), and second corresponding coordinate information indicating the correspondence between the coordinates of the pixel of the three-dimensional shape model (coordinate point, third coordinate) and the coordinates of the pixel of the unfolded drawing (second coordinates).

At this time, the corresponding coordinate information generating unit 104 generates the second corresponding coordinate information and the third corresponding coordinate information for one of the spectral sensitivity groups, and then, by referring to the coordinates in the third corresponding coordinate information, generates a second corresponding coordinate information table (a table into which the second corresponding coordinate information is written) for another spectral sensitivity group so that the second corresponding coordinate information has the same origin as the third corresponding coordinate information, and writes and stores it in the corresponding coordinate information storage unit 111.
As a result, the corresponding coordinate information generating unit 104 uses a common coordinate system for all second corresponding coordinate tables of the spectral sensitivity groups, and sets the coordinate values of the same pixels to be the same between the developments of the spectral sensitivity groups.

Step S104:
The texture image generation unit 105 references the corresponding coordinate information table in the corresponding coordinate information storage unit 111 and the same-viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 109, and generates each of the texture images for each spectral sensitivity group (each imaging device).
That is, the texture image generation unit 105 generates each of the texture images (three bands each of color component R, color component G, and color component B) by assigning pixel values to pixels in the multi-viewpoint image of the optical axis vector of the optical axis of the viewpoint information that is most parallel to the normal of the mesh of the three-dimensional shape model, for the coordinates of the unfolded view in the corresponding coordinate information.
Then, the texture image generating unit 105 writes and stores each of the generated texture images in the texture image storage unit 112 for each spectral sensitivity (image capture device) group.

Step S105:
The texture image correction unit 106 performs processing to correct the deviation of pixel coordinate positions between spectral sensitivity groups in each texture image for each photosensitivity group stored in the texture image storage unit 112 .
That is, the texture image correction unit 106 determines the similarity of all patterns of pixels in the detection frame while shifting the detection frame between texture images of the spectral sensitivity groups, with the center pixel of the a×b pixels in the detection frame as the target pixel.

The texture image correction unit 106 then shifts the detection frame with the target pixel at its center, sets the position with the highest similarity as the correction position, and corrects the coordinate misalignment of pixels between the texture images by aligning the coordinates of each texture image with different spectral sensitivities using the coordinates of one of the texture images between the spectral sensitivity groups as a reference.
The texture image correcting unit 106 writes and stores the corrected texture image in the texture image storage unit 112 (for example, overwrites the original texture image with the corrected texture image).

Step S106:
The spectral texture image generation unit 107 uses the pixel values of the texture image for each spectral sensitivity group in the texture image storage unit 112 (i.e., the pixel values of the pixels in each of the multi-viewpoint images in the same multi-viewpoint image table (the respective gradations of the color components R, G, and B) assigned to each pixel of the development drawing), the spectral sensitivity of the imaging device that captured each of the multi-viewpoint images, and the spectral distribution of the light source, to calculate the intensity of the spectral reflectance spectrum of each pixel of the development drawing corresponding to each wavelength band using each of equations (1) and (2).

That is, the spectral texture image generation unit 107 creates a development in which the polygon mesh of the three-dimensional shape model is developed, and calculates and assigns the intensity of the spectral reflectance spectrum as the pixel values of the pixels in each region in this development, that is, the pixel values of the pixels in the pixel region surrounded by the coordinate values of the vertices of the polygon mesh, using equations (1) and (2) from the pixel values of each color component in the texture image corresponding to the development, the spectral sensitivity, and the spectral distribution.
Here, the spectral texture image generation unit 107 generates a spectral texture image corresponding to one wavelength band by assigning the intensity of the spectral reflectance spectrum as a pixel value to each pixel in all areas of the development drawing.
The spectral texture image generating unit 107 then generates spectral texture images (31 images) for each of the predetermined wavelength bands (31 types), and writes and stores them in a spectral texture image table in the spectral texture image storage unit 113 .

Step S107:
The spectral texture image generating unit 107 determines whether or not spectral texture images for all wavelength bands have been generated.
At this time, if the spectral texture image generating unit 107 determines that spectral texture images for all wavelength bands have been generated, it ends the processing.
On the other hand, if the spectral texture image generator 107 determines that spectral texture images for all wavelength bands have not been generated, the process proceeds to step S106.

As described above, according to this embodiment, a three-dimensional shape model is reproduced from multi-viewpoint images captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a commercially available digital camera that is available to general users. Corresponding coordinate information for each spectral sensitivity group, which has the same coordinate system, is generated based on the pixels (coordinate points) of the three-dimensional shape model. Each of these corresponding coordinate systems is used to generate a texture image (three bands: R, G, and B color components) for each spectral sensitivity group. These texture images can then be used to generate a spectral texture image. This allows the spectral reflectance of the target object to be obtained as a more detailed spectral reflectance spectrum than images captured using the normal RGB color components. This allows for the easy generation of a spectral texture image for the three-dimensional shape model as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of a specified number of wavelength bands (more than the three bands of R, G, and B color components (e.g., 31 bands of different wavelengths)).

Furthermore, according to this embodiment, spectral texture images of multiple wavelength bands (e.g., 31) are generated and can be observed on a display device that displays the color components R, G, and B. By corresponding each pixel value of the spectral texture images for each wavelength to the pixel values of the respective color components R, G, and B, the images can be texture-mapped onto the restored three-dimensional shape model, and the texture-mapped three-dimensional shape model can be observed on the display device, allowing the spectral reflectance state of the target object to be visually recognized in more detail.

Second Embodiment
A second embodiment of the present invention will be described below with reference to the drawings.
The configuration of the second embodiment is similar to the configuration of the first embodiment of the present invention shown in Fig. 1. Below, the operations of the spectral texture image generation system according to the second embodiment that differ from those of the first embodiment will be described.
In the first embodiment, the corresponding coordinate information for each of the spectral sensitivity groups is configured so that the coordinate systems are the same.
On the other hand, the corresponding coordinate information for each of the spectral sensitivity groups in this embodiment is not generated in the same coordinate system, that is, in different coordinate systems.

As in the first embodiment, the corresponding coordinate information generating unit 104 generates, for each spectral sensitivity group, corresponding coordinate information used when generating a texture image from a three-dimensional shape model and a multi-viewpoint image.
At this time, the corresponding coordinate information generation unit 104 generates a corresponding coordinate information table (Figure 5) that shows corresponding coordinate information indicating the correspondence between a pixel (second pixel) in the second coordinate system (UV coordinate system), which is the coordinate system of the texture image, and a pixel (first pixel) in the first coordinate system, which is the coordinate system of the multi-viewpoint image.

The corresponding coordinate information table is provided for each spectral sensitivity group, and shows the correspondence between the coordinate values in the second coordinate system (UV coordinate system) of the pixels that make up the unfolded view and the coordinates in the first coordinate system of the pixels that make up the multi-viewpoint image on a record-by-record basis.
In this embodiment, the second coordinate system is different for each spectral sensitivity group. The difference in the coordinate system includes a shift in the origin, a difference in the position of the mesh in the development, etc., and indicates a state in which the coordinate positions in the second coordinate system are different for each spectral sensitivity group even though the pixels (coordinate points) in the 3D shape model are the same pixels.

The corresponding coordinate information generation unit 104 generates, for each spectral sensitivity group, corresponding coordinate information to be used when generating development diagrams (see FIG. 6(b) below) for each of the color components R, G, and B, using a multi-viewpoint image of each piece of viewpoint information and a three-dimensional shape model (see FIG. 6(a) below, represented by a polygon mesh).
Here, the corresponding coordinate information generation unit 104 reads data of the polygon mesh that forms the three-dimensional shape model from the three-dimensional shape model storage unit 110, and generates second corresponding coordinate information (information indicating the relationship between a third coordinate of the coordinate values of the three-dimensional shape and a second coordinate of the coordinates in the development) for each spectral sensitivity group, which is used when generating a development of the polygon mesh (for example, the development of Figure 6(b) obtained by UV development of the polygon mesh of Figure 6(a)).

The corresponding coordinate information generating unit 104 then writes and stores the generated second coordinate correspondence information in a corresponding coordinate information table for each spectral sensitivity group in the corresponding coordinate information storage unit 111 .
In this second corresponding coordinate information, the coordinates of the same pixel (coordinate point) in the three-dimensional shape model differ from the coordinate positions in the second coordinate system of the development for each spectral sensitivity group.

On the other hand, the correspondence coordinate information generation unit 104 generates third coordinate correspondence information that indicates the correspondence between the coordinates of a pixel in a multi-view image (first coordinates) and the coordinates of a pixel in a three-dimensional shape model (coordinate values, third coordinates), as in the first embodiment.
The corresponding coordinate information generating unit 104 then writes and stores the generated third coordinate correspondence information in a corresponding coordinate information table for each spectral sensitivity group in the corresponding coordinate information storage unit 111 .

Furthermore, because the coordinate system of the development diagram in the second coordinate correspondence information differs for each spectral sensitivity group, when generating a spectral texture image (described later), the pixel values of the same pixels are required to calculate the spectral reflectance of the pixels in the spectral texture image, and in order to extract the same pixels between each spectral sensitivity group, the correspondence coordinate information generation unit 104 generates fourth correspondence coordinate information.
As already mentioned, the coordinate system of the development in the second coordinate correspondence information differs for each spectral sensitivity group at the time of generation.

Therefore, when generating a spectral texture image (described later), the pixel values of the same pixels in each of the multi-viewpoint images are required to calculate the spectral reflectance of the pixels in the spectral texture image.
The corresponding coordinate information generating unit 104 generates fourth corresponding coordinate information used to extract identical pixels between the multi-viewpoint images of the spectral sensitivity group.
The fourth corresponding coordinate information is information indicating the correspondence between the coordinates (coordinate values, third coordinates) of pixels in the three-dimensional shape model and the coordinates (second coordinates) of pixels in the development diagram for each spectral sensitivity group.

FIG. 15 is a diagram showing an example of the configuration of the fourth corresponding coordinate information table written and stored in the corresponding coordinate information storage unit 111 in this embodiment.
The fourth correspondence coordinate information table indicates, for each record, the correspondence between the coordinate values of pixels in each development of each spectral sensitivity group and the coordinate values of pixels in a polygon mesh in a three-dimensional shape model. Fig. 15 shows an example of the configuration of the fourth correspondence coordinate information table when there are three spectral sensitivity groups (e.g., spectral sensitivity groups based on three image capture devices), for example, spectral sensitivity group #1, spectral sensitivity group #2, and spectral sensitivity group #3.

The three-dimensional shape model coordinate values indicate the coordinate values of each pixel (coordinate point) in the three-dimensional shape model.
The development #1 coordinate values indicate the coordinates of pixels in the development #1, which is a development of the spectral sensitivity group #1.
Similarly, the development #2 coordinate values indicate the coordinates of pixels in the development #2, which is a development of the spectral sensitivity group #2.
Moreover, the development #3 coordinate values indicate the coordinates of pixels in the development #3, which is a development of the spectral sensitivity group #3.

Returning to FIG. 1 , the correspondence coordinate information generation unit 104 references the second correspondence coordinate information table for each spectral sensitivity group in the correspondence coordinate information storage unit 111, and integrates the second correspondence coordinate information using the three-dimensional shape model coordinate values as arguments (associated key data) to generate fourth correspondence coordinate information.
Then, the corresponding coordinate information generating unit 104 writes and stores the generated fourth corresponding coordinate information in a fourth corresponding coordinate information table in the corresponding coordinate information storage unit 111 .

The texture image generating unit 105 references each of the same viewpoint multi-viewpoint image tables in the multi-viewpoint image storage unit 109 and the corresponding coordinate information table in the corresponding coordinate information storage unit 111 to generate a texture image for each spectral sensitivity group.
Then, the texture image generating unit 105 writes and stores each of the generated texture images for each spectral sensitivity group in the texture image storage unit 112 .

The texture image correction unit 106 compares each pixel in each texture image in the texture image storage unit 112 using a detection frame of, for example, a × b pixels (a and b are odd integer values of 3 or more), and sequentially calculates the similarity of the pixels in the centers of the detection frames, thereby aligning the positions of pixels between the texture images as positions with high similarity, thereby correcting the coordinates of the pixels in the unfolded view in the second correspondence coordinate information in the second correspondence coordinate information table.
Then, the texture image correcting unit 106 writes and stores the corrected second corresponding coordinate information table in the corresponding coordinate information storage unit 111 (for example, overwrites the second corresponding coordinate information table generated by the corresponding coordinate information generating unit 104).

Furthermore, in conjunction with the correction of the second correspondence coordinate information in the second correspondence coordinate information table, the texture image correction unit 106 also corrects the coordinates of the pixels of the development in the fourth correspondence coordinate information in the fourth correspondence coordinate information table.
The texture image correcting unit 106 writes and stores the corrected fourth correspondence coordinate information table in the correspondence coordinate information storage unit 111 (for example, overwrites the fourth correspondence coordinate information table generated by the correspondence coordinate information generating unit 104).

The spectral texture image generation unit 107 generates spectral texture images for each of a plurality of wavelength bands (e.g., 31 bands, each 10 nm apart from a wavelength of 400 nm to a wavelength of 700 nm) from the texture image generated by the texture image generation unit 105 (or the texture image corrected by the texture image correction unit 106).
Here, the spectral texture image generation unit 107 refers to the texture images of each spectral sensitivity group in the spectral texture image storage unit 113, each of the same-viewpoint multi-viewpoint image tables in the multi-viewpoint image storage unit 109, and the corresponding coordinate information table and fourth corresponding coordinate information table in the corresponding coordinate information storage unit 111, and generates, for example, a 31-band spectral texture image.

The process in which the spectral texture image generating unit 107 calculates the spectral reflectance from the pixel value of the same pixel in the texture image of each spectral sensitivity group is the same as in the first embodiment.
In this embodiment, as already described, the coordinate system in the development of the corresponding coordinate information differs for each spectral sensitivity group.
Therefore, when calculating the spectral reflectance of each pixel in the spectral texture image, the spectral texture image generation unit 107 refers to the fourth correspondence coordinate information table, extracts identical pixels from the texture images of each spectral sensitivity group, and uses the pixel values of the extracted pixels in the texture images to calculate the spectral reflectance of each pixel in the spectral texture image.

Furthermore, the spectral texture image generating unit 107 generates a development of a predetermined spectral sensitivity group, for example, a spectral texture image corresponding to the coordinate system of development #1.
At this time, the spectral texture image generating unit 107 assigns the calculated spectral reflectance to the coordinates of the texture image of the development #1 used to calculate the spectral reflectance, thereby generating a spectral texture image.
The spectral texture image generating unit 107 then writes and stores the generated spectral texture image in the spectral texture image storage unit 113 .

As described above, according to this embodiment, a three-dimensional shape model is reproduced from multi-viewpoint images captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a commercially available digital camera that is accessible to general users. Correspondence coordinate information for each spectral sensitivity group, which has a different coordinate system, is generated based on the pixels (coordinate points) of the three-dimensional shape model. Each of these correspondence coordinate systems is used to generate texture images (three bands of color components R, G, and B) for each spectral sensitivity group. A fourth correspondence coordinate information that integrates the correspondence coordinate information for each spectral sensitivity group is used to generate a spectral texture image using each of the texture images in the different coordinate systems. This allows the spectral reflectance of the target object to be obtained as a more detailed spectral reflectance spectrum than images captured using normal RGB color components. This allows for the easy generation of a spectral texture image for the three-dimensional shape model as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of a specified number of wavelength bands (a number of bands greater than the three bands of RGB color components (e.g., 31 bands of different wavelengths)).

Third Embodiment
A third embodiment of the present invention will be described below with reference to the drawings.
The configuration of the third embodiment is similar to the configuration of the first embodiment of the present invention shown in Fig. 1. Below, the operations of the spectral texture image generation system according to the third embodiment that differ from those of the first embodiment will be described.
In the first embodiment, the corresponding coordinate information for each of the spectral sensitivity groups is configured so that the coordinate systems are the same.
On the other hand, in this embodiment, the corresponding coordinate information generation unit 104 generates the corresponding coordinate information for each spectral sensitivity group in a manner that the coordinate systems for each of the spectral sensitivity groups are not the same, i.e., different coordinate systems, as in the second embodiment.

At this time, the correspondence coordinate information generation unit 104 generates fourth correspondence coordinate information indicating the correspondence between the coordinate values of pixels (coordinate values) in the three-dimensional shape model and the coordinates of pixels in each development diagram of the spectral sensitivity group, as in the second embodiment.
In this embodiment, the texture image generating unit 105 generates texture images in different coordinate systems in accordance with the corresponding coordinate information between the spectral sensitivity groups, as in the second embodiment.
Furthermore, the corresponding coordinate information generating unit 104 refers to the fourth corresponding coordinate information and modifies the corresponding coordinate system in the second corresponding coordinate information of another spectral sensitivity group in accordance with the correspondence between the coordinates of a pixel in the development of one of the spectral sensitivity groups and the coordinates of a pixel (coordinate point) in the three-dimensional shape model.

The texture image generation unit 105 rearranges the pixels in each of the texture images of the different coordinate systems in accordance with the correspondence between the coordinates of pixels in the development diagram of one of the spectral sensitivity groups and the coordinates of pixels (coordinate points) in the three-dimensional shape model, by referring to the fourth correspondence coordinate information.
The texture image generating unit 105 generates texture images having the same coordinate system between spectral sensitivity groups by rearranging pixels in the texture images described above, similarly to the first embodiment.
The subsequent correction of pixel positional deviations in the texture image and generation of the spectral texture image are the same as those in the first embodiment, and therefore will be omitted.

As described above, according to this embodiment, a three-dimensional shape model is reproduced from multi-viewpoint images captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a commercially available digital camera that is available to general users. Corresponding coordinate information for each spectral sensitivity group, which has the same coordinate system, is generated based on the pixels (coordinate points) of the three-dimensional shape model. Each of these corresponding coordinate systems is used to generate a texture image (three bands: R, G, and B color components) for each spectral sensitivity group. These texture images can then be used to generate a spectral texture image. This allows the spectral reflectance of the target object to be obtained as a more detailed spectral reflectance spectrum than an image captured using the normal R, G, and B color components. This allows for the easy generation of a spectral texture image for the three-dimensional shape model as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of a specified number of wavelength bands (more than the three bands of R, G, and B color components (e.g., 31 bands of different wavelengths)).

<Fourth embodiment>
A fourth embodiment of the present invention will be described below with reference to the drawings.
FIG. 16 is a block diagram showing an example of the configuration of a spectral texture image generation system according to the fourth embodiment of the present invention.
In FIG. 16 , the spectral texture image generation system 100B includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a corresponding coordinate information generation unit 104, a spectral texture image generation unit 107, a visualization texture image generation unit 108, a multi-viewpoint image storage unit 109, a three-dimensional shape model storage unit 110, a corresponding coordinate information storage unit 111, a texture image storage unit 112, a spectral texture image storage unit 113, and a corresponding coordinate information correction unit 114.

In the first to third embodiments, texture images are generated based on the corresponding coordinate information of each of the spectral sensitivity groups, and the spectral texture image is generated using the generated texture images.
On the other hand, in this embodiment, the spectral texture image generating unit 107 generates the spectral texture images of each spectral sensitivity group directly from the multi-viewpoint images based on the corresponding coordinate information generated by the corresponding coordinate information generating unit 104.
The configuration and operation of the fourth embodiment, which differ from the first embodiment, will be described below.

As in the first embodiment, the corresponding coordinate information generating unit 104 generates corresponding coordinate information having the same coordinate system among the spectral sensitivity groups from each of the second corresponding coordinate information and the third corresponding coordinate information.
Furthermore, similarly to the second embodiment, the corresponding coordinate information generating unit 104 generates corresponding coordinate information in coordinate systems that differ between the spectral sensitivity groups, and fourth corresponding coordinate information.

Then, the corresponding coordinate information generation unit 104 refers to the fourth corresponding coordinate information and each of the corresponding coordinate information of the spectral sensitivity groups, and, using the corresponding coordinate information of one of the spectral sensitivity groups as a reference, changes the corresponding coordinate information of the other spectral sensitivity groups to the coordinates of the development diagram of the corresponding coordinate information.
As a result, the corresponding coordinate information generating unit 104 generates corresponding coordinate information that indicates the correspondence between the coordinates of pixels in the multi-view image and the coordinates of pixels in the development, using the same coordinate system.

The corresponding coordinate information correction unit 114 refers to the corresponding coordinate information of each spectral sensitivity group and the third coordinate correspondence information, and corrects the coordinate misalignment of pixels in the multi-viewpoint image of the spectral sensitivity group that correspond to pixels (coordinate points) of the same three-dimensional shape model.
At this time, the corresponding coordinate information correction unit 114, for example, compares each of the pixels surrounded by a detection frame of a × b pixels (a and b are odd integer values greater than or equal to 3) in each multi-viewpoint image of the spectral sensitivity group, and sequentially calculates the similarity of the patterns made up of the pixel values of each of the pixels within the detection frame, using the pixel in the center of the detection frame as the target pixel.

Then, the corresponding coordinate information corrector 114 detects coordinates with a high degree of pattern similarity within a detection frame that is set at the center of a target pixel in each of the multi-viewpoint images of each spectral sensitivity group.
Here, the corresponding coordinate information correction unit 114 corrects the coordinates of the pixels of the multi-viewpoint images in each of the corresponding coordinate information by matching the coordinates of each target pixel in each multi-viewpoint image to the coordinates that result in the highest similarity between the multi-viewpoint images, in accordance with the results of the above detection.

In this way, the corresponding coordinate information correcting unit 114 corrects the positional deviation of each pixel of the multi-viewpoint image between the pieces of corresponding coordinate information generated by the corresponding coordinate information generating unit 104 for each spectral sensitivity group.
The corresponding coordinate information correcting unit 114 writes and stores each corrected piece of corresponding coordinate information in the corresponding coordinate information storage unit 111 .

The spectral texture image generation unit 107 references the corresponding coordinate information table in the corresponding coordinate information storage unit 111 and the same viewpoint multi-viewpoint image table in the multi-viewpoint image storage unit 109, and generates a spectral texture image in the same manner as in the first embodiment.
That is, the spectral texture image generation unit 107 reads the image data of the corresponding multi-viewpoint image from the same-viewpoint multi-viewpoint image table for each pixel of the development drawing based on the correspondence between the coordinates of the pixels of the multi-viewpoint image in the corresponding coordinate information of each spectral sensitivity group and the coordinates of the pixels of the development drawing, and calculates the spectral reflectance (spectral reflectance spectrum of each band) using equations (1) and (2).

The spectral texture image generating unit 107 then calculates the spectral reflectance of each pixel of the development for each band, and generates a spectral texture image with four or more bands, for example, 31 bands as in the first embodiment.
The spectral texture image generating unit 107 writes and stores each of the generated 31-band spectral texture images in the spectral texture image storage unit 113 .

As described above, according to this embodiment, a three-dimensional shape model is reproduced from multi-viewpoint images captured with different spectral sensitivities and viewpoint information (viewpoint position, optical axis vector) using an imaging device such as a commercially available digital camera that is accessible to general users. Corresponding coordinate information is generated for each spectral sensitivity group, which has the same coordinate system based on the pixels (coordinate points) of the three-dimensional shape model. Using each piece of corresponding coordinate information, a spectral texture image can be generated from each multi-viewpoint image of the spectral sensitivity group (an image with three bands: R, G, and B color components). This allows the spectral reflectance of the target object to be obtained as a more detailed spectral reflectance spectrum than an image captured with the normal R, G, and B color components. This allows for the easy generation of a spectral texture image for the three-dimensional shape model as a texture image that accurately reflects the spectral reflectance of the target object corresponding to each of a specified number of wavelength bands (a number of bands greater than the three bands of R, G, and B color components (e.g., 31 bands of different wavelengths)).

Fifth Embodiment
A fifth embodiment of the present invention will be described below with reference to the drawings.
17 is a block diagram showing an example of the configuration of a spectral texture image generation system according to a fifth embodiment of the present invention. In Fig. 17, the spectral texture image generation system 100C includes a data input/output unit 101, a multi-viewpoint image classification unit 102, a corresponding coordinate information generation unit 104, a texture image generation unit 105, a texture image correction unit 106, a spectral distribution estimation unit 115, a spectral texture image generation unit 107, a visualization texture image generation unit 108, a multi-viewpoint image storage unit 109, a three-dimensional shape model storage unit 110, a corresponding coordinate information storage unit 111, a texture image storage unit 112, a spectral texture image storage unit 113, and a spectral distribution basis vector storage unit 116.

In this embodiment, too, captured image data of an object to be reconstructed as a three-dimensional shape model is supplied as multi-viewpoint images from an external device (not shown). Only the configuration and operation that differ from the first embodiment will be described below.
In the first to fourth embodiments, the spectral distribution of the light source when each multi-viewpoint image is captured is acquired as already measured spectral distribution data.
In the fifth embodiment, when estimating the spectral reflectance (that is, the spectral reflectance spectrum of a plurality of bands, for example, 31 bands), the spectral distribution is also estimated.

The spectral distribution estimation unit 115 estimates the spectral distribution of the light source in the environment in which the multi-viewpoint images (images taken from different viewpoints) are captured using a plurality of imaging devices with different spectral sensitivities.
The spectral distribution estimation unit 115 reads, for example, a plurality of spectral distribution basis vectors from the spectral distribution basis vector storage unit 116. Then, similar to the case of spectral reflectance, the spectral distribution estimation unit 115 expresses the spectral distribution of the light source in the environment in which the multi-viewpoint image is captured as a weighted sum of the plurality of read spectral distribution basis vectors. The spectral distribution estimation unit 115 estimates the spectral distribution of the light source by estimating weighting coefficients that weight each spectral distribution basis vector.

Here, the spectral distribution basis vectors are derived from a spectral distribution data group, which is a collection of spectral distributions of a plurality of light sources, using principal component analysis as spectral distribution basis vectors for expressing the spectral reflectance data group.
In this principal component analysis, as in the case of spectral reflectance, it becomes possible to express the spectral distribution using a combination of a smaller number of spectral distribution basis vectors than the actual dimensions of the spectral distribution. In other words, by using the spectral distribution basis vectors, the dimensions of the spectral distribution of the light source to be estimated are compressed, reducing the calculation load.

That is, the spectral distribution estimation unit 115 estimates the spectral distribution l by expressing it as a weighted sum of a plurality of spectral distribution basis vectors _ej = [ _ej(λ1) , ej _(λ2) , ...] ^T and finding the weighting coefficient a = [ _a1 , _a2 , ...] ^T of each of the spectral distribution basis vectors, as shown in the following equation (3). Here, the spectral distribution basis vectors are, for example, stored in advance as a set in the spectral distribution basis vector storage unit 116. Furthermore, the spectral distribution basis vector storage unit 116 may store a plurality of spectral distribution basis vectors arranged in descending order of principal component scores in principal component analysis, and a configuration may be adopted in which spectral distribution basis vectors up to a predetermined order are read and used as needed (described later).
The synthesis of this spectral distribution basis vector is similar to the case where the spectral reflectance is expressed by synthesizing the spectral reflectance basis vectors shown in FIG.

The spectral distribution estimation unit 115 then estimates each weighting coefficient a by solving the following equation (4) for the spectral distribution expressed by the above equation (3). The spectral distribution estimation unit 115 then estimates the spectral distribution l using equation (3) with the estimated weighting coefficients. Here, the spectral distribution l is expressed as a 31-dimensional vector sampled in 10 nm increments over the wavelength range from 400 nm to 700 nm, as in the case of spectral reflectance, for example.

In the above formula (4), p _m,k is the pixel value of the k-th channel of the m-th texture image generated from the multi-viewpoint image. That is, in p _m,k , m is a number indicating the texture image generated from the multi-viewpoint image in the viewpoint group for spectral reflectance estimation. Furthermore, k indicates the channel of the color component of a portion in the texture image. For example, if a portion is represented by pixel values of the color components R, G, and B, then channel 1 is the color component R, channel 2 is the color component G, and channel 3 is the color component B.

In addition, in equation (4), the matrix C _m,k indicates the spectral sensitivity of the k-th channel (wavelength band of color components, i.e., band) of the m-th texture image, and the element c _m,k (λ) in this C _m,k indicates the spectral sensitivity of the wavelength λ in the k-th channel of the m-th texture image.
The texture image used in equation (4) is a texture image generated by sampling several parts of a three-dimensional shape model and then using multi-viewpoint images of a group of viewpoints for estimating the spectral reflectance of the sampled parts.
The spectral distribution estimation unit 115 then estimates the spectral distribution of the light source by solving equation (4) using the spectral reflectance of the sampled portion, the pixel values corresponding to the portion in each multi-view image, and the spectral sensitivity of the imaging device that captured the multi-view image, thereby estimating the weighting coefficient a. The portion to be sampled may be selected from the three-dimensional shape model or may be selected from the pixels in the multi-view image.

Here, the spectral distribution estimation unit 115 may be configured to select and read out a set of spectral distribution basis vectors from the spectral distribution basis vector storage unit 116 in accordance with the number of spectral sensitivity groups in the viewpoint group for estimating spectral reflectance, as in the case of estimating spectral reflectance.
Furthermore, when solving equation (3), the upper limit of the number of spectral distribution basis vectors (principal components) N _χ used to represent the spectral distribution data (hereinafter referred to as the number of spectral distribution basis vectors) is determined based on, for example, the number of spectral sensitivity groups that capture multi-viewpoint images (the number of image capture devices, the number of spectral sensitivity adjustments using color-changing filters, etc.).

Furthermore, a group of viewpoints for estimating spectral reflectance is obtained for each part of the three-dimensional shape model, and the number m of spectral sensitivity groups is different for each part. Therefore, the set of spectral distribution basis vectors (the number N _χ of spectral distribution basis vectors) that the spectral distribution estimation unit 115 reads from the spectral distribution basis vector storage unit 116 is different for each part of the three-dimensional shape model.

When estimating the spectral reflectance of each part of a three-dimensional shape model and the spectral distribution of a light source, the estimation of the spectral reflectance and the spectral distribution is alternately repeated multiple times.
When starting from estimating the spectral reflectance, the spectral texture image generation unit 107 reads the spectral distribution of a preset default light source from the spectral texture image storage unit 113, and estimates the spectral reflectance using equations (1) and (2).
Next, when estimating the spectral distribution, the spectral distribution estimation unit 115 uses the spectral reflectance obtained from the spectral distribution of the default light source to estimate the spectral distribution using equations (3) and (4).

On the other hand, when starting from estimating the spectral distribution, the spectral distribution estimation unit 115 estimates the spectral distribution using equations (3) and (4) by using a preset default spectral reflectance for the portion sampled from the three-dimensional shape model.
Next, when estimating the spectral reflectance, the spectral texture image generating unit 107 uses the spectral distribution obtained from the default spectral reflectance and estimates the spectral reflectance using equations (1) and (2).

As described above, according to this embodiment, by repeatedly estimating the spectral reflectance of each voxel portion of a three-dimensional shape model and estimating the spectral distribution using a portion sampled from that portion, it is possible to obtain, with increased accuracy, the spectral reflectance of each portion of the three-dimensional shape model and the spectral distribution of the light source in the environment in which the multi-viewpoint image was captured.
In this case, the number of times to repeat the estimation of the spectral distribution and the estimation of the spectral reflectance may be set in advance, and the processing may be terminated when the number of repetitions reaches this set number, or the processing may be terminated when the difference between the obtained numerical value and the numerical value of the immediately preceding calculation result reaches a predetermined value.

Furthermore, when estimating the spectral reflectance, the spectral distribution of the light source may be estimated using either a multi-view image in which a reference object (e.g., a color chart) with known spectral reflectance is captured in the same image as the target object, or a reference image in which the reference object is captured under the same light source as the target object.
In this case, when using a multi-viewpoint image in which the reference object and the target object are captured as the same image, the spectral distribution estimation unit 115 selects a multi-viewpoint image in which the reference object is captured from at least two of the spectral sensitivity groups, and generates a group of viewpoints for light source estimation.

Then, the spectral distribution estimation unit 115 calculates the spectral distribution of the light source that captured the multi-viewpoint image using the spectral sensitivity corresponding to the multi-viewpoint image in the group of viewpoints for light source estimation, the known spectral reflectance of a specific part of the reference object, and the pixel value of the position corresponding to the above-mentioned specific part of the multi-viewpoint image in the group of viewpoints for light source estimation, using equations (3) and (4).
Furthermore, in this embodiment, the first embodiment has been described as being provided with the spectral distribution estimation unit 115 that estimates the spectral distribution of the light source and the spectral distribution basis vector storage unit 116, but these may also be provided in each of the second to fourth embodiments.

As described above, according to this embodiment, by repeating the process of estimating the three-dimensional shape and spectral reflectance of the target object and the process of generating a grayscale image from a multi-viewpoint image corresponding to the spectral reflectance of the target object, the spectral distribution viewpoint image of the light source, and the spectral sensitivity, it is possible to improve the accuracy of estimating the three-dimensional shape of the target object as well as the accuracy of estimating the spectral reflectance.

Furthermore, the first to fifth embodiments described above may be configured to include a three-dimensional shape restoration unit (not shown) that reads each of the multi-viewpoint images of the target object stored in the multi-viewpoint image storage unit 109, estimates the three-dimensional shape of the target object, and generates a three-dimensional shape model. The three-dimensional shape restoration unit then writes and stores data of the three-dimensional shape model of the target object in the three-dimensional shape model storage unit 110.
Here, with regard to the method of generating a 3D shape model, for example, a depth map may be generated from a multi-view image, and a 3D shape model may be generated from this depth map as a 3D point cloud, and then generated as a 3D mesh model (e.g., a triangular mesh model) represented by a polygonal mesh (see, for example, Shuji Sakai, Takashi Watanabe, Tomohito Masuda, Koichi Ito, and Takafumi Aoki: Proposal of a High-Precision Multi-View 3D Reconstruction Algorithm Based on Phase-Only Correlation Method, 20th Image Recognition and Understanding Symposium (2017)).
Furthermore, when reproducing the object as a three-dimensional point cloud or mesh, the three-dimensional shape restoration unit also estimates the camera parameters of the image capturing device used to capture each of the multi-viewpoint images.

In addition, a program for implementing each function of the spectral texture image generation systems shown in Figures 1, 16, and 17 of the present invention may be recorded on a computer-readable recording medium. The program recorded on the recording medium may then be loaded into a computer system and executed to generate spectral texture images of each wavelength band corresponding to a hyperspectral image of a three-dimensional shape model of a target object. Note that the term "computer system" herein includes hardware such as an operating system (OS) and peripheral devices. It also includes a World Wide Web (WWW) system equipped with a homepage provision environment (or display environment). Furthermore, "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs (Read Only Memory), and CD-ROMs (Compact Disc-Read Only Memory), as well as storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes devices that retain programs for a certain period of time, such as volatile memory (RAM (Random Access Memory)) within computer systems that function as servers or clients when programs are transmitted via networks such as the Internet or communication lines such as telephone lines.

The above program may also be transmitted from a computer system that stores the program in a storage device or the like to another computer system via a transmission medium, or by transmission waves within the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called differential file (differential program) that can realize the above-mentioned functions in combination with a program already recorded on the computer system.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment and includes design modifications and the like that do not deviate from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100, 100B, 100C...Spectral texture image generation system 101...Data input/output unit 102...Multi-viewpoint image classification unit 104...Corresponding coordinate information generation unit 105...Texture image generation unit 106...Texture image correction unit 107...Spectral texture image generation unit 108...Visualization texture image generation unit 109...Multi-viewpoint image storage unit 110...Three-dimensional shape model storage unit 111...Corresponding coordinate information storage unit 112...Texture image storage unit 113...Spectral texture image storage unit 114...Corresponding coordinate information correction unit 115...Spectral distribution estimation unit 116...Spectral distribution basis vector storage unit

Claims (14)

a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images, as a texture image of a three-dimensional shape model of the target object;
a multi-viewpoint image classification unit that classifies each of the multi-viewpoint images obtained by capturing the target object using imaging devices with a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generating unit that generates, for each of the spectral sensitivity groups, corresponding coordinate information that is information indicating a correspondence relationship between first coordinates that indicate coordinates of first pixels that are pixels that constitute the multi-viewpoint image and second coordinates that indicate coordinates of second pixels that are pixels that constitute the texture image;
a texture image generation unit that generates the texture image from the multi-viewpoint image using the corresponding coordinate information, corresponding to each of the spectral sensitivity groups;
a spectral texture image generation unit that references the corresponding coordinate information, estimates a spectral reflectance spectrum of four or more bands of the target object from the texture image, and generates spectral texture images corresponding to each of the spectral reflectance spectra. The corresponding coordinate information generation unit
generating the corresponding coordinate information by using the same coordinate system for the second coordinates of each of the texture images in each of the spectral sensitivity groups;
The spectral texture image generation unit
The spectral texture image generation system according to claim 1 , wherein the spectral texture images are generated from the texture images by referring to the corresponding coordinate information. The corresponding coordinate information generation unit
generating the corresponding coordinate information using the second coordinates of each of the texture images in each of the spectral sensitivity groups as different coordinate systems;
The spectral texture image generation unit
2. The spectral texture image generation system according to claim 1, wherein the spectral texture images are generated from the respective texture images by referring to the corresponding coordinate information and using a third coordinate of a third pixel, which is a coordinate point in the three-dimensional shape model, as a reference . The corresponding coordinate information generation unit
generating corresponding coordinate information for each of the spectral sensitivity groups using the second coordinates of each of the texture images in each of the spectral sensitivity groups as different coordinate systems, and generating integrated corresponding coordinate information that integrates the corresponding coordinate information for each of the spectral sensitivity groups;
The texture image generation unit
generating the texture images for each of the spectral sensitivity groups from each of the multi-viewpoint images with reference to the corresponding coordinate information, and reconstructing the texture images by making the coordinate systems of the texture images identical using the integrated corresponding coordinate information;
The spectral texture image generation unit
The spectral texture image generation system according to claim 1 , wherein the spectral texture images are generated from the respective texture images. a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images as a texture image of a three-dimensional shape model of the target object,
a multi-viewpoint image classification unit that classifies each of the multi-viewpoint images obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generating unit that generates, for each of the spectral sensitivity groups, corresponding coordinate information that indicates a correspondence relationship between first coordinates that indicate coordinates of first pixels that are pixels that constitute the multi-viewpoint image and second coordinates that indicate coordinates of second pixels that are pixels that constitute the spectral texture image;
a spectral texture image generation unit that references the corresponding coordinate information, estimates spectral reflectance spectra of four or more bands of the target object from the multi-viewpoint image , and generates spectral texture images corresponding to each of the spectral reflectance spectra. 5. The spectral texture image generation system according to claim 2, further comprising: a texture image correction unit that detects a positional shift of the pixels between the texture images of each of the spectral sensitivity groups and corrects the positional shift of the texture images. 4. The spectral texture image generation system according to claim 3, further comprising a texture image correction unit that detects a positional shift of the pixel between the texture images of each of the spectral sensitivity groups and corrects the positional shift in the corresponding coordinate information. The spectral texture image generation system according to claim 5, further comprising a texture image correction unit that detects a positional shift of the pixel between the multi-viewpoint images of each of the spectral sensitivity groups and corrects the positional shift in the corresponding coordinate information. The corresponding coordinate information generation unit
6. The spectral texture image generation system according to claim 2, wherein the corresponding coordinate information is generated from second corresponding coordinate information indicating a correspondence relationship between the second coordinates and third coordinates, which are coordinates of pixels of the three-dimensional shape model, and third corresponding coordinate information indicating a correspondence relationship between the third coordinates and first coordinates of the multi-viewpoint image for each of the spectral sensitivity groups. The corresponding coordinate information generation unit
8. The spectral texture image generation system according to claim 3, wherein the corresponding coordinate information is generated from third correspondence coordinate information that indicates a correspondence relationship between third coordinates, which are coordinates of pixels of the three-dimensional shape model, and the first coordinates, which are coordinates of pixels of the multi-viewpoint image, and that is generated for each of the spectral sensitivity groups. A spectral texture image generation method for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images, as a texture image of a three-dimensional shape model of the target object,
a multi-viewpoint image classification step in which a multi-viewpoint image classification unit classifies each of the multi-viewpoint images, which are obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generation step in which a corresponding coordinate information generation unit generates, for each of the spectral sensitivity groups, corresponding coordinate information which is information indicating a correspondence relationship between first coordinates indicating coordinates of first pixels which are pixels constituting the multi-viewpoint image and second coordinates indicating coordinates of second pixels which are pixels constituting the texture image;
a texture image generation step in which a texture image generation unit generates the texture image from the multi-viewpoint image using the corresponding coordinate information in correspondence with each of the spectral sensitivity groups;
a spectral texture image generation step in which a spectral texture image generation unit references the corresponding coordinate information, estimates a spectral reflectance spectrum of four or more bands of the target object from the texture image, and generates spectral texture images corresponding to each of the spectral reflectance spectra. A spectral texture image generation method for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images, as a texture image of a three-dimensional shape model of the target object,
a multi-viewpoint image classification step in which a multi-viewpoint image classification unit classifies each of the multi-viewpoint images, which are obtained by capturing the target object from a plurality of different viewpoints and using imaging devices with a plurality of different spectral sensitivities, into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generation step in which a corresponding coordinate information generation unit generates, for each of the spectral sensitivity groups, corresponding coordinate information which is information indicating a correspondence relationship between first coordinates indicating coordinates of first pixels which are pixels constituting the multi-viewpoint image and second coordinates indicating coordinates of second pixels which are pixels constituting the spectral texture image;
a spectral texture image generation step in which a spectral texture image generation unit references the corresponding coordinate information, estimates a spectral reflectance spectrum of four or more bands of the target object from the multi-viewpoint image , and generates spectral texture images corresponding to each of the spectral reflectance spectra. a program for causing a computer to function as a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images as a texture image of a three-dimensional shape model of the target object,
The computer
a multi-viewpoint image classification means for classifying each of the multi-viewpoint images obtained by capturing the target object using imaging devices with a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generating means for generating, for each of the spectral sensitivity groups, corresponding coordinate information which is information indicating a correspondence relationship between first coordinates indicating coordinates of first pixels which are pixels constituting the multi-viewpoint image and second coordinates indicating coordinates of second pixels which are pixels constituting the texture image;
a texture image generating means for generating the texture image from the multi-viewpoint image using the corresponding coordinate information in correspondence with each of the spectral sensitivity groups;
a program for causing the program to function as a spectral texture image generating means that refers to the corresponding coordinate information, estimates a spectral reflectance spectrum of four or more bands of the target object from the texture image, and generates spectral texture images corresponding to each of the spectral reflectance spectra. a program for causing a computer to function as a spectral texture image generation system for generating a spectral texture image corresponding to a spectral reflectance spectrum of four or more bands in the spectral reflectance of a target object from multi-viewpoint images as a texture image of a three-dimensional shape model of the target object,
The computer
a multi-viewpoint image classification means for classifying each of the multi-viewpoint images obtained by capturing the target object using imaging devices with a plurality of different viewpoints and a plurality of different spectral sensitivities into a spectral sensitivity group having the same spectral sensitivity;
a corresponding coordinate information generating means for generating, for each of the spectral sensitivity groups, corresponding coordinate information which is information indicating a correspondence relationship between first coordinates indicating coordinates of first pixels which are pixels constituting the multi-viewpoint image and second coordinates indicating coordinates of second pixels which are pixels constituting the spectral texture image;
a program for causing the program to function as a spectral texture image generating means that references the corresponding coordinate information, estimates the spectral reflectance spectrum of four or more bands of the target object from the multi-viewpoint image , and generates spectral texture images corresponding to each of the spectral reflectance spectra.