JP7631291B2 - Information processing system, information processing method, and program - Google Patents

Description

This disclosure relates to an information processing system, an information processing method, and a program.

A technique is known in which an original image showing an object is transformed to obtain an image showing the object that is different from the original image. For example, a large number of different images can be obtained by performing operations such as rotating, enlarging/reducing, changing the brightness, or changing the color tone of the original image. The large number of images obtained in this way are used, for example, as training data for a machine learning model for image recognition (see Patent Document 1 below).

Re-tabled publication 2021/038678

The above conventional techniques only allow limited image transformation. For example, it is difficult to change the orientation, pose, shape, or position of an object shown in the original image using the above conventional techniques.

The present invention has been made in consideration of the above problems, and its purpose is to provide an information processing system, information processing method, and program that expand the aspects of image conversion.

The information processing system according to the present disclosure includes an original image acquisition means for acquiring an original image showing an object, a virtual image generation means for generating a virtual image showing the appearance of a virtual space in which the 3D model is placed as seen from a virtual viewpoint based on a 3D model representing the object, lighting data, and texture data, a feature extraction means for extracting original image features from the original image and virtual image features from the virtual image, and a parameter adjustment means for adjusting parameters of the lighting data and parameters of the texture data so as to increase the similarity between the original image features and the virtual image features.

FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing system according to the present embodiment. FIG. 2 is a diagram illustrating an overview of processing in the information processing system according to the present embodiment. FIG. 2 is a functional block diagram showing an example of functions realized in the information processing system of the present embodiment. FIG. 13 is a diagram showing how a virtual image is generated. FIG. 13 is a diagram showing how virtual images viewed from each of a plurality of virtual viewpoints are generated. FIG. 2 is a flow chart showing an example of processing executed in the information processing system according to the present embodiment. FIG. 2 is a flow chart showing an example of processing executed in the information processing system according to the present embodiment.

[1. Hardware configuration of the information processing system according to this embodiment]
An example of an embodiment of an information processing system according to the present disclosure will be described below. Fig. 1 is a diagram showing an example of a hardware configuration of an information processing system 1 according to the present embodiment. As shown in Fig. 1, the information processing system 1 is a computer such as a server computer or a personal computer, and includes a control unit 11, a storage unit 12, a communication unit 13, an operation unit 14, and a display unit 15. Although Fig. 1 shows a case where the information processing system 1 is configured by one computer, the information processing system 1 may be configured by multiple computers.

The control unit 11 includes at least one processor. The storage unit 12 includes a volatile memory such as a RAM and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication. The operation unit 14 is an input device such as a keyboard, a mouse, or a touch panel. The display unit 15 is a display such as a liquid crystal display or an organic EL display.

The program stored in the storage unit 12 may be supplied via the network N. Also, the program stored in a computer-readable information storage medium may be supplied via a reading unit (e.g., an optical disk drive or a memory card slot) that reads the information storage medium, or an input/output unit (e.g., a USB port) that inputs and outputs data to and from an external device.

2. Overview of the Information Processing System of the Present Embodiment
An overview of the information processing system 1 of this embodiment will be described below. Fig. 2 is a diagram showing an overview of the processing of the information processing system 1 of this embodiment.

2, the information processing system 1 of this embodiment adjusts the parameters of the lighting data L and the texture data T so as to increase the similarity between the appearance of the object O shown in the original image OI and the appearance of the object O shown in the virtual image VI generated based on the 3D model M representing the object O, the lighting data L, and the texture data T. Specifically, the similarity between the appearance of the object O shown in the original image OI and the appearance of the object O shown in the virtual image VI is evaluated by evaluating the similarity between the original image feature OF extracted from the original image OI and the virtual image feature VF extracted from the virtual image VI. In this specification, the "appearance of the object O" refers to the appearance of the object O (e.g., color, texture, etc.) other than the orientation, pose, shape, and arrangement of the object O.

According to the information processing system 1 of this embodiment, the lighting data L and texture data T that reflect the appearance of the object O shown in the original image OI can be obtained by adjusting the parameters of the lighting data L and texture data T described above. The lighting data L and texture data T obtained in this way can be used to render the 3D model M under various conditions, thereby obtaining an image in which the original image OI is transformed in various ways. In this specification, the term "transformation" is used in a broad sense. In other words, "transforming" the original image OI means obtaining an image showing the object O that is different from the original image OI (hereinafter referred to as a "transformed image") based on the original image OI showing the object O, and does not only mean performing some operation on the original image OI and obtaining a transformed image directly from the original image OI. Hereinafter, the information processing system 1 of this embodiment will be described in detail.

3. Functions realized in the information processing system of this embodiment
Hereinafter, functions realized by the information processing system 1 of the present embodiment shown in FIG. 3 will be described with reference to FIG.

FIG. 3 is a functional block diagram showing an example of functions realized by the information processing system 1 of this embodiment. As shown in FIG. 3, in this embodiment, the original image storage unit 100, the 3D model storage unit 101, the lighting data storage unit 102, the texture data storage unit 103, the feature extractor storage unit 104, the original image acquisition unit 110, the virtual image generation unit 120, the feature extraction unit 130, the parameter adjustment unit 140, and the converted image generation unit 150 are realized by the information processing system 1. The original image storage unit 100, the 3D model storage unit 101, the lighting data storage unit 102, the texture data storage unit 103, and the feature extractor storage unit 104 are realized mainly by the storage unit 12. The original image acquisition unit 110, the virtual image generation unit 120, the feature extraction unit 130, the parameter adjustment unit 140, and the converted image generation unit 150 are realized mainly by the control unit 11.

[Original image storage section]
The original image storage unit 100 stores the original image OI.

As shown in Figure 2, the original image OI is an image showing an object O.

Object O is an object having a certain shape and size. In this embodiment, as an example, object O is a dog. Note that object O may be a living thing such as a person, animal, or plant, or an inanimate object such as a stone, mountain, or building.

The original image OI shows a part or all of the object O. In this embodiment, the original image OI shows the whole of the object O, but the original image OI may show only a part of the object O. Also, in this embodiment, the original image OI shows one object O, but the original image OI may show multiple objects O. When the original image OI shows multiple objects O, the objects O may be of the same type or different types. Explaining based on the example of this embodiment, the original image OI may show birds, trees, mountains, etc. in addition to dogs.

In this embodiment, the original image OI is a photographed image of an object O placed in real space. This makes it possible to obtain a converted image showing an object O of a shape that does not exist in reality from the photographed image of the object O placed in real space. As an example, by using the information processing system 1 of this embodiment to convert a photographed image of a new screw placed in real space, a converted image showing the screw of a shape that does not exist in reality (an image showing a rusty screw, an image showing a bent screw, an image showing a screw with a crushed drive unit, etc.) can be obtained. The converted images showing various shapes of screws obtained in this way can be used, for example, as learning data for a machine learning model for detecting defective products. Note that the original image OI is not limited to a photographed image of an object O placed in real space, and may be, for example, a hand-drawn painting or a rendering image of a 3D model.

[Original image acquisition section]
The original image acquisition unit 110 acquires the original image OI. In this embodiment, the original image OI is stored in the original image storage unit 100, and therefore the original image acquisition unit 110 acquires the original image OI from the original image storage unit 100. Note that the original image OI may be stored in an external information storage medium or a computer. In this case, the original image acquisition unit 110 may acquire the original image OI from the external information storage medium or the computer.

[3D model storage unit]
The 3D model storage unit 101 stores data regarding the 3D model M.

2, the 3D model M is a model representing the object O. The 3D model M is composed of at least one polygon that represents the three-dimensional shape of the object O. In this embodiment, the 3D model M corresponds to mesh data that represents the three-dimensional shape of the object O, and corresponds to model data that does not include texture data T, which will be described later. In this embodiment, the 3D model M is a solid model, but the 3D model M may be a wireframe model or a surface model. In this embodiment, the 3D model M is created in advance by any 3DCG software such as 3D-CAD. The data for the 3D model M indicates the three-dimensional coordinates of each vertex that defines the polygon of the 3D model M. In this embodiment, the data for the 3D model M is stored in the 3D model storage unit 101, but the data for the 3D model M may be stored in an external information storage medium or computer.

The number of 3D models M is the same as the number of objects O shown in the original image OI. In this embodiment, since the original image OI shows one object O, the number of 3D models M is also one.

By the way, "representing object O" means representing the three-dimensional shape of an object of the same concept as object O shown in original image OI, and does not necessarily mean representing the three-dimensional shape of object O itself shown in original image OI. For example, if object O shown in original image OI is an individual beagle dog named "John", it is sufficient for 3D model M to represent the three-dimensional shape of a beagle dog that is an object of the same concept as object O shown in original image OI, and it does not have to represent the three-dimensional shape of the individual beagle dog named "John" itself.

It is preferable that the 3D model M represents the three-dimensional shape of the object O itself shown in the original image OI. This makes it possible to obtain a converted image in which the orientation, pose, or position of the object O shown in the original image OI is changed while maintaining the appearance and shape of the object O shown in the original image OI with high accuracy.

[Lighting data storage unit]
The illumination data storage unit 102 stores the illumination data L.

The lighting data L is data about a light source that irradiates the 3D model M. Specifically, the lighting data L includes, for example, the intensity and color of the light irradiated from the light source, as well as the position and orientation of the light source in the virtual space VS described below as parameters. Note that the parameters included in the lighting data L are not limited to these, and the lighting data L may or may not include parameters other than these. Any light source can be used as the light source, such as a point light source, a line light source, a surface light source, or a volume light source. Furthermore, when image-based lighting is used, an image that captures light information in all directions in the real world may be used as the light source.

[Texture data storage unit]
The texture data storage unit 103 stores the texture data T.

The texture data T is data about the texture set on the surface of the 3D model M. Texture is also called texture, and includes, for example, the color, glossiness, transparency, metallic feel, and unevenness of the object O. Specifically, the texture data T is image data indicating the texture set on the surface of the 3D model M. In this embodiment, the texture data T includes, for example, the diffuse reflectance (albedo), normal vector (normal), specular reflectance, metallic, roughness, gloss, anisotropy, transparency, and the like of the surface of the object O as parameters. Note that the parameters included in the texture data T are not limited to these, and the texture data T may include parameters other than these, or may not include these parameters.

[Virtual image generation unit]
The process of the virtual image generating unit 120 will be described below with reference to Fig. 4. Fig. 4 is a diagram showing how a virtual image is generated.

2 and 4, the virtual image generating unit 120 generates a virtual image VI showing the state of the virtual space VS in which the 3D model M is placed, as seen from the virtual viewpoint VV, based on the 3D model M, the lighting data L, and the texture data T. In other words, the virtual image generating unit 120 can perform a rendering process of the 3D model M with the lighting data L, the texture data T, and the virtual viewpoint VV as rendering conditions. In other words, the virtual image generating unit 120 can perform physically based rendering based on the 3D model M, the lighting data L, and the texture data T to generate a rendering image. In other words, the virtual image generating unit 120 can generate a new viewpoint image related to the object O indicated by the original image OI. In addition to the above, the rendering conditions may include, for example, a background image of the virtual space VS, and the resolution and aspect ratio of the virtual image VI to be generated. Various known rendering methods (rendering pipelines), such as forward rendering and deferred rendering, can be used to generate the virtual image VI.

The virtual space VS is a virtual three-dimensional space. Three mutually orthogonal coordinate axes are set in the virtual space VS. These three coordinate axes are the coordinate axes of the world coordinate system. The origin may be any position, and positions in the virtual space VS are expressed in three-dimensional coordinates. Note that the virtual image generating unit 120 may set a background image in the virtual space VS. The background image is, for example, an image showing scenery such as trees, mountains, and the sky.

The virtual viewpoint VV is also called a virtual camera, and defines the coordinate axes of the view coordinate system. The virtual image generator 120 sets the parameters of the virtual viewpoint VV (position, angle of view, magnification, etc.).

The detailed processing of the virtual image generating unit 120 will be described below with reference to FIG. 5. FIG. 5 is a diagram showing how virtual images viewed from each of a plurality of virtual viewpoints are generated.

More specifically, as shown in FIG. 5, the virtual image generating unit 120 generates a plurality of virtual images VI1, VI2, and VI3, each of which corresponds to a plurality of different virtual viewpoints VV1, VV2, and VV3, and shows the state of the virtual space VS as seen from the respective virtual viewpoints. Note that, although the number of virtual viewpoints is three in this embodiment, the number of virtual viewpoints may be less than this, or may be more than this.

As described above, the information processing system 1 of this embodiment generates a virtual image VI that shows the state of the virtual space VS as seen from each of the multiple virtual viewpoints VV1, VV2, and VV3. That is, the information processing system 1 of this embodiment can reflect the appearance of the object O shown in the original image OI in the appearance of the object O as seen from each of the multiple virtual viewpoints VV1, VV2, and VV3. This makes it possible to favorably maintain the appearance of the object O shown in the original image OI in a converted image in which the orientation, pose, shape, and position of the object O shown in the original image OI are changed, as will be described later.

[Feature Extractor Storage Unit]
The feature extractor storage unit 104 stores a feature extractor used in the processing in the feature extraction unit 130. Specifically, the feature extractor storage unit 104 stores the program and parameters of the feature extractor.

The feature extractor is a trained machine learning model that extracts image features. The feature extractor is used to extract the first intermediate feature amount and the second intermediate feature amount described below. In this embodiment, a trained CNN (Convolutional Neural Network) without the fully connected layer is used as the feature extractor. Note that, in addition to a CNN without the fully connected layer, a trained, publicly known machine learning model can also be used as the feature extractor.

[Feature extraction section]
The feature extraction unit 130 extracts original image feature values OF from the original image OI, and extracts virtual image feature values VF from the virtual image VI. In this embodiment, the feature extraction unit 130 extracts virtual image feature values VF corresponding to each of the multiple virtual images VI1, VI2, and VI3.

Specifically, the feature extraction unit 130 extracts a first intermediate feature from the original image OI, and extracts an original image feature OF from the first intermediate feature. The feature extraction unit 130 also extracts a second intermediate feature from the virtual image VI, and extracts a virtual image feature VF from the second intermediate feature. The feature extraction unit 130 extracts the first intermediate feature and the second intermediate feature using the feature extractor described above.

The first intermediate feature includes information about the shape and arrangement of the object O in the original image OI. The second intermediate feature includes information about the shape and arrangement of the object O in the virtual image VI. In this embodiment, a trained CNN excluding the fully connected layer is used as a feature extractor, so specifically, the first intermediate feature and the second intermediate feature are each a feature map output from the convolutional layer of the CNN. Here, the number of feature maps is the same as the number of filters (kernels).

Here, the original image feature OF is a feature indicating the frequency distribution of each color of the original image OI. The virtual image feature VF is a feature indicating the frequency distribution of each color of the virtual image VI. In other words, the original image feature OF is a feature that does not include information about the shape and arrangement of the object O in the original image OI. The virtual image feature VF is a feature that does not include information about the shape and arrangement of the object O in the virtual image VI. In this embodiment, the original image feature OF and the virtual image feature VF may each be a vector expression equivalent to a feature that can be output from a trained feature extractor such as a CNN. In this embodiment, specifically, the original image feature OF and the virtual image feature VF are each a Gram matrix obtained by calculating the correlation between feature maps output from the convolution layer of the CNN. This Gram matrix is specifically obtained by calculating the inner product of feature maps output from the convolution layer of the CNN. The image feature may also be a feature that indicates the frequency distribution of the texture pattern.

[Parameter adjustment section]
The parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF increases. That is, the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the feature indicating the frequency distribution of each color of the original image OI and the feature indicating the frequency distribution of each color of the virtual image VI increases. It can also be said that the parameter adjustment unit 140 adjusts the parameters of the illumination data L and the parameters of the texture data T so that the similarity between the color scheme of the original image OI and the color scheme of the virtual image VI increases. As a result, the information processing system 1 of this embodiment can accurately reflect the appearance of the object O shown in the original image OI in the appearance of the object O shown in the converted image even if the arrangement of the object O in the original image OI is different from the arrangement of the object O in the virtual image VI.

Specifically, the parameter adjustment unit 140 calculates the loss based on the original image feature amount OF and the virtual image feature amount VF, and adjusts the parameters of the illumination data L and the parameters of the texture data T so as to reduce the loss. The calculation of the loss itself can use a known calculation method such as squared error or cross entropy. Furthermore, the adjustment of the parameters of the illumination data L and the parameters of the texture data T can use a known parameter adjustment method such as gradient descent or backpropagation. Note that in the parameter adjustment unit 140, the parameters related to the feature extractor (weighting coefficients, bias, etc.) are not adjusted but remain fixed.

In this embodiment, the parameter adjustment unit 140 adjusts the parameters of the lighting data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to each of the multiple virtual images VI1, VI2, and VI3 is increased. Specifically, first, the parameter adjustment unit 140 adjusts the parameters of the lighting data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI1 is increased. Next, the parameter adjustment unit 140 adjusts the parameters of the lighting data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI2 is increased. Finally, the parameter adjustment unit 140 adjusts the parameters of the lighting data L and the parameters of the texture data T so that the similarity between the original image feature OF and the virtual image feature VF corresponding to the virtual image VI3 is increased.

Here, it is preferable that the orientation of the object O indicated by the original image OI coincides with the orientation of the object O indicated by the virtual image VI1 corresponding to the virtual image feature VF that is first subjected to processing by the parameter adjustment unit 140. This reduces the processing load associated with adjusting the parameters of the illumination data L and the parameters of the texture data T.

When a background image is set in the virtual space VS, the parameter adjustment unit 140 may adjust the parameters of the background image in addition to the parameters of the lighting data L and the parameters of the texture data T.

The feature extraction unit 130 and parameter adjustment unit 140 may be configured, for example, according to the description in the reference (Gatys, L.A., Ecker, A.S., Bethge, M.: Image style transfer using convolutional neural networks. In: CVPR. (2016)).

[Converted image generation unit]
The converted image generating unit 150 generates a converted image based on the 3D model M and the illumination data L and texture data T whose parameters have been adjusted by the parameter adjusting unit 140. The converted image can also be said to be a variation of the original image OI. The content of the processing executed by the converted image generating unit 150 is similar to that of the virtual image generating unit 120.

As described above, the information processing system 1 of this embodiment generates a converted image using the lighting data L and texture data T whose parameters have been adjusted. When generating this converted image, various conditions (position, pose, shape, and orientation of the 3D model M, position, magnification, and angle of view of the virtual viewpoint VV, etc.) can be changed to generate a variety of converted images. Furthermore, when generating a converted image, the parameters of the lighting data L and the parameters of the texture data T can be further adjusted to change the position and brightness of the lighting and the texture set on the surface of the object O, thereby generating even more diverse converted images.

In addition, according to the information processing system 1 of this embodiment, it becomes easy to generate a large number of various converted images as described above. The large number of converted images obtained in this manner can be used, for example, as learning data for a machine learning model for image recognition.

Note that the use of the information processing system 1 of this embodiment is not limited to generating the converted image described above. For example, the information processing system 1 of this embodiment can also be used to create game objects that appear in a video game. That is, (1) an original image OI showing a game character as the object O is created. (2) A virtual image VI is generated based on a 3D model M representing the game character, lighting data L, and texture data T. (3) By adjusting the parameters as described above, a game object that reflects the appearance of the original image OI is obtained.

4. Processing Executed in the Information Processing System of the Present Embodiment
Finally, the process executed in the information processing system 1 of this embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 and Fig. 7 are flow charts showing an example of the process executed in the information processing system 1 of this embodiment. The process shown in Fig. 6 and Fig. 7 is executed by the control unit 11 operating according to a program stored in the storage unit 12. The process described below is an example of the process executed by the functional blocks shown in Fig. 3.

As shown in FIG. 6, first, the control unit 11 acquires an original image OI from the storage unit 12 (S100). The control unit 11 extracts a first intermediate feature from the acquired original image OI (S101). The control unit 11 further extracts an original image feature OF from the first intermediate feature, and stores the extracted original image feature OF in the storage unit 12 (S102).

Moving on to FIG. 7, the control unit 11 sets a virtual viewpoint VV (S103). Specifically, the control unit 11 sets parameters such as the position, angle of view, and magnification of the virtual viewpoint VV. Next, the control unit 11 acquires a 3D model M, lighting data L, and texture data T from the storage unit 12 (S104), and generates a virtual image VI based on these (S105).

The control unit 11 extracts second intermediate features from the generated virtual image VI (S106). The storage unit 12 further extracts virtual image features VF from the second intermediate features and stores the extracted virtual image features VF in the storage unit 12 (S107). The control unit 11 adjusts the parameters of the lighting data L and the parameters of the texture data T so that the similarity between the original image features OF stored in the storage unit 12 and the virtual image features VF increases (S108).

When the control unit 11 determines that the similarity between the original image feature amount OF and the virtual image feature amount VF does not satisfy a predetermined condition and that the adjustment of the parameters of the illumination data L and the parameters of the texture data T should be repeated (S109; N), the control unit 11 executes the processes from S105 to S108 again. Specifically, the control unit 11 repeats the adjustment of the parameters of the illumination data L and the parameters of the texture data T until the loss calculated based on the original image feature amount OF and the virtual image feature amount VF becomes less than a predetermined threshold value.

On the other hand, when the control unit 11 determines that the similarity between the original image feature OF and the virtual image feature VF satisfies a predetermined condition and that the parameter adjustment is to be terminated (S109; Y), it then determines whether or not to change the virtual viewpoint VV and generate a virtual image VI (S110). When the control unit 11 determines that the virtual viewpoint VV is changed and a virtual image VI is to be generated (S110; N), it executes the processes from S103 to S109 again. Specifically, the control unit 11 repeats the processes from S103 to S109 for each of a predetermined number of virtual viewpoints VV. On the other hand, when the control unit 11 determines that the virtual viewpoint VV is not changed and a virtual image VI is not to be generated (S110; Y), it terminates this process.

[5. Notes]
For example, the information processing system according to the present disclosure may be configured as follows.

(1)
An original image acquisition means for acquiring an original image showing an object;
a virtual image generating means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data;
a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image;
a parameter adjusting means for adjusting parameters of the illumination data and parameters of the texture data so as to increase a similarity between the original image feature amount and the virtual image feature amount;
An information processing system having the above configuration.

(2)
The original image is a photographed image of the object disposed in real space.
An information processing system according to (1).

(3)
the original image feature amount indicates a frequency distribution of each color of the original image,
the virtual image feature amount indicates a frequency distribution of each color of the virtual image,
the feature extraction means extracts, from the original image, a first intermediate feature amount including information regarding a shape and an arrangement of the object in the original image, and extracts the original image feature amount from the first intermediate feature amount, and also extracts, from the virtual image, a second intermediate feature amount including information regarding a shape and an arrangement of the object in the virtual image, and extracts the virtual image feature amount from the second intermediate feature amount.
An information processing system according to (1) or (2).

(4)
the virtual image generating means generates a plurality of virtual images each corresponding to a plurality of different virtual viewpoints and showing a state of the virtual space as seen from the corresponding virtual viewpoint;
the feature extraction means extracts, from each of the plurality of virtual images, a virtual image feature corresponding to the virtual image;
the parameter adjustment means adjusts the parameters of the illumination data and the parameters of the texture data so as to increase a similarity between the original image feature amount and a virtual image feature amount corresponding to each of the plurality of virtual images.
An information processing system according to any one of (1) to (3).

(5)
a converted image generating means for generating a converted image, which is an image showing the object and is different from the original image, based on the 3D model and the lighting data and the texture data whose parameters have been adjusted by the parameter adjusting means;
An information processing system according to any one of (1) to (4).

According to the information processing system 1 of this embodiment described above, the image conversion mode can be expanded.

1 Information processing system, N Network, 11 Control unit, 12 Memory unit, 13 Communication unit, 14 Operation unit, 15 Display unit, 100 Original image memory unit, 101 3D model memory unit, 102 Lighting data memory unit, 103 Texture data memory unit, 104 Feature extractor memory unit, 110 Original image acquisition unit, 120 Virtual image generation unit, 130 Feature extraction unit, 140 Parameter adjustment unit, 150 Transformed image generation unit, OI Original image, O Object, M 3D model, L Lighting data, T Texture data, VI, VI1, VI2, VI3 Virtual image, OF Original image feature amount, VF Virtual image feature amount, VV, VV1, VV2, VV3 Virtual viewpoint, VS Virtual space.

Claims (7)

An original image acquisition means for acquiring an original image showing an object;
a virtual image generating means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data;
a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image;
a parameter adjustment means for adjusting a parameter of the illumination data and a parameter of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount;
An information processing system having the above configuration. The original image is a photographed image of the object disposed in real space.
The information processing system according to claim 1 . the original image feature amount indicates a frequency distribution of each color of the original image,
the virtual image feature amount indicates a frequency distribution of each color of the virtual image,
the feature extraction means extracts, from the original image, a first intermediate feature amount including information regarding a shape and an arrangement of the object in the original image, and extracts the original image feature amount from the first intermediate feature amount, and also extracts, from the virtual image, a second intermediate feature amount including information regarding a shape and an arrangement of the object in the virtual image, and extracts the virtual image feature amount from the second intermediate feature amount.
3. The information processing system according to claim 1 or 2. the virtual image generating means generates a plurality of virtual images each corresponding to a plurality of different virtual viewpoints and showing a state of the virtual space as seen from the corresponding virtual viewpoint;
the feature extraction means extracts, from each of the plurality of virtual images, a virtual image feature corresponding to the virtual image;
the parameter adjustment means adjusts the parameters of the illumination data and the parameters of the texture data so as to increase a similarity between the original image feature amount and a virtual image feature amount corresponding to each of the plurality of virtual images.
3. The information processing system according to claim 1 or 2. a converted image generating means for generating a converted image, which is an image showing the object and is different from the original image, based on the 3D model and the lighting data and the texture data whose parameters have been adjusted by the parameter adjusting means;
3. The information processing system according to claim 1 or 2. An original image acquisition step of acquiring an original image showing an object;
a virtual image generating step of generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data;
a feature extraction step of extracting an original image feature from the original image and a virtual image feature from the virtual image;
a parameter adjustment step of adjusting parameters of the illumination data and parameters of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount;
An information processing method comprising the steps of: An original image acquisition means for acquiring an original image showing an object;
a virtual image generating means for generating a virtual image showing an appearance of a virtual space in which the 3D model is placed, as viewed from a virtual viewpoint, based on a 3D model representing the object, lighting data, and texture data;
a feature extraction means for extracting an original image feature from the original image and a virtual image feature from the virtual image;
a parameter adjusting means for adjusting a parameter of the illumination data and a parameter of the texture data based on a loss indicating a degree of similarity between the original image feature amount and the virtual image feature amount;
A program that makes a computer function as a

Images