JP7396202B2 - Generation program, generation method, and information processing device - Google Patents

Description

The present invention relates to a generation program, a generation method, and an information processing device.

For example, parts and products are inspected and shapes are determined in manufacturing, sorting, and other fields. Processing such as inspection and determination may be difficult to perform manually because the number and types of objects to be processed may be enormous. Therefore, attempts have been made to process it by image analysis.

Further, when photographing a subject with a photographing device, it may not be possible to photograph the subject from a preferred direction due to, for example, an obstacle. For example, in such cases, an image reprojection technique is used to convert an image of a subject photographed by a photographing device into an image of the subject viewed from a preferred orientation for use. As techniques for image reprojection, techniques such as texture mapping and projected texture mapping are known, for example.

Furthermore, techniques related to image conversion are known (for example, Patent Document 1).

Japanese Patent Application Publication No. 11-025291

However, when reprojecting an image using techniques such as texture mapping and projection texture mapping, a three-dimensional shape model of the subject is generated, which may increase the calculation load. Therefore, when processing a large number of captured images, such as in the case of inspection and judgment of parts and products, for example, it may be difficult to process using these techniques.

In one aspect, an object of the present invention is to reduce the load required for processing to generate an image in which the subject is shown in a different direction from a captured image in which the subject is shown.

A generation program according to one aspect of the present invention includes the optical center of the photographing device used to photograph the photographed image, in which each pixel is associated with depth information indicating the depth of the subject, and the photographic projection onto which the photographed image is projected. A plane, a viewpoint, and a viewpoint projection surface onto which a viewpoint image of the subject is projected from the viewpoint are arranged in virtual space, and the viewpoint image placed on the viewpoint projection surface is extended from the viewpoint toward the processing target pixel. Based on the intersection position of the line of sight and the projection line extending from the optical center toward the pixels of the photographed image placed on the photographic projection plane, and the depth of the subject indicated by the depth information corresponding to the pixels of the photographed image, A computer is caused to execute a process of acquiring a pixel value corresponding to a processing target pixel of a viewpoint image from among the pixels of the photographed image, and generating a viewpoint image using the acquired pixel value for the processing target pixel.

It is possible to reduce the load required for processing to generate an image of the subject in a different orientation from a photographed image of the subject.

1 is a diagram illustrating an exemplary imaging system in which embodiments may be utilized; FIG. FIG. 1 is a diagram illustrating a block configuration of an information processing device according to an embodiment. FIG. 3 is a diagram illustrating a flow of generating a viewpoint image from a photographed image according to an embodiment. FIG. 3 is a diagram illustrating a flow of generating a viewpoint image from a photographed image according to an embodiment. FIG. 3 is a diagram illustrating a flow of generating a viewpoint image from a photographed image according to an embodiment. FIG. 3 is a diagram illustrating a blind spot area according to an embodiment. FIG. 3 is a diagram illustrating a shielding area according to an embodiment. FIG. 3 is a diagram illustrating an operational flow of viewpoint image generation processing according to the embodiment. FIG. 3 is a diagram illustrating an operation flow of a process for specifying a projection line corresponding to a line of sight according to an embodiment. FIG. 3 is a diagram illustrating the flow of a process for specifying a projection line corresponding to a line of sight according to an embodiment. FIG. 3 is a diagram illustrating a photographed image and a viewpoint image according to an embodiment. FIG. 3 is a diagram illustrating generation of a viewpoint image using captured images captured from a plurality of positions according to an embodiment. FIG. 3 is a diagram illustrating generation of a viewpoint image using captured images captured from a plurality of positions according to an embodiment. FIG. 3 is a diagram illustrating an operation flow of a viewpoint image generation process based on captured images captured from two different positions according to the embodiment. 1 is a diagram illustrating a hardware configuration of a computer for realizing an information processing apparatus according to an embodiment.

Hereinafter, some embodiments of the present invention will be described with reference to the drawings. Note that the same reference numerals are given to corresponding elements in a plurality of drawings.

FIG. 1 is a diagram illustrating an exemplary imaging system 100 in which embodiments may be utilized. The photographing system 100 includes, for example, an information processing device 101, a photographing device 102, an installation stand 105, and the like. The installation table 105 is, for example, a table on which the subject 110 is placed, and may be, for example, a production line such as a belt conveyor. An object to be photographed 106 such as a part, a product, a manufactured product, and a molded object is installed on the installation stand 105, for example. The photographing device 102 photographs an object to be photographed 106 installed on an installation stand 105 .

The information processing device 101 is, for example, communicatively connected to the photographing device 102 by wire or wirelessly. The information processing device 101 may be, for example, a personal computer (PC), a mobile PC, a tablet terminal, or a computer equipped with an image processing function such as a server computer connected to the photographing device 102 via a network.

The photographing device 102 may be, for example, a photographing device such as an RGB-D camera or a stereo camera that can acquire a photographed image and depth information corresponding to the pixels of the photographed image. In one example, the imaging device 102 captures a depth image. The photographing device 102 photographs, for example, an image in which the subject 110 appears. The subject 110 may be, for example, an object that appears in an image. In the example of FIG. 1, the installation stand 105 and the object to be photographed 106 may be shown as the object 110 in the photographed image. The object to be photographed 106 is, for example, a part, a product, a manufactured product, a molded object, or the like. The photographing device 102 transmits, for example, a photographed image and a depth image showing the photographed subject 110 to the information processing device 101.

Note that although FIG. 1 illustrates the use of the embodiment in the field, for example, for inspection and determination of parts and products in manufacturing, sorting, and other fields, the embodiment is not limited thereto. do not have. The embodiment can be used in various situations in which processing is performed to convert a photographed image in which the subject 110 is seen into an image of the subject 110 viewed from a different direction.

Note that, as shown in FIG. 1, when a photograph is taken using the photographing device 102 such as an RGB-D camera, a depth image corresponding to the photographed image is also obtained. Therefore, a three-dimensional shape model can be generated using the depth information of the depth image. Then, by attaching the texture of the captured image to the generated three-dimensional shape model and rotating it, captured images of the subject 110 captured from various viewpoints can be obtained. However, since the three-dimensional shape model is generated by processing a large number of polygons, the processing load is high, and it may be difficult to use it in applications that require high-speed processing or processing a large number of captured images. Therefore, it is desired to provide a further technique that can reduce the processing load required for image conversion for generating an image in which the subject is viewed in a different direction from a photographed image in which the subject appears.

In the embodiment described below, pixels of a photographed image and pixels of an image of the subject 110 from another viewpoint, which is generated based on the photographed image, are identified by ray tracing of a projection line and a line of sight. Note that an image of the subject 110 that is generated from a different viewpoint based on the photographed image may be referred to as a viewpoint image below. Further, the projection line is represented by, for example, a straight line extending from the optical center of the photographing device used to photograph the photographed image toward the pixel of the photographed image. The line of sight is represented, for example, by a straight line from the viewpoint to the pixel of the viewpoint image. According to the embodiment, since a viewpoint image can be generated without generating a three-dimensional shape model of the subject 110, it is possible to reduce the processing load required to generate a viewpoint image.

Note that in the embodiment, the photographed image used to generate the viewpoint image may be an image of diffuse reflected light in which a specular reflected light component is removed by performing preprocessing in advance. The specularly reflected light component of the photographed image is a component that is affected by the light source, so it can change when the viewpoint changes. Therefore, for example, if the color of the subject 110 in the viewpoint image is to be used in subsequent processing, it is preferable to remove it in advance from the photographed image used to generate the viewpoint image. However, embodiments are not limited thereto. For example, when using a viewpoint image to inspect the appearance of the outline and shape of the subject 110, it is possible to extract the edge shape even if the specularly reflected light component is included. No removal may be performed. Hereinafter, embodiments will be described in more detail.

FIG. 2 is a diagram illustrating a block configuration of the information processing apparatus 101 according to the embodiment. The information processing device 101 includes, for example, a control unit 201, a storage unit 202, a communication unit 203, and a display device 204. The control unit 201 may control each unit of the information processing device 101. The control unit 201 includes, for example, a placement unit 211, an acquisition unit 212, a generation unit 213, and may also include other functional units. The storage unit 202 stores, for example, a program for executing an operation flow described later, information to be referred to when executing the program, and the like. The communication unit 203 communicates with other devices according to instructions from the control unit 201, for example. The communication unit 203 may, for example, communicate with the photographing device 102 according to instructions from the control unit 201 and obtain a photographed image of the subject 110 from the photographing device 102. The display device 204 displays information on a display screen according to instructions from the control unit 201, for example. The details of each of these units and the information stored in the storage unit 202 will be described later.

3 to 5 are diagrams illustrating the flow of generating a viewpoint image from a photographed image according to the embodiment. In the following description, when a drawing shows the arrangement of objects in a three-dimensional virtual space, an arrow indicating the direction of the axis is attached. For example, in FIG. 3, the Z axis is in the vertical direction of the drawing, the X axis is in the horizontal direction of the drawing, and the Y axis is in the depth direction of the drawing. Note that the direction of the axis of the virtual space according to the embodiment is not limited to this, and the axis may be taken in any direction as long as the coordinates in the space can be specified.

As shown in FIG. 3, in the process of generating a viewpoint image from a photographed image, the control unit 201 places the optical center 301 of the photographing device 102 in virtual space. Further, the control unit 201 arranges a photographic projection plane 303 as a projection plane of a photographed image perpendicularly to the optical axis at a position of a focal length in front of the optical axis 302 from the optical center 301 . Note that a captured image and a depth image captured by the imaging device 102 can be arranged to be superimposed on the projected image projected onto the imaging projection surface 303. In this case, the image centers of the photographed image and the depth image are on the optical axis 302, for example. Further, the control unit 201 arranges a three-dimensional plane 305 parallel to the imaging projection plane 303. In addition, in FIG. 3, a broken line extending from the optical center 301 represents the photographing field of view.

Furthermore, the depth information of the depth image placed on the photographic projection plane 303 represents the position of the subject 110. In FIG. 3, the positions of the installation table 105 and the object to be photographed 106 as the object 110 are indicated by broken lines based on the depth information for each pixel of the depth image. The subject 110 is, for example, at a depth position indicated by the depth information of the pixel 307 on a straight line extending from the optical center 301 toward the pixel 307 of the depth image arranged on the imaging projection plane 303. Note that an arrow extending a straight line toward a pixel of a photographed image and a depth image arranged on the photographic projection plane 303 to a depth indicated by the depth information of the pixel is hereinafter referred to as a projection line 308, for example.

Subsequently, the control unit 201 places the viewpoint 310 at an arbitrary position in the half space on the side of the optical center 301 with respect to the three-dimensional plane 305. The control unit 201 arranges a viewpoint projection surface 311 in front of the viewpoint 310 as a projection surface for perspective projection onto the viewpoint 310 . Note that in FIG. 3, a broken line extending from the viewpoint 310 represents the viewpoint field of view.

In this case, in the embodiment, the control unit 201 executes processing to generate a viewpoint image of the subject 110 projected onto the viewpoint projection plane 311 from the position of the viewpoint 310. Note that, hereinafter, a straight line extending from the viewpoint 310 toward a certain pixel 312 of a viewpoint image arranged on the viewpoint projection plane 311 is referred to as a line of sight 313, for example.

Next, the control unit 201 determines, from among the projection lines 308 that intersect with the line of sight 313, the depth of the intersection position of the line of sight 313 and the projection line 308, and the depth information of the pixels through which the projection line 308 passes on the imaging projection plane 303. The projection line 308 with the minimum depth difference from the depth of the subject 110 is identified. For example, three projection lines 308 are shown in FIG. 4, from projection line 308a to projection line 308c. Here, the projection line 308c is excluded because it does not intersect the line of sight 313. On the other hand, the projection line 308a and the projection line 308b intersect the line of sight 313. Furthermore, the projection line 308a intersects the line of sight 313 at the position of the point 401, and the depth information of the pixel 412 of the depth image through which the projection line 308a passes on the imaging projection plane 303 indicates the depth of the subject 110 at the position of the point 402. It shows. For the projection line 308a, the depth at the intersection with the line of sight 313 (point 401 in FIG. 4) and the distance to the subject (point 402 in FIG. 4) indicated by the depth information of the pixel 412 through which the projection line 308a passes on the imaging projection plane 303 are determined. The difference in depth is large. On the other hand, for the projection line 308b, the depth at the intersection with the line of sight 313 (point 403 in FIG. 4) is almost the same as the depth of the subject 110 indicated by the depth information of the pixel 413 through which the projection line 308b passes on the imaging projection plane 303. depth, and the difference in depth is small. Therefore, the control unit 201 may specify the projection line 308b as the projection line 308 with the minimum depth difference.

Note that, for example, it is assumed that the pixels of the photographed image on the photographic projection plane 303 through which the projection line 308 passes and the pixels of the viewpoint image on the viewpoint projection plane 311 through which the line of sight 313 passes reflect the same area of the subject. In this case, the depth at the intersection of the projection line 308 and the line of sight 313 approximately matches the depth of the subject 110 indicated by the depth information of the pixel of the depth image on the imaging projection plane 303 through which the projection line 308 passes. Therefore, by specifying the projection line 308 in which the depth difference is the minimum and is within a predetermined error range (for example, 5 mm or less), the control unit 201 can detect the same subject as the pixel on the viewpoint projection plane 311 through which the line of sight 313 passes. 110 can be specified on the imaging projection plane 303.

Note that the depth difference is, for example, the depth between the depth from the three-dimensional plane 305 parallel to the imaging projection plane 303 to the intersection position of the line of sight 313 and the projection line 308, and the depth from the three-dimensional plane 305 to the subject 110. It can be determined from the difference. The depth from the three-dimensional plane 305 to the subject 110 is determined by, for example, when the three-dimensional plane 305 is placed on the photographic projection plane 303, the focal length is calculated from the depth indicated by the depth information of the pixel of the depth image on the photographic projection plane 303. It can be found by subtracting it.

Then, for example, as shown in FIG. 5, the control unit 201 changes the pixel value of the pixel 413 on the imaging projection plane 303 through which the specified projection line 308 passes, to the pixel value of pixel 312 on the viewpoint projection plane 311 through which the line of sight 313 passes. Can be used as a value.

Furthermore, for all pixels of the viewpoint image on the viewpoint projection surface 311, if there is a pixel corresponding to the photographed image on the photographic projection surface 303, the control unit 201 can obtain a pixel value from that pixel. . Thereby, the control unit 201 can generate a viewpoint image seen from the position of the viewpoint 310.

Further, in the process of generating a viewpoint image from a photographed image described above, for example, it is possible to generate a viewpoint image without generating a three-dimensional shape model of the subject 110. Therefore, a viewpoint image can be generated with a reduced processing load, and a viewpoint image can be obtained in situations where high-speed processing is required and in situations where a large amount of images are to be processed.

Next, identification of a blind spot area in a viewpoint image according to an embodiment will be described.

(Identification of blind spot area)
For example, in a captured image, an area that was hidden by another subject 110 in the foreground may become visible as a result of moving the viewpoint. In this case, an area on the viewpoint image that is not visible in the captured image is called a blind spot area, for example. The blind spot area represents an area of the viewpoint image that cannot be drawn because the area is a blind spot in the captured image and no image information has been captured.

FIG. 6 is a diagram illustrating identification of a blind spot area according to the embodiment. For example, as shown in FIG. 6A, the projection line 308 and the line of sight 313 may intersect at a position deeper than the depth indicated by the depth information of the pixel on the imaging projection plane 303 through which the projection line 308 passes. In this case, the area of the subject 110 that is supposed to be captured in the pixel through which the line of sight 313 passes is a blind spot in the captured image, indicating that image information for that part cannot be acquired.

For example, in the example of FIG. 6A, the line of sight 313a points to the position of point 603 on the subject 110, and the viewpoint image obtained when shooting from the viewpoint 310 shows the subject 110 at the position of point 603. Take a picture. However, in the photographed image taken from the optical center 301, this point 603 is hidden behind the subject 110 and becomes a blind spot. Therefore, the captured image does not include image information at the position of point 603. In this case, an image of a blind spot area such as the position of point 603 does not need to be generated in the viewpoint image. Further, in this case, the control unit 201 may set the blind spot area in a viewpoint image in a predetermined color representing invalidity, so that the blind spot area is displayed in an identifiable manner. Thereby, the user viewing the viewpoint image can know that something that is not shown in the captured image is shown in the blind spot area.

Note that in identifying the blind spot area, the control unit 201 first determines the depth of the intersection of the line of sight 313 and the projection line 308 and the depth of the subject 110 indicated by the depth information of the pixel on the imaging projection plane 303 through which the projection line 308 passes. A projection line 308 with the minimum depth difference between the two points is identified. In the example of FIG. 6, it is assumed that a projection line 308 pointing to a point 601 has been identified. In this case, the control unit 201 determines the intersection position of the line of sight 313 and the projection line 308 (for example, point 602) and the position of the subject 110 indicated by the depth information of the pixel through which the projection line 308 passes on the imaging projection plane 303 (for example, It is determined whether or not the point 601) satisfies a predetermined condition and is far away from the point 601). Then, when the intersection position (for example, point 602) and the position of the subject 110 indicated by the depth information (for example, point 601) meet a predetermined condition and are separated, the line of sight 313 passes on the viewpoint projection plane 311. A pixel of the viewpoint image may be determined to be a blind spot area.

The predetermined conditions include, for example, the intersection position of the line of sight 313 and the projection line 308 (for example, point 602), and the position of the subject 110 indicated by the depth information of the pixel through which the projection line 308 passes on the photographic projection plane 303 (for example, the point 602). 601) is greater than or equal to a predetermined distance. In another example, the predetermined condition is the depth of the intersection position (for example, point 602) of the line of sight 313 and the projection line 308, and the depth indicated by the depth information of the pixel through which the projection line 308 passes on the imaging projection plane 303 (for example, , depth of point 601) may be a predetermined length or more. Note that FIG. 6(b) is a partially enlarged view of FIG. 6(a). As shown in FIG. 6(b), the depth difference represents a component of the distance difference in the depth direction. When the angle θ that the projection line 308 makes with a plane parallel to the imaging projection plane 303 approaches an acute angle, the depth difference becomes small. Therefore, when determining the blind spot area with higher accuracy, it is preferable to use the distance difference.

Next, identification of a shielding area in a viewpoint image according to an embodiment will be described.

(Identification of shielded area)
For example, as a result of moving the viewpoint, an area that is visible in the photographed image may be obscured by another subject 110 in the foreground. In this way, a region of the viewpoint image that is visible in the photographed image but becomes invisible in the viewpoint image is called, for example, a shielding region.

FIG. 7 is a diagram illustrating the shielding area according to the embodiment. For example, as shown in FIG. 7, one line of sight 313 may intersect with a plurality of projection lines 308 on the subject 110. In FIG. 7, the line of sight 313 intersects the projection line 308a at a point 701 on the subject 110, and intersects the projection line 308b at a point 702 on the subject 110.

However, the position of the point 701 is hidden behind the subject 110 and cannot be seen from the viewpoint 310. Therefore, it is correct to use the pixel values of the pixels of the photographed image on the photographic projection plane 303 through which the projection line 308b passes as the values of the pixels of the viewpoint image on the viewpoint projection plane 311 through which the line of sight 313 passes. In this way, the line of sight 313 may intersect with a plurality of projection lines 308 on the subject 110. In this case, the control unit 201 may specify the projection line 308 that corresponds to the line of sight 313 so that, for example, the projection line 308 that intersects on the side closer to the viewpoint 310 is selected. Thereby, a viewpoint image viewed from the position of viewpoint 310 can be correctly generated.

For example, as described above, a viewpoint image can be generated from a captured image. The operation flow of the viewpoint image generation process according to the embodiment will be described below.

FIG. 8 is a diagram illustrating an operational flow of viewpoint image generation processing according to the embodiment. For example, the control unit 201 may start the operation flow shown in FIG. 8 when an instruction to perform viewpoint conversion on a photographed image having depth information is input. The photographed image having depth information may be, for example, a set of photographed image and depth image photographed with an RGB-D camera. In another example, the photographed image having depth information may be an image photographed with a stereo camera. Hereinafter, an example will be described in which a photographed image having depth information is a set of a photographed image and a depth image.

In step 801 (hereinafter, step will be abbreviated as "S", for example, S801), the control unit 201 arranges a perspective projection model of the imaging device 102 on the virtual space. For example, as described in FIG. 3, the control unit 201 acquires the camera parameters of the photographing device 102 used to photograph the photographed image having depth information, and displays the optical center 301, optical axis 302, photographing image, etc. on the virtual space. A perspective projection model such as the projection plane 303 may be arranged. The camera parameters may be stored in the storage unit 202 in advance, for example.

In S802, the control unit 201 arranges the photographed image and the depth image on the photographic projection plane 303, for example.

In S803, the control unit 201 arranges, for example, a viewpoint 310 and a viewpoint projection plane 311 for projecting a viewpoint image on the virtual space, and also arranges a three-dimensional plane 305 parallel to the imaging projection plane 303. In one example, the three-dimensional plane 305 may be arranged to overlap the imaging projection plane 303.

In S804, the control unit 201 selects one pixel from among the pixels of the viewpoint image generated on the viewpoint projection plane 311 as a processing target pixel.

In S805, the control unit 201 identifies the projection line 308 corresponding to the line of sight 313 passing through the selected processing target pixel of the viewpoint image from the viewpoint 310. For example, as described with reference to FIGS. 4 and 5, the control unit 201 determines the depth of the subject 110 indicated by the depth information of the pixel through which the projection line 308 passes, and the depth of the intersection position of the projection line 308 and the line of sight 313. The projection line 308 that has the minimum depth difference between the projection line and the projection line 308 may be identified. Note that the operational flow of an exemplary process for specifying the projection line 308 will be described later with reference to FIG. 9 .

In S806, the control unit 201 determines whether the processing target pixel of the viewpoint image is in a blind spot area, as described with reference to FIG. For example, the control unit 201 determines that the position indicated by the depth of the pixel of the photographed image corresponding to the projection line 308 where the depth difference is the minimum, and the intersection position of the projection line 308 and the line of sight 313 where the depth difference is the minimum are predetermined. If the conditions are met and the pixel is far away, it may be determined that the pixel to be processed is in the blind spot area.

If the pixel to be processed is a blind spot area in S806 (YES in S806), the flow advances to S809. In S809, the control unit 201 sets the processing target pixel in the viewpoint image as a blind spot area, and the flow advances to S810.

On the other hand, if the pixel to be processed is not a blind spot area in S806 (NO in S806), the flow advances to S807. In S807, the control unit 201 acquires pixel values from pixels of the photographed image on the photographic projection plane 303 through which the projection line 308 through which the identified depth difference is the minimum passes.

In S808, the control unit 201 sets the acquired pixel value as the processing target pixel of the viewpoint image, and the flow advances to S810.

In S810, the control unit 201 determines whether all pixels of the viewpoint image have been processed. If the viewpoint image still contains unprocessed pixels, the flow returns to S804, selects the unprocessed pixels as processing target pixels, and repeats the process. On the other hand, if processing of all pixels of the viewpoint image has been completed in S810, the flow advances to S811.

In S811, the control unit 201 generates a viewpoint image based on the pixel value set for the pixel of the viewpoint image and the blind spot area information.

In S812, the control unit 201 outputs the viewpoint image, and this operation flow ends. For example, the control unit 201 may store the image data of the viewpoint image in the storage unit 202, or in another example, may display the viewpoint image on a display device 204 such as a display.

Further, for example, when displaying a viewpoint image on the display device 204, the control unit 201 may display the blind spot area of the viewpoint image in a predetermined display format so that it can be identified. Thereby, the user viewing the viewpoint image can know that something that is not shown in the photographed image is visible in the blind spot when viewed from the viewpoint position.

As described above, according to the operation flow shown in FIG. 8, it is possible to generate a viewpoint image in which the subject 110 appearing in the photographed image having depth information is viewed from an arbitrary viewpoint.

In addition, as described in the operation flow of FIG. 8, in the embodiment, a perspective projection model of the photographed image and the viewpoint image is placed in the virtual space, and the pixels of the photographed image that correspond to the pixels of the viewpoint image are identified by ray tracing. are doing. Therefore, for example, a viewpoint image can be generated without generating a three-dimensional shape model of the subject.

For example, techniques such as texture mapping and projected texture mapping are known as image-based rendering techniques. These texture mappings and projected texture mappings generate a three-dimensional shape model of the subject, and therefore use a large amount of memory and also require a large amount of calculation. In the above-described embodiment, since a viewpoint image can be generated without generating a three-dimensional shape model, it is possible to reduce the amount of memory used for generating a viewpoint image. Furthermore, it is possible to generate viewpoint images at high speed.

In addition, in the above embodiment, calculations are performed in the virtual space at the same scale as the actual space at the time of shooting, and for example, there is no need to perform calculations or normalization in the projected coordinate system. It is possible to generate a viewpoint image in which the viewpoint is strictly converted in units of length.

Further, for example, homography conversion is known as an image conversion technique. In homography conversion, an area that should be hidden by another subject 110 and become invisible after conversion may end up being displayed in the converted image. According to the above-described embodiment, as illustrated with reference to FIG. 7, for example, the projection line 308 that points to an occluded area that is occluded by another subject and becomes invisible after the viewpoint conversion is not selected, so that the occluded area is not occluded. It can be expressed correctly using a viewpoint image.

In the above embodiment, the three-dimensional arrangement of the optical center 301, the optical axis 302, the imaging projection plane 303, the viewpoint 310, the viewpoint projection plane 311, etc. is defined, and the calculation is performed assuming that these are arranged in the defined three-dimensional arrangement. It is carried out. In other words, the optical center 301, optical axis 302, imaging projection plane 303, viewpoint 310, and viewpoint projection plane 311 can be calculated without actually being arranged as polygons or the like in the virtual space. Furthermore, in the embodiment described above, the three-dimensional plane 305 is arranged as a polygon in the virtual space. By arranging the three-dimensional plane 305 as a polygon in the virtual space, it becomes possible to pass the pixels of the perspective projection result to a pixel shader of a GPU (Graphics Processing Unit) and process them at high speed. However, in one example, the polygons of the three-dimensional plane 305 do not have to be placed in the virtual space.

Therefore, according to the embodiment, it is possible to reduce the processing load of generating a viewpoint image in which the subject is seen in a different direction from a photographed image in which the subject is seen.

Next, a process for specifying the projection line 308 corresponding to the line of sight 313 according to the embodiment will be illustrated. FIG. 9 is a diagram illustrating the operational flow of the process of specifying the projection line 308 corresponding to the line of sight 313 according to the embodiment. Further, FIG. 10 is a diagram illustrating the flow of the process of specifying the projection line 308 corresponding to the line of sight 313 according to the embodiment. For example, when the control unit 201 proceeds to S805 in FIG. 8, it may start the operation flow in FIG. 9.

In S901, the control unit 201 obtains, for example, a vector 1001 from the viewpoint 310 to the three-dimensional plane 305 passing through the processing target pixel 312 selected from the viewpoint image arranged on the viewpoint projection plane 311 (((a) in FIG. 10A). 1)). Note that, hereinafter, this vector may be referred to as the line-of-sight vector 1001.

In S902, the control unit 201 determines, for example, the intersection between the line-of-sight vector 1001 and the three-dimensional plane 305.

In S903, the control unit 201 arranges the layer 1002 at a predetermined depth parallel to the three-dimensional plane 305 ((2) in FIG. 10A), for example. Note that the predetermined depth may be set, for example, so that the layer 1002 is placed at a shallower depth than the foreground subject 110 in the captured image.

In S904, the control unit 201 determines, for example, the intersection 1003 between the line of sight 313 and the layer 1002 ((3) in FIG. 10A).

In S905, the control unit 201 obtains, for example, a projected line vector 1004 from the optical center 301 toward the intersection 1003 ((4) in FIG. 10(a)).

In S906, the control unit 201, for example, finds the intersection between the projection line vector 1004 and the imaging projection plane 303, and specifies the pixel 307 of the depth image located at the intersection ((5) in FIG. 10A). For example, the control unit 201 may find the intersection with the imaging projection plane 303 based on the optical center 301, the focal length, and the projection line vector 1004, and specify the pixel 307 of the depth image located at the intersection.

In S907, the control unit 201 identifies the depth of the subject 110, for example, from the depth information of the pixel 307 identified from the depth image ((6) in FIG. 10A). Note that in the example (6) of FIG. 10A, the depth of the subject 110 from the three-dimensional plane 305 is shown as the depth of the subject 110 specified from the depth information of the pixel 307. The depth of the subject 110 from the three-dimensional plane 305 can be determined, for example, by subtracting the focal length from the depth indicated by the depth information of the pixel 307.

In S908, the control unit 201 determines that, for example, the arrangement in the depth direction between the depth of the subject 110 identified from the depth information of the pixel 307 and the depth of the layer 1002 at the intersection of the line-of-sight vector 1001 and the projection line vector 1004 is reversed. Determine whether or not.

For example, in FIG. 10A, in the initial arrangement of the layer 1002, the depth of the layer 1002 at the intersection of the line-of-sight vector 1001 and the projection line vector 1004 is greater than the depth of the subject 110 identified from the depth information of the pixel 307. is shallower. In this case, in the process of S908, if the depth of the layer 1002 is deeper than the depth of the object 110 specified from the depth information of the pixel 307, the depth of the object 110 and the arrangement of the layer 1002 in the depth direction are It may be determined that the image has been reversed. If the depth of the subject 110 and the arrangement of the layer 1002 in the depth direction are not reversed, the control unit 201 determines NO in S908, and the flow advances to S909.

In S909, the control unit 201 moves the layer 1002 by a predetermined amount of movement in the direction of the depth of the subject 110 indicated by the depth information of the pixel 307, and the flow returns to S904 to repeat the process. For example, the control unit 201 may move the position of the layer 1002 to bring the layer 1002 closer to a certain direction of the depth of the subject 110 indicated by the pixel 307, as shown in (7) of FIG. 10(b). As a result, the control unit 201 gradually changes the position of the layer 1002 to locate the layer 1002 where the line-of-sight vector 1001 and the projection line vector 1004 intersect at a depth close to the depth of the subject 110 indicated by the depth information of the pixel 307. You can explore.

Further, in S908, for example, if the depth of the subject 110 and the arrangement of the layer 1002 in the depth direction are reversed, the control unit 201 determines YES, and the flow advances to S910. In S910, the control unit 201 determines whether the amount and direction of movement of the layer 1002 have been changed a predetermined number of times.

If the amount and direction of movement of the layer 1002 have not been changed a predetermined number of times in S910 (NO in S910), the flow advances to S911. In S911, the control unit 201 reverses the moving direction of the layer 1002 and changes the moving amount by a predetermined ratio such as 1/2 to 1/10, and the flow returns to S909. For example, as shown in (8) of FIG. 10C, the control unit 201 may reverse the moving direction of the layer 1002 to reduce the moving amount and continue the search. Thereby, the layer 1002 where the line-of-sight vector 1001 and the projection line vector 1004 intersect can be searched more precisely at a depth close to the depth of the subject 110 indicated by the depth information of the pixel 307.

Further, if the amount and direction of movement of the layer 1002 have been changed a predetermined number of times in S910 (YES in S910), the flow advances to S912. In S912, the control unit 201 specifies the projection line vector 1004 extended toward the intersection of the final layer 1002 and the line of sight vector 1001 as the projection line 308 corresponding to the line of sight 313, and this operation flow ends. The process advances to S806.

As described above, according to the operation flow of FIG. 9, the depth difference between the depth of the subject 110 indicated by the pixels of the depth image through which the projection line 308 passes and the depth at the intersection position of the projection line 308 and the line of sight 313 is The minimum projection line 308 can be identified. Note that the projection line 308 with the minimum depth difference may be, for example, the projection line 308 with the minimum depth difference among the projection lines drawn from the optical center 301 to the pixels of the photographed image. In another example, the projection line 308 with the minimum depth difference may be the projection line 308 with the minimum depth difference among the searched projection lines 308.

Note that if the predetermined number of times in S910 is large, the search accuracy for the layer 1002 where the line-of-sight vector 1001 and the projection line vector 1004 intersect will be improved at a depth close to the depth of the subject 110 indicated by the depth information of the pixel 307. On the other hand, when the number of times increases, the amount of calculation increases, leading to a decrease in processing speed. Therefore, the predetermined number of times may be set to a preferable value depending on the situation in which the embodiment is applied. For example, the predetermined number of times may be set to 2 to 5 times, and in one example, may be set to 3 times.

Further, in the operation flow of FIG. 9, the initial depth of the layer 1002 arranged parallel to the three-dimensional plane 305 may be set to be shallower than the depth of the subject 110. As a result, for example, as described with reference to FIG. 7, even when there are a plurality of projection lines 308 that intersect the line of sight 313 on the subject 110, the projection lines 308 that intersect the line of sight 313 at a shallower depth can be can be specified. Thereby, even when another subject 110 is covered in the foreground in a viewpoint image in which the viewpoint has been moved to an area that was visible in the photographed image, the occlusion can be accurately expressed in the viewpoint image.

Further, as shown in the process of FIG. 9, by tracing the light transport path back from the viewpoint 310 toward the optical center 301, calculation can be performed using only the minimum number of light rays. For example, pixels on the captured image that are not used in the viewpoint image can be omitted from the processing target. Further, it is possible to search only for the projection line 308 that intersects the line of sight 313. Therefore, the amount of calculation can be reduced.

FIG. 11 is a diagram illustrating a photographed image and a viewpoint image according to the embodiment. In FIG. 11, a captured image 1101 is shown in the center. Additionally, on the top, bottom, left and right of the photographed image 1101, there are viewpoint images obtained by fixing the gaze point on the optical axis and moving the viewpoint in the direction of the x-axis or y-axis of the camera coordinate system with the optical center as the origin. 1102 is shown. The viewpoint image 1102 includes a blind spot area 1105, and an area of the subject 110 that is not captured in the captured image 1101 is displayed as the blind spot area 1105.

Further, according to the embodiment, as shown in FIG. 11, viewpoint images 1102 viewed from various viewpoints 310 can be generated. The viewpoint images 1102 seen from various viewpoints 310 can be used for inspecting, determining, and identifying the shape and color of parts, products, manufactured products, molded objects, etc. that are photographed as the subject 110.

Furthermore, for example, due to the presence of an obstacle, there are restrictions on the installation position of the photographing device 102, and it may not be possible to photograph the subject 110 from an angle suitable for inputting it into processes such as position recognition and shape recognition of the subject 110. . According to the embodiment, since captured images can be converted into viewpoint images seen from various viewpoints 310, even when there are restrictions on the installation position of the imaging device 102, the images can be input into processing such as position recognition and shape recognition. It is possible to generate viewpoint images suitable for

Note that, as shown in FIG. 11, when a viewpoint image 1102 is generated by moving the viewpoint using a photographed image 1101 having depth information taken from one location, the viewpoint may change due to unevenness of the subject 110, etc. The image 1102 may include a blind spot area. Such a blind spot area can be replaced with pixels that capture the subject 110 using a photographed image in which the blind spot area is photographed from a different direction. Hereinafter, generation of a viewpoint image using captured images captured from a plurality of positions will be explained.

FIG. 12 is a diagram illustrating generation of a viewpoint image using captured images captured from a plurality of positions according to the embodiment. In FIG. 12, an image obtained by photographing an image of the photographic projection plane 303a from the first optical center 301a is defined as a first photographed image. Further, an image obtained by photographing an image on the photographing projection surface 303b from the second optical center 301b is set as a second photographed image. Note that in both the first captured image and the second captured image, pixels are associated with depth information. An example of generating a viewpoint image when viewing the subject 110 from the viewpoint 310 using the first captured image and the second captured image will be described below.

In FIG. 12, for example, a point 1211 cannot be seen from the first optical center 301a because it is hidden by the subject 110 in the foreground and forms a blind spot. Therefore, the first captured image captured from the first optical center 301a does not include information representing the pixel value of the subject 110 at the point 1211. On the other hand, the point 1211 is visible from the second optical center 301b, and the second photographed image taken from the second optical center 301b includes information representing the pixel value of the subject 110 at the point 1211. . Therefore, when generating the viewpoint image seen from the viewpoint 310, it is possible to supplement the pixel values of pixels that would be in the blind area if only the first captured image was used, from the second captured image. be.

For example, the control unit 201 obtains a viewpoint image from each of two captured images captured from two different positions. It is assumed that a pixel that is a blind spot area in one of the two obtained viewpoint images has a pixel value in the other viewpoint image. In this case, the control unit 201 may generate a viewpoint image by combining the two viewpoint images so that the pixel value obtained from the other viewpoint image is used as the pixel value of the pixel. Thereby, it is possible to generate a viewpoint image with reduced blind spot areas. Note that a viewpoint image obtained by combining a plurality of viewpoint images in this manner may be referred to as a composite viewpoint image below.

Furthermore, for example, both viewpoint images obtained from two photographed images taken from two different positions may have a pixel value for a certain pixel. FIG. 13 illustrates this case, and the position of the point 1311 of the subject 110 is visible from both the first optical center 301a and the second optical center 301b. In this case, the control unit 201 may acquire pixel values from a captured image captured at a smaller angle between the projection line 308 and the line of sight 313. For example, in the example of FIG. 13, the angle θ2 between the projection line 308b and the line of sight 313 is smaller than the angle θ1 between the projection line 308a and the line of sight 313. Therefore, the control unit 201 may acquire the pixel value of pixel 1303 on the viewpoint image through which the line of sight passes from pixel 1301 on the second captured image through which the projection line 308b passes, and use it in the composite viewpoint image. Thereby, pixel values can be acquired from a photographed image in which the viewpoint image is photographed from a direction close to the line of sight 313. Therefore, it is possible to improve the accuracy of the color tone, etc. of the generated synthetic viewpoint image.

In addition, for example, for a pixel that is set in a blind spot area in each of the viewpoint images obtained from two captured images taken from two different positions, the control unit 201 controls the pixel in the composite viewpoint image. It may be set in the blind spot area.

As described above, by using a plurality of captured images captured from different positions, it is possible to generate a composite viewpoint image by supplementing the pixel values in the blind spot area. Note that although the example in FIG. 13 shows an example in which photographed images photographed from two photographing positions are used, the embodiment is not limited to this, and photographed images photographed from three or more photographing positions are used. One viewpoint image may be generated using .

In addition, by shooting from another shooting position so that the area that becomes a blind spot when shooting from a certain shooting position is captured, a captured image can be obtained that can efficiently compensate for the pixel values of pixels in the blind spot area. be able to.

FIG. 14 is a diagram illustrating an operational flow of viewpoint image generation processing based on captured images captured from two different positions according to the embodiment. For example, the control unit 201 may start the operation flow shown in FIG. 14 when an instruction to generate a viewpoint image is input.

In S1401, the control unit 201 generates a viewpoint image from each of the captured images having depth information captured from a plurality of shooting positions, for example. Note that the control unit 201 may generate a viewpoint image from a photographed image having depth information, for example, by the processing illustrated in the operational flows of FIGS. 8 and 9.

In S1402, the control unit 201 superimposes a plurality of viewpoint images generated for each shooting position.

In S1403, the control unit 201 may set the obtained pixel value for each pixel in the superimposed viewpoint images for a pixel whose pixel value is obtained from only one viewpoint image. For example, suppose that in perspective images generated from two captured images taken at two shooting positions, a pixel that is set as a blind spot area in one perspective image has a pixel value in the other perspective image. . In this case, the control unit 201 may generate a composite viewpoint image using the pixel value in the other viewpoint image as the pixel value of that pixel.

In S1404, for each pixel in the superimposed viewpoint images, the control unit 201 selects a pixel whose pixel value is obtained from a plurality of viewpoint images, the pixel corresponding to the projection line 308 that forms the smallest angle with the line of sight 313. The pixel value obtained from is set to the pixel of the composite viewpoint image. Thereby, pixel values can be acquired from a photographed image photographed from a direction close to the line of sight 313.

In S1405, the control unit 201 sets a blind spot area. For example, the control unit 201 may set pixels that are set as blind spot areas in all of the plurality of superimposed viewpoint images to be blind spot areas in the composite viewpoint image as well.

In S1406, the control unit 201 synthesizes a plurality of viewpoint images based on the pixel values set in the processing from S1403 to S1405, and generates a composite viewpoint image based on the plurality of captured images.

In S1407, the control unit 201 outputs a viewpoint image based on the plurality of captured images, and this operation flow ends. For example, the control unit 201 may store the image data of the viewpoint image in the storage unit 202, or in another example, may display the viewpoint image on a display device 204 such as a display.

As described above, according to the operation flow shown in FIG. 14, by generating a viewpoint image based on a plurality of captured images, it is possible to generate a viewpoint image while compensating for a blind spot area.

Although the embodiments have been illustrated above, the embodiments are not limited thereto. For example, the above-described operational flow is an example, and the embodiments are not limited thereto. If possible, the operational flow may be executed by changing the order of processing, may include additional processing, or may omit some processing.

Furthermore, in the embodiment described above, the projection line 308 corresponding to the line of sight 313 is specified based on the depth difference between the depth of the subject 110 indicated by the pixels of the depth image and the depth at the intersection position of the projection line 308 and the line of sight 313. An example is given. However, embodiments are not limited thereto. In another embodiment, the control unit 201 specifies the projection line 308 such that the distance between the position based on the depth of the subject 110 indicated by the pixels of the depth image and the intersection position of the projection line 308 and the line of sight 313 is minimized. You may.

Further, in the above-described embodiment, an example is described in which the information processing apparatus 101 generates a viewpoint image, but the embodiment is not limited to this. For example, in another embodiment, the processing described above may be shared and performed by multiple devices using a client-server system or the like.

Note that in the above-described embodiment, the control unit 201 operates as the placement unit 211 in, for example, the processing of S801 to S803. Further, the control unit 201 operates as the acquisition unit 212 in the process of S807, for example. For example, the control unit 201 operates as the generation unit 213 in the process of S811.

FIG. 15 is a diagram illustrating a hardware configuration of a computer 1500 for realizing the information processing apparatus 101 according to the embodiment. A hardware configuration for realizing the information processing apparatus 101 in FIG. 15 includes, for example, a processor 1501, a memory 1502, a storage device 1503, a reading device 1504, a communication interface 1506, and an input/output interface 1507. Note that the processor 1501, memory 1502, storage device 1503, reading device 1504, communication interface 1506, and input/output interface 1507 are connected to each other via a bus 1508, for example.

The processor 1501 may be, for example, a single processor, a multiprocessor, or a multicore. The processor 1501 provides some or all of the functions of the control unit 201 described above by using the memory 1502 to execute, for example, a program that describes the procedure of the operation flow described above. For example, the processor 1501 of the information processing device 101 operates as the placement unit 211, the acquisition unit 212, and the generation unit 213 by reading and executing a program stored in the storage device 1503.

Memory 1502 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 1503 is, for example, a hard disk, a semiconductor memory such as a flash memory, or an external storage device. Note that RAM is an abbreviation for Random Access Memory. Further, ROM is an abbreviation for Read Only Memory.

Reading device 1504 accesses removable storage medium 1505 according to instructions from processor 1501. The removable storage medium 1505 is realized by, for example, a semiconductor device, a medium in which information is input/output by magnetic action, a medium in which information is input/output by optical action, or the like. Note that the semiconductor device is, for example, a USB (Universal Serial Bus) memory. Further, a medium in which information is input/output by magnetic action is, for example, a magnetic disk. Examples of media on which information is input and output through optical action include CD-ROMs, DVDs, and Blu-ray Discs (Blu-ray is a registered trademark). CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk.

The storage unit 202 includes, for example, a memory 1502, a storage device 1503, and a removable storage medium 1505. For example, the storage device 1503 of the information processing apparatus 101 may store, for example, a program that describes the processing of the above-described operational flow and information that is referenced when executing the processing.

Communication interface 1506 communicates with other devices according to instructions from processor 1501. For example, the communication interface 1506 may receive imaging data such as a captured image and a depth image from the imaging device 102 via wired or wireless communication. The communication interface 1506 is, for example, an example of the communication unit 203 described above.

Input/output interface 1507 may be, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard, mouse, or touch panel that receives instructions from a user. The output device is, for example, an audio device such as a speaker. Furthermore, the input/output interface 1507 may be connected to, for example, the display device 204 described above.

Each program according to the embodiment is provided to the information processing apparatus 101 in the following format, for example.
(1) Installed in the storage device 1503 in advance.
(2) Provided by a removable storage medium 1505.
(3) Provided by a server such as a program server.

Note that the hardware configuration of the computer 1500 for realizing the information processing apparatus 101 described with reference to FIG. 15 is an example, and the embodiment is not limited to this. For example, some of the configurations described above may be deleted, or new configurations may be added. In another embodiment, for example, some or all of the functions of the information processing device 101 described above may be implemented as hardware such as an FPGA, an SoC, an ASIC, and a PLD. Note that FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip. ASIC is an abbreviation for Application Specific Integrated Circuit. PLD is an abbreviation for Programmable Logic Device. For example, the above-described viewpoint image generation processing may be implemented as hardware. Further, the hardware that executes this viewpoint image generation processing may be integrated with the imaging device 102, such as being installed in the imaging device 102, for example.

Above, several embodiments are described. However, the embodiments are not limited to the embodiments described above, but should be understood to include various modifications and alternative forms of the embodiments described above. For example, it will be understood that the various embodiments can be embodied by changing the components without departing from the spirit and scope thereof. Furthermore, it will be understood that various embodiments can be implemented by appropriately combining the plurality of components disclosed in the embodiments described above. Furthermore, various embodiments may be implemented by deleting some components from all the components shown in the embodiments or adding some components to the components shown in the embodiments. It will be understood by those skilled in the art.

Reference Signs List 100 Photographing system 101 Information processing device 102 Photographing device 105 Installation stand 106 Photographing object 110 Subject 201 Control unit 202 Storage unit 203 Communication unit 204 Display device 211 Arrangement unit 212 Acquisition unit 213 Generation unit 301 Optical center 302 Optical axis 303 Photographic projection plane 305 Three-dimensional plane 308 Projection line 310 Viewpoint 311 Viewpoint projection plane 313 Line of sight 1500 Computer 1501 Processor 1502 Memory 1503 Storage device 1504 Reading device 1505 Removable storage medium 1506 Communication interface 1507 Input/output interface 1508 Bus

Claims (8)

The optical center of the photographing device used to photograph the photographed image, in which each pixel is associated with depth information indicating the depth of the subject, the photographic projection plane on which the photographed image is projected, the viewpoint, and the A viewpoint projection plane onto which a viewpoint image depicting the subject is projected is arranged in virtual space,
A line of sight extending from the viewpoint toward a pixel to be processed of the viewpoint image arranged on the viewpoint projection surface, and a projection line extending from the optical center toward a pixel of the photographed image arranged on the photographic projection surface. and the depth of the subject indicated by the depth information corresponding to the pixels of the photographed image, the pixel value corresponding to the processing target pixel of the viewpoint image is determined from among the pixels of the photographed image. retrieved from
generating the viewpoint image using the pixel value acquired for the processing target pixel;
A generation program that causes a computer to perform a process. The acquiring process includes determining the depth indicated by the intersection position of the line of sight and the projection line extending from the optical center toward the pixel, among a plurality of pixels of the photographed image arranged on the photographic projection plane, and the pixel. 2. The generation program according to claim 1, wherein the pixel value of the processing target pixel is acquired from a pixel that has a minimum depth difference from the depth of the subject based on the depth information. The acquiring process is based on the intersection position of the line of sight and a projection line extended from the optical center toward the pixel where the depth difference is the minimum, and the depth information of the pixel where the depth difference is the minimum. 3. The generation program according to claim 2, wherein when the distance to the subject position is large and satisfies a predetermined condition, the processing target pixel of the viewpoint image is set to a blind spot area. The acquiring process includes, when the line of sight intersects a plurality of projection lines on the subject, the captured image corresponding to the projection line having the shallowest intersection among the intersection positions of the plurality of projection lines and the line of sight. 4. The generation program according to claim 2, wherein the pixel value of the pixel to be processed is acquired from the pixel of . The generating process further includes:
Generating a second viewpoint image of the subject from the viewpoint based on a second captured image in which each pixel is associated with the depth information indicating the depth of the subject;
When the processing target pixel is set in the blind spot area in either the viewpoint image or the second viewpoint image, the pixel value of the other viewpoint image is set as the pixel value of the processing target pixel. 4. The generation program according to claim 3, comprising a process for using. The generating process further includes:
Generating a second viewpoint image of the subject from the viewpoint based on a second captured image in which each pixel is associated with the depth information indicating the depth of the subject;
When both the viewpoint images and the second viewpoint image have pixel values at the processing target pixels, the pixel values at the processing target pixels of both viewpoint images are determined by the processing from the viewpoint. 5. The method according to claim 1, further comprising a process of setting a pixel value obtained from a pixel corresponding to a projection line extending toward the target pixel and forming a smaller angle with the line of sight as the pixel value of the target pixel. The generation program according to any one of the items. The optical center of the photographing device used to photograph the photographed image in which each pixel is associated with depth information indicating the depth of the subject, the photographic projection plane on which the photographed image is projected, the viewpoint, and the A viewpoint projection plane onto which a viewpoint image depicting the subject is projected is arranged in virtual space,
A line of sight extending from the viewpoint toward a pixel to be processed of the viewpoint image arranged on the viewpoint projection surface, and a projection line extending from the optical center toward a pixel of the photographed image arranged on the photographic projection surface. and the depth of the subject indicated by the depth information corresponding to the pixels of the photographed image, the pixel value corresponding to the processing target pixel of the viewpoint image is determined from among the pixels of the photographed image. retrieved from
generating the viewpoint image using the pixel value acquired for the processing target pixel;
computer-implemented generation methods, including The optical center of the photographing device used to photograph the photographed image, in which each pixel is associated with depth information indicating the depth of the subject, the photographic projection plane on which the photographed image is projected, the viewpoint, and the an arrangement unit that arranges, in virtual space, a viewpoint projection surface onto which a viewpoint image depicting the subject is projected;
A line of sight extending from the viewpoint toward a pixel to be processed of the viewpoint image arranged on the viewpoint projection surface, and a projection line extending from the optical center toward a pixel of the photographed image arranged on the photographic projection surface. and the depth of the subject indicated by the depth information corresponding to the pixels of the photographed image, the pixel value corresponding to the processing target pixel of the viewpoint image is determined from among the pixels of the photographed image. an acquisition unit that acquires from
a generation unit that generates the viewpoint image using the pixel value acquired for the processing target pixel;
Information processing equipment, including.

Images