JP5601375B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing technique.

  In recent years, 3D television using a moving image that can be stereoscopically viewed (also referred to as a 3D moving image) is in the spotlight. In 3D television, an image (also referred to as a 3D image) that can be stereoscopically viewed is generated using two images with different viewpoints. In this 3D image technology, the focus adjustment function of the user's eyes is used to realize a sense of depth of the image.

  However, regarding a 3D moving image, when a sudden scene change occurs, it becomes easy for the user to feel tired. Thus, a technique has been proposed that can reduce the fatigue of a user who performs stereoscopic viewing by gradually changing the sense of depth at the scene switching position (for example, Patent Document 1).

  In addition, 3D image technology has been adopted in various video fields. For example, an endoscopic device that enables stereoscopic viewing of an image in a wide range by adjusting parallax so as to fall within a human fusion range. Has been proposed (for example, Patent Document 2).

JP 2009-239388 A JP-A-8-313825

  By the way, if an image (also referred to as a virtual viewpoint image) that captures the same subject from a virtual viewpoint that is different from the viewpoint related to the captured image is generated based on one captured image, a 3D image is generated. Is possible. Then, if a moving image (also referred to as a virtual viewpoint moving image) that captures the same subject from a virtual viewpoint that is different from the viewpoint related to the moving image is generated based on one moving image, a 3D moving image can be generated. It becomes.

  However, in the virtual viewpoint image, an area corresponding to a portion of the subject that is not captured in the original captured image is an area where the pixel value is unknown (also referred to as an occlusion area). Due to the occurrence of the occlusion area, there is no video that should originally exist in the 3D moving image, which makes the user feel uncomfortable. Furthermore, the higher the proportion of the occlusion area in the image, the greater the sense of discomfort experienced by the user. In particular, in a scene where a foreground appears suddenly in an image, an increase in discomfort experienced by the user becomes significant.

  With respect to such a problem, none of the techniques disclosed in Patent Documents 1 and 2 cannot reduce the user's uncomfortable feeling corresponding to an occlusion region in which different sizes can occur. In order to solve such a problem, it is required that the virtual viewpoint moving image that is the basis of the 3D moving image is visually uncomfortable.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for reducing visual discomfort given to a user when generating a pseudo moving image viewed from a virtual viewpoint. .

In order to solve the above problem, an image processing apparatus according to a first aspect includes an image acquisition unit that acquires a moving image including a plurality of frames, and a predetermined rule for each pixel for each of the frames. A target parallax setting unit that sets a target parallax, an initial parallax setting unit that sets an initial parallax for each pixel for the first frame of the plurality of frames, and the one of the plurality of frames A parallax adjustment unit that adjusts the parallax of each pixel from the initial parallax toward the target parallax for a plurality of frames according to a first period , a second period, and a third period that are time-sequential following the first frame. The first rate of change obtained by dividing the amount of change in parallax in the first period by the length of the first period is obtained by dividing the amount of change in parallax in the second period by the length of the second period. Required by Rather smaller than 2 change rate, the second rate of change, not larger than the third change rate obtained by dividing the amount of change of the disparity in length of the third period in the third period.

  The image processing device according to the second aspect is the image processing device according to the first aspect, and stores a plurality of types of adjustment patterns related to a change in parallax from the initial parallax toward the target parallax, The parallax adjustment unit adjusts the parallax of each pixel along an adjustment pattern according to a difference between the initial parallax and the target parallax among the plurality of types of adjustment patterns.

  An image processing device according to a third aspect is the image processing device according to the first aspect, wherein the parallax adjustment unit is adjusted to the Nth after being adjusted by the parallax adjustment unit when N is a natural number. The parallax of the (N + 1) th and subsequent frames is adjusted based on the parallax of the current frame and the target parallax.

  An image processing device according to a fourth aspect is the image processing device according to the first aspect, wherein when the parallax adjustment unit sets Q as a natural number, the target parallax for each pixel in the N + Q-th frame and The parallax of the (N + 1) th and subsequent frames is adjusted based on the parallax of the Nth frame after being adjusted by the parallax adjusting unit.

  An image processing device according to a fifth aspect is the image processing device according to the first or second aspect, wherein the parallax adjustment unit calculates the parallax of each pixel for the number of frames corresponding to a predetermined time. The parallax is adjusted to the target parallax.

  An image processing device according to a sixth aspect is the image processing device according to any one of the first to fourth aspects, wherein the parallax adjustment unit has a parallax adjustment amount between temporally adjacent frames. The parallax of each pixel is adjusted so as not to exceed the upper limit value.

  An image processing apparatus according to a seventh aspect is the image processing apparatus according to any one of the first to sixth aspects, wherein the parallax adjustment unit has a parallax of a first subject captured by the moving image. Accordingly, the parallax of each pixel is adjusted for a second subject different from the first subject.

  An image processing device according to an eighth aspect is the image processing device according to any one of the first to sixth aspects, wherein the parallax adjustment unit is in accordance with the position of the subject captured in each frame. Then, the parallax of each pixel is adjusted.

  An image processing device according to a ninth aspect is the image processing device according to any one of the first to eighth aspects, wherein the target parallax is a separation between a viewpoint related to each frame and a virtual viewpoint. This corresponds to the distance and the distance from the imaging device to the subject when each of the frames is captured.

  An image processing apparatus according to a tenth aspect is the image processing apparatus according to any one of the first to ninth aspects, wherein N is a N number among the plurality of frames when N and p are natural numbers. A detection unit that detects an adjustment region in the Nth frame by comparing the Nth frame and the Npth frame that is temporally prior to the Nth frame, and the parallax adjustment The unit adjusts the parallax of each pixel related to the adjusted region from the initial parallax toward the target parallax.

An image processing method according to an eleventh aspect includes (a) a step of acquiring a moving image including a plurality of frames, and (b) target parallax for each pixel based on a predetermined rule for each frame. (C) setting an initial parallax for each pixel for the first frame of the plurality of frames, and (d) setting the first frame of the plurality of frames to the first frame. Adjusting the parallax of each pixel from the initial parallax toward the target parallax for a plurality of frames in the subsequent time-sequential first period , second period, and third period , and in the first period The first change rate obtained by dividing the amount of change in parallax by the length of the first period is the second change obtained by dividing the amount of change in parallax in the second period by the length of the second period. rather smaller than the rate, the second Rate is not larger than the third change rate obtained by dividing the amount of change of the disparity in length of the third period in the third period.

  A program according to a twelfth aspect is executed by a control unit included in the information processing apparatus to cause the information processing apparatus to function as the image processing apparatus according to any one of the first to tenth aspects. It is.

  According to the image processing apparatus according to any one of the first to tenth aspects, it is possible to reduce a visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint.

  According to the image processing device according to the second aspect, accurate parallax adjustment can be easily performed.

  According to the image processing apparatus according to the third aspect, it is possible to adjust parallax appropriately.

  According to the image processing device according to the fourth aspect, it is possible to easily adjust the parallax corresponding to the change in the target parallax.

  According to the image processing device according to the fifth aspect, since the variation in time required for parallax adjustment is suppressed, the visual discomfort given to the user can be appropriately suppressed.

  According to the image processing device according to the sixth aspect, since a significant change in parallax is suppressed, the visual discomfort given to the user can be appropriately suppressed.

  According to the image processing apparatus according to the seventh aspect, since the consistency of parallax between objects is improved, the visual discomfort given to the user can be appropriately suppressed.

  With the image processing device according to the eighth aspect, it is possible to adjust parallax according to the degree of attention of the user.

  With the image processing device according to the ninth aspect, it is possible to set a target parallax suitable for generating an image that can be viewed stereoscopically.

  With the image processing device according to the tenth aspect, it is possible to adjust parallax as necessary.

  Any of the image processing method according to the eleventh aspect and the program according to the twelfth aspect can reduce the visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint. it can.

FIG. 1 is a diagram for explaining the characteristics of a process according to an embodiment. FIG. 2 is a diagram illustrating a schematic configuration of an image processing system according to an embodiment. FIG. 3 is a block diagram illustrating a functional configuration according to the image processing apparatus. FIG. 4 is a diagram for explaining how to obtain the target parallax using the principle of triangulation. FIG. 5 is a diagram schematically illustrating an example of an Nth right eye frame. FIG. 6 is a diagram schematically illustrating an example of an Nth right eye frame. FIG. 7 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 8 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 9 is a flowchart showing an operation flow of the image processing apparatus. FIG. 10 is a flowchart showing an operation flow of the image processing apparatus. FIG. 11 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image. FIG. 12 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 13 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 14 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image. FIG. 15 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 16 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 17 is a schematic view illustrating an aspect of parallax adjustment related to a virtual viewpoint image. FIG. 18 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 19 is a graph schematically illustrating the relationship between time and adjusted parallax. FIG. 20 is a graph schematically illustrating the relationship between time and adjusted parallax.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, parts having the same configuration and function are denoted by the same reference numerals, and redundant description is omitted in the following description. Further, the drawings are schematically shown, and the size and positional relationship of objects on the images in each drawing are not accurately shown. Furthermore, the image data and images displayed based on the image data are collectively referred to as “image”.

<(1) Features of processing according to one embodiment>
FIG. 1 is a diagram for explaining the characteristics of a process according to an embodiment.

As shown in FIG. 1, for example, an Nth (N is a natural number less than n) frame (also referred to as an Nth frame) GN of a moving image composed of n frames (n is a natural number greater than or equal to 2) is _N , With a rectangular outer edge. Here, the upper left pixel of the N frame G _N is the origin, the direction (in this case the transverse direction) along the long sides of the N-th frame G _N is the X-axis direction, the short side of the N-th frame G _N The direction along the vertical axis (here, the vertical direction) is the Y-axis direction. Then, the right direction of the N-th frame G _N is the + X direction, the downward direction of the N-th frame G _N is the -Y direction. In FIG. 1 and other drawings after FIG. 1, two orthogonal XY axes are shown as necessary.

Here, as shown in FIG. 1, for example, an Nth frame (also referred to as an Nth frame) of N frames (N is a natural number of n or less) of n frames (n is a natural number of 2 or more). from G _N, it is assumed that stereoscopic vision is 3D image can be generated. For example, the N-th frame G _N is (also referred to as a first N left eye frame) as the N-th frame for the left eye is used as a G _N L, N-th frame for the right eye from the N frame G _N (No. N is also referred to as the right-eye frame) is assumed that G _N R is generated.

Here, the position of the camera (specifically, the image sensor) when the Nth frame G _N (the Nth left eye frame G _N L) is captured is the Nth viewpoint, and the Nth right eye frame. position of the virtual camera for capturing G _N R is the N-th virtual viewpoint. The Nth viewpoint and the Nth virtual viewpoint are in a positional relationship between a human standard left eye and right eye. As a result, a displacement (that is, parallax) of the pixel positions where the same subject is captured occurs between the N-th left eye frame G _N L and the N-th right eye frame G _N R.

Incidentally, the N-th right eye frame G _N R includes not only the area corresponding to the object (here, a person) OB _N and the background captured in the N-th left eye frame G _N L, but also the N-th left eye. In the eye frame G _N L, there is an area OC _N (also referred to as a non-imaging area or an occlusion area) corresponding to a portion of the subject that is not captured because it is located behind the person OB _N. This occlusion region OC _N is the actual pixel value is not known. Therefore, in the scene to switch to one scene and the object is completely different at the beginning of the video, the occlusion region OC _N is suddenly displayed on the screen, there is a possibility of giving an uncomfortable feeling to the user.

Therefore, in the present embodiment, when the N-th right eye frame G _N R is generated, the amount of change in parallax is appropriately adjusted according to various conditions so that the occlusion area OC _N is not as noticeable as possible. , the amount of the occlusion region OC _N is adjusted.

<(2) Schematic configuration of image processing system>
FIG. 2 is a diagram showing a schematic configuration of the information processing system 1 according to the embodiment of the present invention.

  The information processing system 1 includes a camera 2, a distance measuring device 3, and an information processing device 4. The information processing device 4 is connected to the camera 2 and the distance measuring device 3 so as to be able to transmit and receive data.

  The camera 2 is, for example, an imaging device configured with a digital camera. A digital camera has an image sensor such as a CCD. The camera 2 receives light from the subject and performs an operation (so-called photographing) in which information indicating a distribution relating to the luminance of the subject is acquired as image data by photoelectric conversion.

  The camera 2 acquires n pieces of image data by performing photographing a plurality of times at a predetermined timing. The n pieces of image data correspond to n frames constituting the moving image. For example, in moving image shooting including a series of multiple shooting operations, the Nth frame is obtained by the Nth shooting. The Nth frame is, for example, an array of pixels in the form of a matrix composed of A pixels horizontally (A is a natural number, for example, A = 1280) and B pixels vertically (B is a natural number, for example, B = 960). Consists of. Data relating to n frames obtained by the camera 2 is transmitted to the information processing device 4 via the data line CB.

  The distance measuring device 3 is a device that is attached to the camera 2 while maintaining the positional relationship and posture with respect to the camera 2 and measures the distance to the subject using a laser, for example. The distance measuring device 3 measures the distance from the camera 2 (specifically, the image sensor) to each part of the subject, thereby measuring the distance from the camera 2 to each part of the subject at the time of each photographing. The

  For example, information (also referred to as Nth distance information) indicating the distance (also referred to as subject distance) from the camera 2 (that is, the Nth viewpoint) to each part of the subject at the time of the Nth shooting is obtained. In other words, the Nth distance information includes information indicating the distance (subject distance) between the portion of the subject captured by each pixel constituting the Nth frame and the Nth viewpoint. The n pieces of distance information obtained by the distance measuring device 3 corresponding to the n frames are transmitted to the information processing device 4 through the data line CB.

  The information processing apparatus 4 is configured by, for example, a personal computer (personal computer), and includes an operation unit 41, a display unit 42, and an interface (I / F) unit 43. The operation unit 41 includes a mouse and a keyboard, the display unit 42 includes a liquid crystal display and the like, and the I / F unit 43 receives data from the camera 2 and the distance measuring device 3. In addition, the information processing apparatus 4 includes a storage unit 44, an input / output unit 45, and a control unit 46.

  The storage unit 44 is configured by a hard disk or the like, for example, and stores each image obtained by the camera 2. The storage unit 44 stores a program PG for realizing various operations in the information processing apparatus 4. These various operations include an operation of creating a stereoscopically viewable moving image.

  The input / output unit 45 includes, for example, a disk drive, receives the storage medium 9 such as an optical disk, and exchanges data with the control unit 46.

  The control unit 46 includes a CPU 46 a that functions as a processor and a memory 46 b that temporarily stores information, and comprehensively controls each unit of the information processing apparatus 4. Further, the control unit 46 reads and executes the program PG in the storage unit 44, thereby realizing various functions and various information processing. Data temporarily generated in this information processing is appropriately stored in the memory 46b. Then, under the control of the control unit 46, the information processing apparatus 4 functions as an image processing apparatus that creates a stereoscopically viewable moving image. Note that the program stored in the storage medium 9 can be stored in the memory 46 b via the input / output unit 45.

<(3) Functional configuration related to image processing>
FIG. 3 is a block diagram illustrating a functional configuration of the image processing apparatus realized by the control unit 46. The information processing apparatus 4 includes an image acquisition unit 461, a distance information acquisition unit 462, an area detection unit 463, a target parallax setting unit 464, an initial parallax setting unit 465, a parallax adjustment unit 466, a virtual viewpoint image generation unit 467, and a stereoscopic image A generation unit 468 is provided.

  The image acquisition unit 461 acquires a moving image including a plurality of frames stored in the storage unit 44 or the like. For example, the moving image is configured to include a plurality of frames acquired by the camera 2 performing a plurality of shootings at a predetermined timing. The plurality of frames corresponds to, for example, n frames (n is a natural number of 2 or more) constituting a moving image.

  The distance information acquisition unit 462 is distance information indicating the distance from the camera 2 to the subject when the Nth frame constituting the moving image stored in the storage unit 44 or the like is captured (also referred to as Nth distance information). To get. The distance from the camera 2 to the subject is the distance from the camera 2 (specifically, the image sensor) to the portion of the subject captured by each pixel of the Nth frame when the Nth frame is captured. .

  The region detection unit 463 includes an Nth frame (Nth frame) of the n frames and an Npth frame (p is a natural number) of the same n frames (Nth frame). ). By this comparison, the region detection unit 463 detects a region (also referred to as an adjusted region) where the parallax should be adjusted in the Nth frame. Note that p is preferably 1 from the viewpoint of recognizing a detailed subject change.

  The target parallax setting unit 464 sets a target parallax (also referred to as target parallax) based on a predetermined rule for each pixel for the Nth frame included in the n frames. The target parallax is predicted to be obtained from the Nth virtual viewpoint (also referred to as the Nth virtual viewpoint) with the Nth frame captured from the Nth viewpoint as a reference. This corresponds to the shift amount of each pixel for generating (also referred to as an Nth virtual viewpoint image or an Nth virtual viewpoint frame). The relationship between the Nth viewpoint and the Nth virtual viewpoint is such that if the position of the Nth viewpoint is the position of the left eye of the person, the position of the Nth virtual viewpoint is the position of the right eye of the same person. Therefore, for example, a condition can be considered in which a virtual straight line (also referred to as a virtual straight line) connecting the Nth viewpoint and the Nth virtual viewpoint is orthogonal to the shooting direction of the camera 2. Note that the Nth viewpoint and the Nth virtual viewpoint may be arranged on a virtual circumference surrounding the subject, for example, or other arrangements may be adopted.

  The initial parallax setting unit 465 sets an initial parallax for each pixel with respect to a frame (also referred to as an appearance frame) in which the adjusted area appears among the n frames for each adjusted area. The initial parallax may be anything corresponding to a predetermined number (for example, 0) of pixels in the −X direction (or + X direction). For example, when the distance from the Nth viewpoint and the Nth virtual viewpoint to the subject is short, the subject viewed with the right eye shifts to the left side with respect to the subject viewed with the left eye, so the initial parallax is 0 or more in the −X direction. It is preferable to set the number of pixels. When the distance from the Nth viewpoint and the Nth virtual viewpoint to the subject is long, for example, the initial parallax is 0 in the + X direction in order to strongly give the user a sense that the distance to the subject is far in the 3D image. A mode in which the target parallax is set in the + X direction with the number of pixels set as described above is also conceivable.

  The parallax adjustment unit 466 adjusts the parallax of each pixel from the initial parallax toward the target parallax for a plurality of frames subsequent to the appearance frame of the n frames for each adjusted region. The parallax adjustment from the initial parallax toward the target parallax varies depending on various conditions. Various conditions include the size of the target parallax in the appearance frame, the difference between the adjusted parallax and the target parallax, the position of the adjusted area in the frame, the distance from the camera 2 to the object captured in the adjusted area, and the like. Can be mentioned.

  The virtual viewpoint image generation unit 467 generates n virtual viewpoint frames for the n frames based on the parallax information adjusted by the parallax adjustment unit 466. That is, n right-eye frames are generated from n left-eye frames, respectively.

  The stereoscopic image generation unit 468 is a moving image (3D moving image) that can be stereoscopically viewed based on the n left-eye frames and the n right-eye frames generated by the virtual viewpoint image generation unit 467. Is generated. The generated 3D moving image data is output to, for example, the storage unit 44 or the input / output unit 45 and stored in the storage unit 44 or the storage medium 9.

<(4) Method for detecting adjustment area>
Here, a method of detecting the adjusted region by the region detection unit 463 will be described.

  When the N-th frame and the N-p frame are compared, for example, when N = 1 and p = 1, there is no N-p frame. At this time, the Nth frame is the first frame of the moving image, and the entire area in the Nth frame is detected as the adjusted area.

  For example, when N is 2 or more and p = 1, the region detection unit 463 searches for a corresponding point between the N-th frame and the N-th frame, An area (also referred to as an appearance area) in which a portion of a subject that does not exist but is present in the Nth frame is detected. Here, as a search for corresponding points, for example, processing using a phase-only correlation method (POC method), template matching, or the like can be cited.

  Furthermore, the appearance area in the Nth frame is distinguished for each object based on the Nth distance information, so that an area indicating an object that did not exist in the Np frame is detected as the adjusted area. Thereby, the parallax can be adjusted as necessary. In addition, as a method of classifying the appearance region for each object, for example, when the subject distance is within a predetermined threshold (for example, 10 cm) for pixels adjacent to each other, the pixels adjacent to each other represent the same object. There is a method of recognizing the captured pixel.

  In addition, the region detection unit 463 recognizes an appearance frame in which the adjusted region appears for each adjusted region among the n frames.

<(5) Target parallax setting method>
Here, a target parallax setting method by the target parallax setting unit 464 will be described.

  Examples of the predetermined rule related to the setting of the target parallax include a rule using the principle of triangulation. This rule is based on the separation distance between the Nth viewpoint and the Nth virtual viewpoint at the time of shooting the Nth frame, and the distance (subject distance) from the camera 2 to each part of the subject when the Nth frame is shot. From this, the target parallax is set.

  Specifically, as shown in FIG. 4, the distance from the camera 2 (that is, the Nth viewpoint) to the subject 20B is D, the separation distance between the Nth viewpoint and the Nth virtual viewpoint is B, and the lens of the camera 2 is When the focal length is f, and the shift amount (target parallax) of the pixels capturing the same portion of the subject between the Nth frame and the Nth virtual viewpoint frame is Δd, the relationship of the following expression (1) is established.

    D = f × B / Δd (1).

  In the above formula (1), the separation distance B is uniquely determined according to the separation distance between the standard human right eye and left eye, for example. The focal length f is determined by the design of the camera 2. The distance D is given by the Nth distance information related to the Nth frame. For this reason, the target parallax Δd is obtained for each pixel of the Nth frame.

<(6) Specific example of parallax adjustment>
Hereinafter, the parallax adjustment performed by the parallax adjustment unit 466 will be described with a specific example.

<(6-1) Adjustment of parallax according to target parallax size>
<(6-1-1) Outline of parallax adjustment>
FIG. 5 schematically shows a specific example of the N-th right eye frame G _N R generated according to the target parallax as it is when the target parallax of the adjusted area in the appearance frame in which the adjusted area appears is large. FIG. The N-th right-eye frames G _N R shown in Figure 5, (also referred to as an object region) is a region captured person corresponding to the adjusted region OB _N and, occlusion area OC _N corresponding to the area OB _N And are included. In FIG. 5, a relatively large occlusion region OC _N is shown.

  Here, the user has an increase in the occlusion area between a case where a frame in which no occlusion area exists is used as a reference and a case in which a frame in which an occlusion area already exists is used as a reference. Impression is different. Therefore, when the target parallax is large, the parallax adjustment amount between temporally adjacent frames (also referred to as parallax adjustment amount per frame) is gradually increased as the occlusion area increases. The uncomfortable feeling given to the user is reduced. Then, in the number of frames corresponding to a relatively long time, it is preferable that the parallax related to the adjusted region is adjusted from the initial parallax toward the target parallax.

FIG. 6 schematically shows a specific example of the N-th right eye frame G _N R generated according to the target parallax as it is when the target parallax of the adjusted area in the appearance frame in which the adjusted area appears is small. FIG. In the N-th right eye frame G _N R shown in FIG. 6, similarly to the case shown in FIG. 5, a region (also referred to as an object region) OB _{N in} which a person corresponding to the adjusted region is captured, An occlusion area OC _N corresponding to the area OB _N is included.

However, since the occlusion area OC _N shown in FIG. 6 is relatively small, it is unlikely that the user will feel uncomfortable. Therefore, under such conditions, the parallax related to the adjusted area is adjusted from the initial parallax toward the target parallax in a number of frames corresponding to a relatively short time. Thereby, it is possible to suppress both a sense of incongruity and the time required for parallax adjustment. However, since an excessively rapid change in the parallax adjustment amount gives the user a sense of incongruity, for example, it is preferable that the parallax adjustment amount per frame is suppressed to be low in at least one of the initial stage and the final stage when the parallax is adjusted. . That is, it is preferable that the parallax is adjusted relatively slowly in the initial and final stages.

<(6-1-2) Determination of target parallax size>
As a method for determining whether or not the target parallax is large, for example, the distance used to set the target parallax, that is, the distance from the camera 2 to the object captured in the adjusted area is a predetermined threshold (for example, If the value is less than 1 m), a method for determining that the target parallax is large is conceivable.

Further, for example, the number of pixels corresponding to the lateral width along the X direction of the object region OB _N (also referred to as the object width), the method being determined is employed on the basis of the ratio between the shift amount of the pixel corresponding to the target parallax May be. In this method, for example, if the value obtained by dividing the shift amount by the object width is equal to or greater than a predetermined threshold (for example, 0.1), it is determined that the target parallax is large.

Further, for example, the area of the object region OB _N, the method being determined based on a ratio of the area of the occlusion area OC _N may be employed. In this way, for example, an occlusion region OC _N area the object region OB _N area divided by the value predetermined threshold (e.g., 0.1) if more, it is determined that the target parallax is large.

  When it is not determined that the target parallax is large, it is determined that the target parallax is small.

<(6-1-3) When the target parallax is large>
Here, a simple example in which the target parallax is a constant value when the parallax is adjusted will be described in order to prevent the description from becoming complicated.

FIG. 7 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the target parallax in the adjusted region of the appearing frame is large. In FIG. 7, the horizontal axis indicates the passage of time, and the vertical axis indicates the parallax. The passage of time indicates the passage of time with reference to the time of the appearance frame in the moving image. The target parallax da _{ob of the} adjusted area in the appearance frame is indicated by a straight line L10 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L11 drawn by a thick solid line. Yes. In addition, a predetermined threshold d _TH for determining whether or not the target parallax is large is shown.

  As the parallax adjustment corresponding to the curve L11 in FIG. 7, for example, an operation in which the following steps (A1) to (A3) are sequentially performed is conceivable.

(A1) When parallax adjustment is started and the evaluation value V _i is equal to or less than the threshold value α, a state in which the parallax adjustment amount per frame is relatively small is maintained. (A2) When the evaluation value V _i exceeds the threshold α, the amount of parallax adjustment per frame increases. (A3) When the evaluation value V _i exceeds the threshold value β, the amount of parallax adjustment per frame decreases.

Here, the evaluation value V _i may be, for example, the time that has elapsed since the start of parallax adjustment, the number of frames since the start of parallax, or the parallax after adjustment. It may be. Further, the threshold values α and β may be fixed values determined in advance, respectively, the time that has elapsed since the start of parallax adjustment, the number of frames since the start of parallax, and the post-adjustment It may be calculated from at least one value of parallax. For example, it can be considered that the evaluation value V _i is the adjusted parallax and the threshold values α and β are a predetermined number% of the target parallax. As a specific example, a case where the threshold value α is 20% of the target parallax and the threshold value β is 80% of the target parallax can be considered.

Thus, in the period in which the adjustment of the parallax is carried out, the first period A1 of the evaluation value V _i is 0 up to the threshold value alpha, the evaluation value V _i exceeds the threshold alpha up to the threshold β Two periods A2 and a third period A3 after the evaluation value V _i exceeds the threshold value β are included. For example, the parallax adjustment amount per frame in the first period A1 is a constant value a1, the parallax adjustment amount per frame in the second period A2 is a constant value a2, and 1 in the third period A3. A case where the amount of parallax adjustment per frame is a constant value a3 can be considered.

However, the parallax adjustment amount per frame is gradually changed in a certain period including before and after the evaluation value V _i reaches the threshold values α and β from the viewpoint of giving the user as much discomfort as possible. preferable. That is, as shown by a curve L11 in FIG. 7, it is preferable that the parallax change in a curve by gradually changing the parallax adjustment amount. As a result, the parallax gradually reaches the target parallax after increasing in an accelerated manner.

In the curve L11 in FIG. 7, the parallax reaches from the initial parallax d0 to d1a in the first period A1 in which the elapsed time is T0 to T1a, and the parallax from d1a in the second period A2 in which the elapsed time is T1a to T2a. reaches d2a, disparity in the third period A3 elapsed time is T2a~T3a is shown an example to get from d2a to da _ob. That is, the parallax of each pixel is directed from the initial parallax to the target parallax for a plurality of frames related to the first period A1, the second period A2, and the third period A3 following the appearance frame of the n frames. Adjusted.

  More specifically, the parallax change rate (also referred to as first change rate) obtained by dividing the parallax change amount in the first period A1 by the length of the first period A1 is the parallax change rate in the second period A2. It is smaller than the parallax change rate (also referred to as second change rate) obtained by dividing the change amount by the length of the second period A2. By adjusting the parallax as described above, it is possible to reduce a visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint. Further, the second change rate is larger than the parallax change rate (also referred to as the third change rate) obtained by dividing the parallax change amount in the third period A3 by the length of the third period A3. The second rate of change is preferably several times to several tens of times the first rate of change and the third rate of change, for example.

  By the way, the parallax change of the parallax as shown by the curve L11 can be realized by various methods. Examples of the various methods include a method using a pattern in which the parallax changes in a curve (also referred to as an adjustment pattern) that is prepared in advance, and a method of changing the parallax according to a predetermined function.

  First, as a method of adjusting the parallax according to the adjustment pattern, for example, the parallax of each pixel is adjusted along the adjustment pattern according to the difference between the initial parallax and the target parallax among a plurality of types of adjustment patterns prepared in advance. It may be possible. In this case, each type of adjustment pattern may be indicated by data in which the parallax changes in a curve from the initial parallax toward the target parallax over time, and may be stored in advance in the storage unit 44. And this adjustment pattern should just be expanded-contracted in the direction of passage of time according to the period required for the parallax adjustment determined according to the difference between the initial parallax and the target parallax, for example.

  By adjusting parallax according to such an adjustment pattern, accurate parallax adjustment can be easily performed. When the number of frames per second in the moving image is 60, that is, when the frame rate is 60 frames per second, the number of frames required for parallax adjustment is, for example, a relatively large number such as about 120 to 180. Become.

  Next, as a method of adjusting the parallax according to a predetermined function, for example, when the Nth Nth frame of the n frames is an appearance frame of the adjusted region related to the parallax adjustment, A mode in which the parallax of the (N + 1) th and subsequent frames is adjusted based on the initial parallax and the target parallax can be considered. Further, for example, a mode in which the parallax of the (N + 1) th and subsequent frames is adjusted based on the adjusted parallax and the target parallax in the Nth Nth frame among the n frames is conceivable. By adjusting the parallax according to such a function, an appropriate parallax can be adjusted.

As a specific example of the parallax adjustment method according to a predetermined function, PID control based on the difference between the initial parallax and the target parallax and the difference between the parallax of the adjusted Nth frame and the target parallax is possible. Conceivable. In the PID control, for example, every time the evaluation value V _i reaches the threshold values α and β, the integral term coefficient, the differential term coefficient, and the proportional term coefficient may be changed. As a result, the parallax is adjusted so as not to reach the target parallax immediately, but to reach the target parallax from the initial parallax with a certain delay time.

  By the way, threshold values such as threshold values α and β are not used, and the parallax may be adjusted according to a predetermined parameter. Examples of the predetermined parameter include the time already required for parallax adjustment, the time used for parallax adjustment, the relationship between the adjusted parallax and the target parallax, and the like. For example, a mode in which the amount of parallax adjustment is calculated by substituting the time already required for parallax adjustment into a function is conceivable. Further, for example, a mode in which a time used for parallax adjustment is set according to the size of the target parallax, and a curved adjustment pattern indicating the relationship between the passage of time and the parallax is expanded and contracted according to the time is considered. It is done. Further, for example, a mode in which a curvilinear adjustment pattern indicating the relationship between the passage of time and the parallax is expanded or contracted according to the relationship between the adjusted parallax and the target parallax is conceivable.

<(6-1-4) When the target parallax is small>
Here, in order to prevent the description from becoming complicated, a simple example in which the target parallax is a constant value when the parallax is adjusted will be described.

FIG. 8 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the target parallax in the adjusted region of the appearing frame is small. In FIG. 8, as in FIG. 7, the horizontal axis indicates the passage of time and the vertical axis indicates the parallax, and the passage of time indicates the passage of time with reference to the time of the appearing frame in the moving image. The target parallax db _{ob of the} adjusted area in the appearance frame is indicated by a straight line L20 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L21 drawn by a thick solid line. Yes. In addition, a predetermined threshold value d _TH for determining the magnitude of the target parallax is shown.

  As the adjustment of the parallax corresponding to the curve L21 in FIG. 8, for example, an operation in which the following steps (B1) to (B3) are sequentially performed can be considered as in the case of the parallax adjustment corresponding to the curve L11 in FIG.

(B1) When parallax adjustment is started and the evaluation value V _i is equal to or less than the threshold value α, a state in which the parallax adjustment amount per frame is relatively small is maintained. (B2) When the evaluation value V _i exceeds the threshold value α, the amount of parallax adjustment per frame increases. (B3) When the evaluation value V _i exceeds the threshold value β, the amount of parallax adjustment per frame decreases.

Here, the evaluation value V _i and the threshold values α and β may be values of the types described above in the description of parallax adjustment when the target parallax is large. Here, an example in which the evaluation value V _i is the adjusted parallax, the threshold α is 5% of the target parallax, and the threshold β is 80% of the target parallax can be considered.

Thus, in the period in which the adjustment of the parallax is carried out, the first period B1 from evaluation value V _i is 0 up to the threshold value alpha, the evaluation value V _i exceeds the threshold alpha up to the threshold β Two periods B2 and a third period B3 after the evaluation value V _i exceeds the threshold value β are included. The amount of parallax adjustment per frame may be, for example, a constant amount b1 in the first period B1, a constant amount b2 in the second period B2, and a constant amount b3 in the third period B3. However, from the viewpoint of giving the user a sense of incongruity as much as possible, as shown by a curve L21 in FIG. 8, the parallax per frame in a certain period including before and after the evaluation value V _i reaches the threshold values α and β. It is preferable that the adjustment amount is gradually changed.

In the curve L21 in FIG. 8, the parallax reaches from the initial parallax d0 to d1b in the first period B1 where the elapsed time is T0 to T1b, and the parallax starts from d1b in the second period B2 where the elapsed time is T1b to T2b. reaches d2b, disparity in the third period B3 elapsed time is T2b~T3b is shown an example that reaches d3 _ob from d2b. Here, the parallax curve change with respect to time as shown by the curve L21 is, for example, a method for realizing the parallax curve change with respect to time as shown by the curve L11 in FIG. It is realizable by the same method. Here, for example, when parallax adjustment according to an adjustment pattern is adopted, accurate parallax adjustment can be easily performed. If the frame rate of a moving image is 60 frames per second, the number of frames required for parallax adjustment Is a relatively small number, for example, about 30.

<(6-1-5) Operation flow>
FIG. 9 and FIG. 10 are flowcharts illustrating an operation flow relating to parallax adjustment according to the target parallax size. This operation flow is realized by reading and executing the program PG in the storage unit 44 by the control unit 46. For example, the generation of a 3D moving image in the information processing apparatus 4 is requested in accordance with the operation of the operation unit 41 by the user, and this operation flow is started.

  In step S1 of FIG. 9, the image acquisition unit 461 acquires a moving image including n frames stored in the storage unit 44 or the like.

  In step S2, the distance information acquisition unit 462 acquires distance information corresponding to each of the n frames acquired in step S1 from the storage unit 44 or the like. This distance information is information indicating the distance from the camera 2 to the subject when each of n frames constituting the moving image is photographed. Here, an example is shown in which the process of step S1 and the process of step S2 are executed in parallel, but the process of step S1 and the process of step S2 may be executed in this order. These steps may be executed in the reverse order.

  In step S3, the region detection unit 463 compares the Nth frame of the n frames with the Np frame (p is a natural number) of the same n frames. A region to be adjusted in the Nth frame is detected. Here, N is set in order from 1 to n, and among the Nth frames, areas indicating objects that did not exist in the Npth frame are detected as adjusted areas. When N = 1, an area indicating all objects in the first frame is detected as an adjusted area. At this time, identification information (here, an identification number) is attached to each adjusted area.

  In step S4, the target parallax setting unit 464 sets the target parallax for each pixel based on a predetermined rule for each frame included in the n frames.

  In step S5, the initial parallax setting unit 465 sets the identification number M of the adjustment target area to be processed to 1.

  In step S6, the initial parallax setting unit 465 designates the M-th adjusted region among all the adjusted regions detected in step S3, and the process proceeds to step S11 in FIG.

  In step S11, the initial parallax setting unit 465 sets the initial parallax for each pixel of the appearing frame in which the adjusted area appears for the adjusted area specified in step S6.

In step S12, the parallax adjustment unit 466 determines whether or not the target parallax for the adjusted area specified in step S6 is _{equal to} or greater than a predetermined threshold value _dTH . Here, for example, it is determined whether or not the minimum value of the target parallax set in step S4 for the adjusted region specified in step S6 is equal to or greater than a predetermined threshold value _dTH . If the target parallax is greater than or equal to the predetermined threshold d _TH , the process proceeds to step S13, and if the target parallax is less than the predetermined threshold d _TH , the process proceeds to step S14. Here, the determination is performed using the minimum value of the target parallax. However, the determination is not limited to this. For example, the maximum value of the target parallax or the average value of the target parallax may be used. The determination may be made using the calculated parameter.

  In step S13, the parallax adjustment unit 466 performs parallax adjustment when the target parallax is large.

  In step S14, the parallax adjustment unit 466 performs parallax adjustment when the target parallax is small.

  In step S15, the initial parallax setting unit 465 determines whether all the adjustment areas detected in step S3 have already been specified in step S6. If all the adjusted areas have already been specified, the process proceeds to step S16. If all the adjusted areas have not been specified yet, the process proceeds to step S17.

  In step S16, the initial parallax setting unit 465 sets the identification number M of the adjustment target area to be processed to one larger value, and the process proceeds to step S6 in FIG.

  In step S <b> 17, n virtual viewpoint frames are generated by the virtual viewpoint image generation unit 467 based on the parallax information adjusted in steps S <b> 13 and S <b> 14 for n frames constituting the moving image.

  In step S18, the stereoscopic image generating unit 468 can perform stereoscopic viewing (3D moving image) based on the n frames acquired in step S1 and the n virtual viewpoint frames generated in step S17. Is generated. Thereby, this operation flow is completed.

<(6-1-6) When an upper limit is set for the adjustment amount>
FIG. 11 is a schematic view illustrating an aspect of parallax adjustment related to a virtual viewpoint image. If the amount of parallax adjustment per frame is large, for example, the first to third right eye frames corresponding to the first to third virtual viewpoint frames, as shown by the left route in FIG. In G ₁ C to G ₃ C, the occlusion area increases rapidly. Thereafter, the parallax in the N-th right eye frame G _N R corresponding to the N-th virtual viewpoint frame becomes the target parallax. In such a case, the amount of parallax adjustment per frame becomes too large, increasing the burden on the user's eyes.

Therefore, a method is conceivable in which the parallax adjustment amount per frame is suppressed to a certain amount or less for the purpose of giving the user's eyes a gentle and uncomfortable feeling. For example, when parallax adjustment is performed according to the above-described target parallax size, an upper limit value may be provided for the parallax adjustment amount per frame. At this time, as indicated by the right route in FIG. 11, for example, the occlusion area gradually increases in the _{first to} third right eye frames G ₁ CR to G ₃ R corresponding to the first to third virtual viewpoint frames. .

  In such a configuration, for example, the parallax adjustment unit 466 adjusts the parallax of each pixel so that the parallax adjustment amount per frame does not exceed the upper limit value. Thereby, since the remarkable change of parallax is suppressed, the visual discomfort given to a user is suppressed appropriately.

FIG. 12 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when an upper limit is provided for the parallax adjustment amount per frame. In FIG. 12, as in FIGS. 7 and 8, the horizontal axis indicates the passage of time and the vertical axis indicates the parallax, and the passage of time is the passage of time with reference to the time of the appearing frame in the moving image. Indicates. The target parallax dc _{ob of the} adjusted area in the appearance frame is indicated by a straight line L30 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L31 drawn by a thick solid line. Yes. For comparison, the relationship between the passage of time and the adjusted parallax when the upper limit value is not set for the parallax adjustment amount per frame is indicated by a curved line L32 drawn by a one-dot chain line. Yes.

  If no upper limit is set for the parallax adjustment amount per frame, the upper limit is set for the parallax adjustment amount per frame even when the parallax increases rapidly as shown by the curve L32. Is provided, the parallax does not increase abruptly as indicated by the curve L31. Such control can be realized by simple control in which, for example, when the parallax adjustment amount exceeds the upper limit value, the adjustment amount derived based on the function or the adjustment pattern is replaced with the upper limit value. . If an upper limit is set for the amount of parallax adjustment per frame, for example, if the frame rate is 60 frames per second, the number of frames required for parallax adjustment is relatively large, for example, about 120 to 180 Number.

  Further, for example, a method of controlling the parallax adjustment amount per frame to be equal to or less than the upper limit value by expanding and contracting a curvilinear adjustment pattern indicating the relationship between the passage of time and the parallax may be considered.

<(6-1-7) Adjustment of parallax according to change in target parallax>
In the parallax adjustment described above, a simple example in which the target parallax is a constant value has been described, but in an actual movie, the target parallax changes over time even for the same object. Is common. Therefore, the amount of parallax adjustment per frame may be changed based on the adjusted parallax and the target parallax related to the frame that is temporally ahead of the frame related to the adjusted parallax. .

  For example, the parallax adjustment unit 466 uses the target parallax of each pixel in the N + Q-th frame (N and Q are natural numbers) out of n frames constituting the moving image, and the parallax adjustment unit 466 The parallax of the (N + 1) th and subsequent frames may be adjusted based on the parallax of the Nth frame (Nth frame) that has already been adjusted. Here, when the frame rate is 60 frames per second, for example, a configuration in which Q is set to about 1 to 60 is conceivable. Thereby, it is possible to easily adjust the parallax corresponding to the change in the target parallax.

Specifically, for each pixel in the adjusted region, when the difference between the adjusted parallax for the Nth frame and the target parallax in the N + Q frame is equal to or greater than a threshold, the amount of parallax adjustment per frame is increased. On the contrary, when the difference is less than the threshold value, a configuration in which the amount of parallax adjustment per frame is reduced can be considered. For example, when the target parallax in the N + Q frame is Occ _{N + Q} , the adjusted parallax in the Nth frame is Occ _N, and the threshold is δ _th , When the parallax adjustment amount per frame is increased and the following equation (2) is not satisfied, the parallax adjustment amount per frame is decreased.

(Occ _{N + Q} −Occ _N ) ≧ δ _th (2).

  Not only the amount of parallax adjustment per frame but also the time required for parallax adjustment may be changed.

  However, such parallax adjustment is preferably performed after, for example, the above-described parallax adjustment when the target parallax is large or the parallax adjustment when the target parallax is small. Thereby, the visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint is reduced.

  FIG. 13 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the parallax is adjusted according to the target parallax that varies with the passage of time. In FIG. 13, the target parallax that fluctuates with the passage of time is indicated by a curve L40 drawn with a thick broken line, and the relationship between the elapsed time and the adjusted parallax is shown by a curve L41 drawn with a thick solid line. Has been.

  Further, considering that the target parallax increases and decreases with the passage of time, it is preferable that the parallax adjustment in the direction in which the parallax increases and the parallax adjustment in the direction in which the parallax decreases are performed, respectively. The adjustment of the parallax in the direction in which the parallax decreases may be performed according to the amount of change in the difference between the target parallax and the adjusted parallax, for example.

  Specifically, such parallax adjustment according to the target parallax variation is realized by, for example, PID control based on the parallax adjusted for the Nth frame and the target parallax in the N + Q frame. More specifically, the parallax is adjusted according to a function including an integral term, a differential term, a proportional term, and the like according to the difference between the adjusted parallax for the Nth frame and the target parallax in the N + Q frame. A possible embodiment is conceivable.

  It should be noted that, for example, an arithmetic value such as an average value and a center value for the target parallax related to the (N + Q) th frame and the target parallax related to one or more frames that are temporally adjacent to the (N + Q) th frame, The parallax may be adjusted according to the difference from the adjusted parallax.

<(6-1-8) Change in parallax adjustment according to position on screen>
Furthermore, the parallax of each pixel may be adjusted according to the position of the subject captured in each frame. Thereby, the parallax can be adjusted according to the degree of attention of the user.

  For example, the vicinity of the center of each frame in the moving image tends to be the most watched area by the user. For this reason, it is considered that the occurrence of the occlusion area near the center of each frame tends to give the user a sense of incongruity. From such a viewpoint, it is preferable that parallax adjustment is performed so that the parallax gradually increases without increasing the parallax for each pixel in the adjusted region located near the center of each frame. . On the other hand, it is preferable that parallax adjustment is performed so that the parallax increases relatively quickly and approaches the target parallax for each pixel in the adjusted region located near the end of each frame.

FIG. 14 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image. As shown in FIG. 14, for example, in the N-th right-eye frame G _N R, the parallax gradually increases in the adjusted region OB1 _N where the object is captured near the center, and the object is near the end. For the captured adjustment area OB2 _N , a mode in which parallax adjustment is performed such that parallax increases relatively quickly can be considered.

  FIGS. 15 and 16 are graphs schematically illustrating the relationship between time and adjusted parallax when the parallax is adjusted according to the position of the adjusted region in the frame. In FIG. 15, the target parallax is indicated by a straight line L10 drawn with a thick broken line, and the relationship between the elapsed time and the adjusted parallax for the adjusted region near the center of the frame is indicated by a curved line L11 drawn with a thick line. ing. Further, in FIG. 16, the target parallax is indicated by a straight line L10 drawn with a thick broken line, and a curve L13 in which the relationship between the elapsed time and the adjusted parallax for the adjusted area near the end of the frame is drawn with a thick line. It is shown in

  Such parallax adjustment is performed, for example, according to the position of the adjusted region in the appearance frame in which the adjusted region appears. More specifically, the frame may be divided into a plurality of regions from the vicinity of the center to the vicinity of the end portion, and different parallax adjustment may be performed for each pixel in the adjusted region for each of the divided regions. In addition, it is preferable that the number of areas to be divided is large because the user does not feel uncomfortable. Further, the parallax adjustment may be switched for each divided area, but the parallax adjustment amount may be calculated according to the position information (for example, address) of the pixel on the frame.

  Furthermore, the amount of parallax adjustment is not only position information but also information on at least one of adjusted parallax, changed target parallax, time already used for parallax adjustment, time required for parallax adjustment, etc. A mode determined according to the above is also conceivable. In addition, an adjustment pattern in which the parallax changes in a curve as time passes is prepared for each area divided for the frame, the adjustment pattern is selected according to the position information of the to-be-adjusted adjustment area, and the selected adjustment pattern The parallax may be adjusted according to the above. The adjustment pattern may be expanded or contracted in at least one of the time direction and the magnitude direction of the parallax, depending on the time required for parallax adjustment, the size of the target parallax, and the like.

  Moreover, in a moving image, it is possible that the position of the to-be-adjusted area | region in a flame | frame changes as time passes. At this time, the parallax adjustment content may be changed in accordance with the change in the position of the adjustment region in the frame. For example, when a region to be adjusted positioned near the center of the frame moves to the vicinity of the end of the frame, the parallax may be adjusted according to the vicinity of the end.

  Note that the parallax adjustment performed according to the position of the adjusted region in such a frame depends on the parallax adjustment when the target parallax is small, the parallax adjustment when the target parallax is large, and the variation of the target parallax. It may be appropriately combined with one or more parallax adjustments among the parallax adjustments.

<(6-1-9) Grouping of Adjustment Areas>
Further, the parallax of each pixel may be adjusted for another subject different from the one subject in accordance with the parallax related to the one subject captured in the moving image. Thereby, since the consistency of parallax between objects improves, the visual discomfort given to a user is suppressed appropriately.

  For example, when the distances from the camera 2 to a plurality of objects as subjects are substantially the same, in the virtual viewpoint image, substantially the same parallax may be set for the adjusted areas related to the plurality of objects. preferable. If a completely different parallax is set for the adjusted regions related to the plurality of objects, the user is visually uncomfortable. Therefore, based on the distance information acquired by the distance information acquisition unit 462, a plurality of adjusted areas related to a plurality of objects having substantially the same distance from the camera 2 are grouped, and the same parallax adjustment is performed. It is preferable.

FIG. 17 is a schematic diagram illustrating an aspect of parallax adjustment related to a virtual viewpoint image in a case where parallax is adjusted by grouping a plurality of adjusted areas according to distance information. As shown in FIG. 17, for example, first to third adjusted regions OB1 _N to 3 in which three people having the same distance from the camera 2 in the N-th right eye frame G _N R are captured. for OB3 _N, conceivable manner such as the adjustment of the same parallax is performed.

  FIG. 18 is a graph schematically illustrating the relationship between time and adjusted parallax when a plurality of adjusted areas are not grouped for comparison. FIG. 19 is a graph schematically illustrating the relationship between time and adjusted parallax when a plurality of adjusted areas are grouped.

In FIG. 18, the target parallax is drawn with a thick broken line L50, and the relationship between the elapsed time and the adjusted parallax is drawn with a solid curve L51 as the relationship relating to the first adjusted area OB1 _N. The relationship related to the adjusted region OB2 _N is drawn with a bold curve L52, and the relationship related to the third adjusted region OB3 _N is drawn with a dashed-dotted curve L53. And about the adjustment of the parallax shown by the curves L51-L53, even if the elapsed time is the same, considerable variation occurs in the parallax after the adjustment. In this case, the user is visually uncomfortable.

Therefore, as shown in FIG. 19, the parallax adjustment related to the first to third adjusted areas OB1 _{N to} OB3 _N is performed in a unified manner. Is preferred. Thereby, since the consistency of parallax between objects improves, the visual discomfort given to a user is suppressed appropriately.

  By the way, in such a configuration, for example, a mode in which a plurality of adjusted regions are grouped with respect to an object whose distance from the camera 2 is within a predetermined value (for example, ± several tens of centimeters) can be considered. In addition, a mode in which a plurality of adjusted areas in which the target parallax is included in a predetermined range (for example, ± several pixels) can be considered.

  Then, the amount of parallax adjustment is determined according to at least one or more information among the adjusted parallax, the changed target parallax, the time already used for parallax adjustment, the time required for parallax adjustment, and the like. A possible embodiment is possible. The parallax may be adjusted according to an adjustment pattern in which the parallax changes in a curve with time. The adjustment pattern may be expanded or contracted in at least one of the time direction and the magnitude direction of the parallax, depending on the time required for parallax adjustment, the size of the target parallax, and the like.

  In addition, among the plurality of adjusted areas, for example, a mode in which the adjusted area used as a reference for parallax adjustment is the adjusted area closest to the center of the frame is conceivable. In addition, if distance information is obtained based on the principle of triangulation using a stereo image, the evaluation value indicating the highest probability of searching for a corresponding point in the stereo image is the highest among the plurality of adjusted regions. The adjustment area may be used as a reference for parallax adjustment. As the evaluation value, when the corresponding point search process is performed by POC (phase-only correlation method), a value (for example, an average value, an integrated value, etc.) corresponding to the peak value of the POC value for the adjusted region is used. ) May be adopted as the evaluation value.

  Note that the parallax adjustment performed by grouping a plurality of adjusted areas according to such distance information includes parallax adjustment when the target parallax is small, parallax adjustment when the target parallax is large, and target parallax. One or more parallax adjustments among the parallax adjustment according to the fluctuation and the parallax adjustment according to the position in the frame may be appropriately combined.

<(7) Modification>
It should be noted that the present invention is not limited to the above-described embodiment, and various changes and improvements can be made without departing from the gist of the present invention.

  For example, in the above-described embodiment, an example in which the parallax is adjusted according to the magnitude of the target parallax, the fluctuation of the target parallax, the position on the screen, and the distance information is given. However, the present invention is not limited to this, and other parallax adjustment methods may be employed. For example, the parallax may be adjusted by increasing / decreasing the time required for parallax adjustment according to the time during which a region to be adjusted related to an object exists in the moving image. In other words, the parallax may be adjusted by increasing or decreasing the time required for the parallax adjustment according to the number of frames in the moving image having n frames that include the adjustment area related to an object.

  Specifically, the parallax adjustment unit 466 may adjust the parallax of each pixel in the adjusted region from the initial parallax to the target parallax for a number of frames corresponding to a predetermined time. The predetermined time mentioned here is set to a time within 50% of the time during which a region to be adjusted relating to a certain object exists in the moving image.

  With regard to such parallax adjustment, for example, according to a predetermined time, an adjustment pattern in which the parallax changes in a curve as time passes is expanded and contracted in at least one of the time direction and the magnitude direction of the parallax, The parallax may be adjusted according to the adjustment pattern after expansion / contraction. Thereby, since the fluctuation | variation of the time required for parallax adjustment is suppressed, the visual discomfort given to a user is suppressed appropriately.

FIG. 20 schematically illustrates the relationship between the time and the adjusted parallax when the parallax of each pixel in the adjusted region is adjusted from the initial parallax to the target parallax in a number of frames corresponding to a predetermined time. It is a graph. In FIG. 20, the target parallax dd _ob is drawn by a thick broken line L60. Also, the relationship between the elapsed time and the parallax after adjustment, change in the parallax is adjusted from an initial disparity d0 to the target parallax dd _ob is shown by the solid curve L61 in time T1d exceeding the predetermined time T _th d as a comparative example A change in parallax adjusted from the initial parallax d0 to the target parallax dd _ob at a predetermined time T _th d is indicated by a curve L62.

  Note that the predetermined time may be set so that the time required for adjusting the parallax related to each pixel in the adjusted area is always constant, or the parallax related to each pixel in the adjusted area depending on the size of the target parallax The time required for the adjustment may be adjusted uniquely.

  In the above embodiment, the parallax is adjusted so that the parallax changes in a curve with the passage of time. However, the present invention is not limited to this. For example, the parallax may be adjusted so that the parallax changes in proportion to the passage of time in each of the first period A1, the second period A2, and the third period A3. That is, the parallax change rate (also referred to as first change rate) obtained by dividing the parallax change amount in the first period A1 by the length of the first period A1 is the parallax change rate in the second period A2. It may be smaller than the change rate of parallax obtained by dividing by the length of the two periods A2 (also referred to as second change rate).

  For example, the parallax may be adjusted so that the parallax changes in proportion to the passage of time in each of the first period B1, the second period B2, and the third period B3. That is, the parallax change rate obtained by dividing the parallax change amount in the first period B1 by the length of the first period B1 divides the parallax change amount in the second period B2 by the length of the second period B2. The parallax may be adjusted so as to be smaller than the change rate of the parallax obtained by this.

  In the above embodiment, the target parallax is set using the principle of triangulation, but the present invention is not limited to this. For example, the target parallax may be set by other methods such as an MTD (Modified Time Difference) method and a CID (Computed Image Depth) method. That is, the target parallax may correspond to the separation distance between the Nth viewpoint and the Nth virtual viewpoint related to the Nth frame and the distance from the camera 2 to the subject when the Nth frame is captured.

  In the embodiment described above, the stereoscopic image is generated based on the moving image shot by the camera 2, but the present invention is not limited to this. For example, a moving image shot by another camera is stored in the storage medium 9 or the like, and is read from the storage medium 9 into the control unit 46 via the input / output unit 45, and a stereoscopic image is generated based on the moving image. May be.

◎ Further, in the above embodiment, the N-th frame G _N is used directly as the N left-eye frame G _N L, but the N-th frame G _N from the N right-eye frames G _N R is generated, It is not limited to this. For example, the N-th frame G _N is used directly as the N right-eye frames G _N R, N-th frame G _N N-th left-eye frame G _N L from may be generated. Furthermore, the Nth frame G _N is not used as either the Nth left eye frame G _N L or the Nth right eye frame G _N R, and the Nth frame for the left eye G based on the Nth frame G _{N. N} L and the N-th right eye frame G _N R may be generated.

In the above embodiment, the Nth distance information is acquired by the distance measuring device 3, but the present invention is not limited to this. For example, the Nth distance information may be acquired by using a stereo image taken by a stereo camera or the like and using the principle of triangulation. When the parallax being adjusted corresponds to the baseline length of the stereo camera, the stereo image can be used as it is as the Nth left eye frame G _N L and the Nth right eye frame G _N R. Become.

  In the above embodiment, an example has been given in which the parallax adjustment processing is switched between a case where the target parallax is large and a case where the target parallax is small. For example, the size of the target parallax may be divided into three or more levels, and parallax adjustment may be performed according to the level corresponding to the target parallax size.

  In the above-described embodiment, the relationship between the Nth viewpoint and the Nth virtual viewpoint is described as being a standard positional relationship between the right eye and the left eye of a human, but is not limited thereto. For example, the relationship between the Nth viewpoint and the Nth virtual viewpoint may be another relationship such as the relationship between the right eye and the left eye of an insect.

  In the above-described embodiment, various functions, various information processing, and the like are realized by executing the program PG in the control unit 46. However, the present invention is not limited to this. For example, various functions and various information processing may be realized by a dedicated electronic circuit, and various functions and various information processing may be realized by executing a dedicated electronic circuit and a program. .

  It goes without saying that all or a part of each of the above-described embodiment and various modifications can be appropriately combined within a consistent range.

DESCRIPTION OF SYMBOLS 1 Information processing system 2 Camera 3 Distance measuring device 4 Information processing apparatus 9 Storage medium 44 Storage part 46 Control part 461 Image acquisition part 462 Distance information acquisition part 463 Area | region detection part 464 Target parallax setting part 465 Initial parallax setting part 466 Parallax adjustment part 467 Virtual viewpoint image generation unit 468 Stereoscopic image generation unit PG program

Claims (12)

An image acquisition unit for acquiring a moving image including a plurality of frames;
A target parallax setting unit that sets a target parallax for each pixel based on a predetermined rule for each of the frames;
An initial parallax setting unit that sets an initial parallax for each pixel for a first frame of the plurality of frames;
Among the plurality of frames, the parallax of each pixel is adjusted from the initial parallax toward the target parallax with respect to the plurality of frames related to the first period , the second period, and the third period in time sequence following the first frame. A parallax adjustment unit that
With
The first change rate obtained by dividing the parallax change amount in the first period by the length of the first period is obtained by dividing the parallax change amount in the second period by the length of the second period. rather smaller than the second rate of change that is required,
The image processing apparatus the second rate of change, characterized by the third magnitude than the third change rate obtained by dividing the amount of change of the disparity in length of the third period in the period Ikoto. The image processing apparatus according to claim 1,
A storage unit that stores a plurality of types of adjustment patterns related to a change in parallax from the initial parallax toward the target parallax;
The parallax adjustment unit
An image processing apparatus that adjusts the parallax of each pixel along an adjustment pattern according to a difference between the initial parallax and the target parallax among the plurality of types of adjustment patterns. The image processing apparatus according to claim 1,
The parallax adjustment unit
Image processing characterized in that when N is a natural number, the parallax of the (N + 1) th and subsequent frames is adjusted based on the parallax of the Nth frame after adjustment by the parallax adjustment unit and the target parallax apparatus. The image processing apparatus according to claim 1,
The parallax adjustment unit
When Q is a natural number, the parallax of the (N + 1) th and subsequent frames based on the target parallax for each pixel in the (N + Q) th frame and the parallax of the Nth frame after being adjusted by the parallax adjustment unit An image processing apparatus characterized by adjusting the above. The image processing apparatus according to claim 1 or 2,
The parallax adjustment unit
An image processing apparatus that adjusts the parallax of each pixel from the initial parallax to the target parallax for a number of frames corresponding to a predetermined time. The image processing apparatus according to any one of claims 1 to 4, wherein:
The parallax adjustment unit
An image processing apparatus that adjusts the parallax of each pixel so that a parallax adjustment amount between temporally adjacent frames does not exceed an upper limit value. The image processing apparatus according to any one of claims 1 to 6, wherein
The parallax adjustment unit
An image processing apparatus, wherein the parallax of each pixel is adjusted for a second subject different from the first subject in accordance with the parallax of the first subject captured in the moving image. The image processing apparatus according to any one of claims 1 to 6, wherein
The parallax adjustment unit
An image processing apparatus that adjusts the parallax of each pixel according to the position of a subject captured in each of the frames. The image processing apparatus according to any one of claims 1 to 8, wherein
The target parallax is
An image processing apparatus that corresponds to a separation distance between a viewpoint and a virtual viewpoint related to each frame and a distance from an imaging device to a subject when each of the frames is captured. An image processing apparatus according to any one of claims 1 to 9, wherein
When N and p are natural numbers, the N-th frame among the plurality of frames is compared with the Np-th frame that is temporally prior to the N-th frame. A detection unit for detecting a region to be adjusted in the frame of
Further comprising
The parallax adjustment unit
An image processing apparatus, wherein the parallax of each pixel related to the adjustment area is adjusted from the initial parallax toward the target parallax. (a) obtaining a moving image including a plurality of frames;
(b) for each of the frames, setting a target parallax based on a predetermined rule for each pixel;
(c) setting an initial parallax for each pixel for a first frame of the plurality of frames;
(d) Of the plurality of frames, the parallax of each pixel is changed from the initial parallax to the target parallax for a plurality of frames related to the first period , the second period, and the third period in time sequence following the first frame. Steps to adjust,
With
The first change rate obtained by dividing the parallax change amount in the first period by the length of the first period is obtained by dividing the parallax change amount in the second period by the length of the second period. rather smaller than the second rate of change that is required,
Said second rate of change, an image processing method, wherein said third magnitude than the third change rate obtained by dividing the amount of change of the disparity in length of the third period in the period Ikoto.   A program that causes the information processing apparatus to function as the image processing apparatus according to any one of claims 1 to 10 when executed by a control unit included in the information processing apparatus.

Images