JP7479840B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

A stereo camera is known that has two imaging units and can capture images for stereoscopic vision. The stereo camera simultaneously captures images of the same subject using each imaging unit, and obtains two types of images, a first image and a second image, which have parallax. The stereo camera can calculate the distance from the camera to the subject by calculating the amount of parallax for each pixel from the images with parallax. This distance information is generally expressed in the camera coordinate system, and can be converted into a point cloud representing the position of the subject in the world coordinate system by coordinate transformation. The stereo camera can set a virtual light source, and use the position of the virtual light source and the point cloud to calculate the area of shadow created by the virtual light source and generate an image with a shadow added.

Non-Patent Document 1 discloses a method for calculating a shadow area by calculating the difference in distance between the virtual light source position and each point in a point cloud on the front side and the distance to each point in a point cloud on the back side, and determining that an area where the distance difference between each point in the back side point cloud is equal to or exceeds a threshold value is a shadowed area. If the threshold value is too small, unnecessary shadows called shadow acne will occur, and if the threshold value is too large, necessary shadows will not be cast, so a method for appropriately controlling the threshold value is required.

Patent document 1 discloses a method for reducing errors in shadow areas by referencing the ray direction vector of a virtual light source and the normal direction vector of the point cloud to be shaded, and increasing the threshold value when the degree of difference between the two vectors is large, and decreasing the threshold value when the degree of difference between the two vectors is small.

JP 2012-522283 A

Hang Dou, Yajie Yan, Ethan Kerzner, Zeng Dai, Chris Wyman, Washington University in St. Louis, SCI Institute University of Iowa, NVIDIA, "Adaptive Depth Bias for Shadow Maps", Journal of Computer Graphics Techniques, Vol. 3, No.4, 2014

In conventional technology, the subject's position is designed using computer graphics, and this method is effective on the assumption that it can be acquired accurately. However, the subject's position acquired using a method that calculates the amount of parallax is likely to include errors when calculating the amount of parallax, and it is not possible to reduce errors in the shadow area caused by errors in the subject's position.

The objective of the present invention is to reduce the error in the shadowed area when there is an error in the position of the subject.

The image processing device of the present invention has a point cloud acquisition means for acquiring a set of points having information on the position of a subject and the surface inclination of a local area at the position of the subject as a point cloud based on an image; a threshold setting means for setting a threshold such that a larger difference between the surface inclination of the local area of each point of the first point cloud and the surface inclination of the local area of each point of the second point cloud is larger, the threshold being smaller; and an area setting means for setting points of the second point cloud where the difference in distance between the distance from the light source to each point of the first point cloud and the distance from the light source to each point of the second point cloud is equal to or greater than the threshold as a shaded area.

According to the present invention, when there is an error in the position of the subject, it is possible to reduce the error in the shadowed area.

FIG. 1 is a diagram illustrating an example of the configuration of an imaging device and an image processing device. 4 is a flowchart showing an image processing method of the image processing device. 11 is a flowchart illustrating acquisition of a point cloud. 11 is a flowchart illustrating acquisition of location information. 2 is a diagram illustrating a pixel array configuration of an imaging unit. FIG. FIG. 13 is a diagram illustrating setting of minute blocks. 11A and 11B are diagrams illustrating the relationship between an image shift amount and a correlation value. FIG. 11 is a diagram illustrating a threshold setting method. 1A and 1B are diagrams illustrating an object for which a shadow region is calculated. FIG. 11 is a diagram illustrating calculation of a shadow region. FIG. 13 is a diagram illustrating a method for setting a reference threshold value.

Fig. 1(a) is a block diagram showing an example of the configuration of an image processing device 100 and an imaging device 111 according to this embodiment. The image processing device 100 has a control unit 101, a memory 102, an image information input unit 105, a parameter setting unit 106, a point cloud acquisition unit 107, a threshold setting unit 108, a shadow region calculation unit 109, and a shadowed image generation unit 110. The imaging device 111 has a control unit 101, an imaging unit 103, and a light emission unit 104.

The control unit 101 controls the exchange of information between the components of the image processing device 100 and the imaging device 111. The memory 102 is controlled by the control unit 101, and reads out information necessary for processing in each component and stores the processing results of each component.

The imaging unit 103 captures an image of a subject by photoelectric conversion and outputs image information. The control unit 101 supplies the image information captured by the imaging unit 103 to the image information input unit 105 and the point cloud acquisition unit 107. The light emitting unit 104 emits light when the imaging unit 103 captures an image under the control of the control unit 101.

The image information input unit 105 inputs image information captured by the imaging unit 103 or image information captured by a device other than the imaging device 111, and supplies it to the memory 102. The parameter setting unit 106 sets each piece of parameter information input by the user to the image processing device 100. The control unit 101 stores the parameter information set by the parameter setting unit 106 in the memory 102.

The point cloud acquisition unit 107 calculates a point cloud from image information captured by the imaging unit 103, and supplies the calculated point cloud information to the memory 102. The threshold setting unit 108 calculates a threshold from the point cloud calculated by the point cloud acquisition unit 107. The control unit 101 stores the threshold calculated by the threshold setting unit 108 in the memory 102.

The shadow region calculation unit 109 calculates a shadow region from the point cloud calculated by the point cloud acquisition unit 107 and the threshold value calculated by the threshold setting unit 108, and supplies information on the calculated shadow region to the memory 102. The shadow-added image generation unit 110 generates an image to which a shadow has been added based on the shadow region calculated by the shadow region calculation unit 109, and supplies the generated image to the memory 102.

Figure 1 (b) is a diagram showing an optical system of the imaging unit 103 of the imaging device 111. The imaging unit 103 forms an image of light emitted from an object 112 on a planned imaging plane 114 using a lens 113, and receives the light on a sensor plane 115. The imaging unit 103 converts the received light information into image information.

Figure 2 is a flowchart showing the image processing method of the image processing device 100. In step S201, the parameter setting unit 106 acquires parameters and sets the parameters. The parameters include at least the following parameters:

Distance from the imaging device 111 to the focal position of the subject...(a)
Pixel pitch of the imaging unit 103...(b)
A conversion coefficient determined by the magnitude of the opening angle of the center of gravity of the light beam passing through a pair of distance measuring pupils...(c)
F value...(d)
Pupil distance...(e)
Distance from the image-side principal point of the lens 113 of the image pickup device 111 to the sensor surface 115 (f)
Distance from the object side principal point of the lens 113 of the imaging device 111 to the subject... (g)
Focus position of the imaging device 111...(h)
Virtual light source position...(i)
Reference threshold value...(j)
An arbitrary coefficient to multiply the difference in slope...(k)

The parameters are required to calculate the defocus amount, calculate point cloud coordinates in three-dimensional space, and calculate shadow areas.

In step S202, the image information input unit 105 acquires image information. For example, the image information input unit 105 acquires image information captured by the imaging unit 103 of the imaging device 111, or image information captured by a device other than the imaging device 111.

In step S203, the point cloud acquisition unit 107 acquires a point cloud based on the image information acquired in step S202. A "point cloud" is a collection of points having information on the position of the subject and the surface inclination of the local area at the subject's position. The "position of the subject" is the position of the subject in space, and can be determined by measuring the distance from a specified position to the subject. The "surface inclination of the local area" is the surface inclination of the local area of the subject shape, and is information such as the amount of displacement of the shape of the local area and the surface normal.

Figure 3 is a flowchart showing the details of step S203 in Figure 2. In step S301, the point cloud acquisition unit 107 acquires the position of the subject from the image information acquired in step S202. Step S301 will be described with reference to Figure 4.

Figure 4 is a flowchart showing the details of step S301 in Figure 3. In step S401, the point cloud acquisition unit 107 generates a pair of image data from the image information acquired in step S202. A method for generating a pair of image data from image information will be described with reference to Figures 5(a) and 5(b).

Figure 5(a) is a diagram showing an example of the arrangement of pixels 512 of the imaging unit 103 used when capturing an image. Figure 5(b) is an enlarged view of the pixel 512. The imaging unit 103 has a plurality of pixels 512 arranged in a two-dimensional matrix. The pixel 512 has a microlens 511 and a pair of pupil division pixels 513A and 514B. The pupil division pixels 513A and 514B are each a photoelectric conversion unit, and generate a pixel signal by photoelectric conversion. The image captured by the pupil division pixel 513A is image A, and the image captured by the pupil division pixel 514B is image B. The image obtained by combining the images A and B is image AB. The image AB is an image acquired by the user at the time of capturing. The point cloud acquisition unit 107 acquires the image A and the image B as a pair of image data from the image information of a plurality of images as information necessary for image processing. The images A and B are a pair of images based on the light beams of the subject passing through different pupil regions of the optical system.

In step S402, the point cloud acquisition unit 107 sets minute blocks for the A and B images generated in step S401. FIG. 6A is a diagram for explaining step S402. The point cloud acquisition unit 107 sets minute blocks 603 for the A and B images 601 and 602. The point cloud acquisition unit 107 sets a pixel of interest at the center of the minute block for each pixel in the input image. For example, the point cloud acquisition unit 107 sets a minute block 603 for a pixel of interest 604. FIG. 6B is a diagram for explaining the shape of the minute block. The point cloud acquisition unit 107 can set minute blocks 605 or 606 of various shapes or sizes for the A image 601. The size or shape of the minute block is not limited.

In step S403, the point cloud acquisition unit 107 performs correlation calculation processing on the minute block set in step S402 to calculate the image shift amount for each point in the images A and B. In the correlation calculation, a pair of pixel data in the minute block is generalized and expressed as E(1) to E(m) and F(1) to F(m), where m is the number of data. In this case, the point cloud acquisition unit 107 shifts the data series F(1) to F(m) relatively to the data series E(1) to E(m), and calculates the correlation amount C(k) at the shift amount k between the two data strings using formula (1).
C(k)=Σ|E(n)−F(n+k)| (1)

In formula (1), the Σ operation is performed for n. In this Σ operation, the range of n and n+k is limited to the range of 1 to m. The shift amount k is an integer, and is the relative shift amount in units of the detection pitch of a pair of image data.

Figure 7 is a graph showing an example of the calculation result of formula (1), with the horizontal axis representing the shift amount k and the vertical axis representing the correlation amount C(k). As can be seen from Figure 7, the correlation amount C(k) is minimum at a shift amount k where the correlation between a pair of data series is high. Then, the point cloud acquisition unit 107 uses a three-point interpolation method according to the following formulas (2) to (5) to find the image shift amount x that gives the minimum correlation amount C(x) for continuous correlation amounts.

x = kj + D / SLOP (2)
C(x)=C(kj)−|D| (3)
D = [C(kj-1) - C(kj+1)] / 2 ... (4)
SLOP=MAX[C(kj+1)-C(kj), C(kj-1)-C(kj)]... (5)

Here, kj is k at which the discrete correlation amount C(k) is minimized. The point cloud acquisition unit 107 obtains the image shift amount x in a pair of pupil division images using equation (2). Note that the unit of the image shift amount x is pixels.

In step S404, the point cloud acquisition unit 107 calculates a defocus amount DEF of the subject image plane with respect to the intended imaging plane 114 from the image shift amount x calculated by equation (2) using the following equation (6):
DEF = KX PY x ... (6)

In equation (6), PY is the pixel pitch of the image capture unit 103 (the distance between the pixels that make up the image capture unit 103). KX is a conversion coefficient determined by the magnitude of the opening angle of the center of gravity of the light beams passing through a pair of ranging pupils, and its unit is dimensionless. The point cloud acquisition unit 107 calculates the defocus amount DEF for each pixel position by repeatedly shifting the pixel position of interest by one pixel at a time in this manner.

In step S405, the point cloud acquisition unit 107 calculates the distance z from the sensor surface 115 of the imaging unit 103 of the imaging device 111 to the subject using the defocus amount DEF calculated in step S404. The point cloud acquisition unit 107 calculates the distance z at each point using equations (7) to (8).

dist = 1 / (1 / (dist_d + DEF) - 1 / f) ... (7)
z = length - dist ... (8)

Here, dist is the distance from the object side principal point of the lens 113 of the imaging unit 103 to the subject. dist_d is the distance from the image side principal point of the lens 113 of the imaging unit 103 of the imaging device 111 to the sensor surface 115. f is the focal length. length is the distance from the sensor surface 115 of the imaging device 111 to the focus position.

The distance (length) from the sensor surface 115 of the imaging device 111 to the focal position can be measured, for example, by a laser distance measuring means. In addition, the point cloud acquisition unit 107 has a data table that stores the relationship between the lens position at the time of imaging and the focal position, and can estimate the distance to the focal position corresponding to the lens position at the time of imaging. This allows the point cloud acquisition unit 107 to reduce the effort required to measure the distance from the sensor surface 115 of the imaging device 111 to the focal position.

In this way, the point cloud acquisition unit 107 can calculate the distance z from the sensor surface 115 of the imaging unit 103 of the imaging device 111 to the subject. In this way, the point cloud acquisition unit 107 can calculate the distance z by generating a pair of image data from a single image, so the imaging device 111 does not need to be a compound eye, and a monocular camera will suffice, simplifying the configuration of the imaging device 111. Furthermore, calibration processing when installing multiple cameras can be simplified or eliminated. This allows the point cloud acquisition unit 107 to improve the accuracy of calculating the distance z.

The distance z from the sensor surface 115 of the imaging unit 103 of the imaging device 111 to the subject can be calculated by the image processing device 100 from the image information captured by the imaging device 111 in step S301. In addition to this method, it is also possible to calculate the distance z using a stereo camera or to obtain laser reflection information such as LiDAR and calculate the distance z. Furthermore, it is also possible to calculate these distances z by an external device of the imaging device 111, and for the image processing device 100 to obtain the distance z calculated by the external device. Note that the distance z is not limited to the distance from the sensor surface 115, and may be the distance from any position, such as the tip of the lens.

In step S406, the point cloud acquisition unit 107 converts the distance z calculated in step S405 from the camera coordinate system to a point cloud in the world coordinate system, and acquires the position of the subject. Note that the point cloud acquisition unit 107 acquires the position of the subject by converting from the camera coordinate system to the world coordinate system, but the position of the subject may remain in the camera coordinate system, and distance information in the camera coordinate system may be used as the position of the subject.

In step S302, the point cloud acquisition unit 107 acquires the inclination of the surface of the local region. The inclination of the surface of the local region can be acquired by photometric stereo. The point cloud acquisition unit 107 acquires highly accurate normal information in pixel units by photometric stereo, and uses the normal information as the inclination of the surface of the local region. The point cloud acquisition unit 107 can also calculate the inclination of the surface of the local region based on the displacement amount of each point adjacent to the position of the subject calculated in step S301. For example, when each point is defined by three-dimensional spatial coordinates of X, Y, and Z, the displacement amount of the position in the X direction, or the displacement amount of the position in the Y direction, or the square root of the square sum of the displacement amount of the position in the X direction and the displacement amount in the Y direction, etc. can be used as the inclination of the surface of the local region of each point. At this time, since the position acquired by a sensor such as the image processing device 100 or LiDAR contains an error, when the displacement amount of each point is calculated, there is a high possibility that an error will occur in the displacement amount. In such a case, the point cloud acquisition unit 107 can reduce the error in the amount of displacement by calculating a corrected position by averaging the positions of the point clouds included in the local region, and then using the corrected position to calculate the above-mentioned amount of displacement. In this way, the inclination of the surface of the local region can be acquired even if there is no device for acquiring normal lines. In this way, the point cloud acquisition unit 107 can acquire the inclination of the surface of the local region. The point cloud acquisition unit 107 can acquire a point cloud by acquiring the position of the subject and the inclination of the surface of the local region.

In step S204, the threshold setting unit 108 sets a threshold. The method for setting the threshold will be described with reference to FIG. 8. FIG. 8 is a flowchart showing the details of step S204 in FIG. 2. The flowchart in FIG. 8 will be described with reference to FIG. 9.

FIG. 9 is a diagram illustrating a subject for which a shadow region is calculated. The point cloud acquisition unit 107 obtains a point cloud 903 from the position of the subject by image processing of the shape of an object 902 captured by an image capture device (camera) 901. A virtual light source 904 is set for the point cloud 903. Among the point cloud 903, the point cloud on the near side when the position of the virtual light source 904 is set as a reference position is set as a first point cloud (points 905 and 906), and the point cloud on the far side when the position of the virtual light source 904 is set as a reference position is set as a second point cloud (points 907 and 908). When the position of the virtual light source 904 is set as a reference position, the point on the near side is the point on the light ray that is drawn from the virtual light source 904 to the point cloud, and the point on the far side is a point other than the point on the light ray that is on the near side. Based on this relationship between the position of virtual light source 904 and the position of the point cloud, each point of the second point cloud is shown to be a region where a shadow is cast due to the relationship between each point of the first point cloud and the position of virtual light source 904. These points 905 to 908 each have local area surface slope 909 to local area surface slope 912 as a point cloud. In addition, the distance difference between points 905 and 907, and between points 906 and 908 can be defined as distance difference 913 and distance difference 914, respectively.

Points 905 to 908 include an error with respect to the correct shape of object 902. If there is a distance difference due to such an error in the position of object 902, the shape of object 902 at point 906 is erroneously determined to be a shadowed area even though it is not a shape that would cast a shadow on the shape of object 902 at point 908. In response to this, the distance difference 914 between points 906 and 908 is calculated, and if the distance difference is equal to or greater than a threshold, it is determined to be a shadowed area, thereby preventing point 908 from being determined to be a shadowed area. However, if the threshold is determined uniformly, there is a problem that, as in the relationship between points 905 and 907, even if the distance difference is equal to or less than the threshold but it is correct to cast a shadow, it is erroneously determined to be a non-shadowed area. In response to this problem, by performing the processing of steps S204 and S205, it is possible to determine point 907 as a shadowed area while preventing point 908 from being determined to be a shadowed area.

In step S801, the threshold setting unit 108 sets a reference threshold. The reference threshold can be set as a parameter in step S201. The reference threshold can also be calculated by a method of obtaining it from measurement information, imaging information, and image information, which will be described later.

In step S802, the threshold setting unit 108 acquires the distance from the virtual light source to each point of the first point group and the surface slope of the local area of each point of the first point group. The distance to each point of the first point group and the surface slope of the local area of each point of the first point group refer to the distance from the virtual light source 904 to point 905 and point 906, respectively, and the corresponding local area surface slope 909 and local area surface slope 910 in FIG. 9.

In step S803, the threshold setting unit 108 acquires the distance from the virtual light source to each point of the second point group and the surface slope of the local area of each point of the second point group. The distance to each point of the second point group and the surface slope of the local area of each point of the second point group refer to the distance from the virtual light source 904 to point 907 and point 908, respectively, and the corresponding local area surface slope 911 and local area surface slope 912 in FIG. 9.

In step S804, the threshold setting unit 108 calculates the difference between the surface inclination of the local region for each point in the first point group and the surface inclination of the local region for each point in the second point group. The difference in the surface inclination of the local regions may be considered as the degree of dissimilarity between the surface inclinations of the two local regions. A large degree of dissimilarity indicates that the amount of local shape displacement is large or that the directions in which the shape surfaces are facing are significantly different. In FIG. 9, the surface inclination 909 of the local region and the surface inclination 911 of the local region indicate a large difference. In contrast, the surface inclination 910 of the local region and the surface inclination 912 of the local region indicate a small difference.

In step S805, the threshold setting unit 108 sets a threshold based on the difference between the surface inclination of the local region of each point of the first point group calculated in step S804 and the surface inclination of the local region of each point of the second point group. The difference between the surface inclination of the local region of each point of the first point group and the surface inclination of the local region of each point of the second point group is defined as the inclination difference N_diff. The threshold to be set is set as the threshold Th, the reference threshold is set as the reference threshold Th_be, and the coefficient by which the inclination difference N_diff is multiplied is set as the coefficient q, and the threshold setting unit 108 can calculate the threshold Th by equation (9).
Th = Th_be-q × N_diff (9)

At this time, the coefficient q is determined so that the threshold Th satisfies Th≧0, where q≧0. By defining the threshold Th in this way, it is possible to change the value of the threshold Th within the range from 0 to Th_be. In equation (9), if the coefficient q is a certain constant, the threshold Th becomes smaller as the difference N_diff increases. This indicates that the threshold Th becomes smaller as the difference between the surface slope of the local region of the first point cloud and the surface slope of the local region of the second point cloud increases.

In step S205, the shadow region calculation unit 109 calculates the shadow region. Details of step S205 will be described with reference to Figures 9 and 10.

Figure 10 is a flowchart showing the details of step S205 in Figure 2. In step S1001, the shadow region calculation unit 109 calculates the distance difference between the distance from the virtual light source to each point in the first point group and the distance from the virtual light source to each point in the second point group. In Figure 9, the distance difference corresponds to distance difference 913 and distance difference 914.

In step S1002, the shadow region calculation unit 109 determines whether the distance difference DD is equal to or greater than a threshold value Th, as in equation (10).
Th≦DD (10)

If the distance difference DD is equal to or greater than the threshold value Th, the shadow region calculation unit 109 proceeds to step S1003, and if the distance difference DD is less than the threshold value Th, the shadow region calculation unit 109 proceeds to step S1004. Because the difference in the slope of the surfaces of the local regions of points 905 and 907 is large, the threshold value Th is set to a smaller value than the reference threshold value Th_be. As a result, equation (10) is satisfied for points 905 and 907, and the shadow region calculation unit 109 proceeds to step S1003.

In contrast, the difference in the surface inclination of the local regions of points 906 and 908 is small, so the threshold value Th is set to a value that is the same as or close to the reference threshold value Th_be. As a result, points 906 and 908 do not satisfy equation (10), so the shadow region calculation unit 109 proceeds to step S1004.

In step S1003, the shadow region calculation unit 109 functions as a region setting unit, and sets, among the second point group, points whose distance difference DD is equal to or greater than a threshold value Th as a shadowed region. In FIG. 9, point 907 is set as a region shaded by point 905.

In step S1004, the shadow region calculation unit 109 functions as a region setting unit, and sets points in the second point group whose distance difference DD is less than the threshold value Th as a region that is not shaded. In FIG. 9, point 908 is set as a region that is not shaded by point 906.

With the above, the shadow region calculation unit 109 finishes setting the shadow region. The image processing device 100 sets the threshold value using the inclination of the surface of the local region. This allows the image processing device 100 to reduce erroneous determinations of shadowed regions that occur due to errors in the position of the subject, while also reducing erroneous determinations of regions that should be shadowed as not being shadowed by setting a uniform threshold value.

Next, the shadowing process based on the shadowed area will be described. In step S205, the shadowed image generating unit 110 performs image processing to add a shadow to the image based on the set shadowed area, and generates a shadowed image. The shadowed image generating unit 110 generates an image to which a shadow has been added by a virtual light source by reducing the RGB values of the image corresponding to the point cloud set as the shadowed area.

Next, a method for calculating the reference threshold from the measurement information, imaging information, and image information will be described. In step S801, the threshold setting unit 108 arbitrarily set the reference threshold, but it is possible to calculate the reference threshold based on the measurement information obtained when the position of the subject was measured, the imaging information obtained at the time of imaging, and the image information of the captured image. The greater the position error, the greater the determination error in the shaded area, so it is preferable to set the reference threshold high.

The threshold setting unit 108 can determine the reference threshold Th_be from the reference threshold Th_a determined from the measurement information, the reference threshold Th_b determined from the imaging information, and the reference threshold Th_c determined from the image information, using the following formula (11) or formula (12).

Th_be=Th_a+Th_b+Th_c (11)
Th_be=Th_a×Th_b×Th_c (12)

The threshold setting unit 108 determines the reference threshold Th_be from, for example, the sum or product of the reference thresholds calculated from each piece of information, but the relationship between the product and the sum may be changed, or the reference thresholds calculated from each piece of information may be weighted. The reference thresholds Th_a, Th_b, and Th_c are described with reference to FIG. 11.

Figures 11(a)-(b) are diagrams explaining the parameters evaluated when calculating the reference threshold value Th_a obtained from the measurement information and the reference threshold value of each parameter at that time. The reference threshold value Th_a obtained from the measurement information is calculated from one or more pieces of information on the standard deviation of the position of the local region and the defocus amount of the imaging unit 103 of the imaging device 111.

11A is a diagram showing the relationship between the standard deviation of the position of a local region and the reference threshold value Ap of the standard deviation of the position of a local region. The standard deviation of the position of a local region is evaluated to evaluate the variation in the position of a subject. The larger the value of the standard deviation, the greater the variation. From arbitrary values Ab and Ac and the standard deviation value Ad, the reference threshold value Ap of the standard deviation value is expressed by the following formula (13), where Ab>0 and Ac>0.
Ap = Ac × Ad + Ab ... (13)

11B is a diagram showing the relationship between the defocus amount of the imaging unit 103 and the reference threshold value Bp of the defocus amount of the imaging unit 103. In an area where the defocus amount of the imaging unit 103 is large, the symmetry of the left and right parallax images is low, and the calculated position of the subject varies greatly. From arbitrary values Bb and Bc and the defocus amount Bd of the imaging unit 103, the reference threshold value Bp of the defocus amount of the imaging unit 103 is expressed by the following formula (14). However, Bb>0, Bc>0.
Bp = Bc × Bd + Bb ... (14)

The threshold setting unit 108 calculates the reference threshold Th_a from the measurement information using equation (15).
Th_a = Ap + Bp ... (15)

Figures 11(c) to (e) are diagrams explaining each parameter evaluated when calculating the reference threshold value Th_b obtained from the imaging information and the reference threshold value of each parameter at that time. The reference threshold value Th_b obtained from the imaging information is calculated from information on one or more of the aperture value of the imaging unit 103 of the imaging device 111, the distance to the focus position of the imaging unit 103 of the imaging device 111, and the focal length of the imaging unit 103 of the imaging device 111.

11C is a diagram showing the relationship between the aperture value of the imaging unit 103 and the reference threshold value Cp of the aperture value of the imaging unit 103. If the aperture value is increased during imaging, the variation in the subject position calculated in step S501 increases. From arbitrary values Cb, Cc and the aperture value Cd, the reference threshold value Cp of the aperture value is expressed by the following formula (16), where Cb>0 and Cc>0.
Cp=Cc×Cd+Cb (16)

11D is a diagram showing the relationship between the distance to the focus position of the imaging unit 103 and the reference threshold value Dp of the distance to the focus position of the imaging unit 103. When the distance to the focus position is increased during imaging, the variation in the position of the subject increases. This is because the calculation accuracy of the subject position is inversely proportional to the square of the distance to the focus position. From any values Db, Dc and the distance Dd to the focus position of the imaging unit 103, the reference threshold value Dp of the distance to the focus position of the imaging unit 103 is expressed by the following formula (17). However, Db>0, Dc>0.
Dp = Dc × Dd + Db ... (17)

11E is a diagram showing the relationship between the focal length of the imaging unit 103 and the reference threshold value Ep of the focal length of the imaging unit 103. When the focal length is increased during imaging, the angle of view becomes narrower, and the variation in the position of the subject becomes smaller. From arbitrary values Eb and Ec and the focal length Ed of the imaging unit 103, the reference threshold value Ep of the focal length of the imaging unit 103 is expressed by the following formula (18), where Eb>0 and Ec<0.
Ep = Ec × Ed + Eb (18)

The threshold setting unit 108 calculates the reference threshold Th_b from the imaging information using equation (19).
Th_b=Cp+Dp+Ep... (19)

Figures 11(f) to (g) are diagrams explaining the parameters evaluated when calculating the reference threshold value Th_c obtained from the image information and the reference threshold value for each parameter at that time. The reference threshold value Th_c obtained from the image information is calculated from one or more pieces of information on the image contrast and the image signal-to-noise ratio.

11F is a diagram showing the relationship between the image contrast and the reference contrast threshold value Fp. As the image contrast increases, the variation in the subject position decreases. From arbitrary values Fb, Fc and the image contrast Fd, the reference contrast threshold value Fp of the image is expressed by the following formula (20), where Fb>0 and Fc<0.
Fp = Fc × Fd + Fb ... (20)

11G is a diagram showing the relationship between the S/N ratio of an image and the standard threshold value Gp of the S/N ratio of an image. When the S/N ratio of an image is large, the ratio of signal to noise increases, and therefore the variation in the position of the subject decreases. From arbitrary values Gb, Gc and the S/N ratio Gd of an image, the standard threshold value Gp of the S/N ratio of an image is expressed by the following formula (21), where Gb>0 and Gc<0.
Gp = Gc × Gd + Gb ... (21)

The threshold setting unit 108 calculates the reference threshold Th_c from the image information using equation (22).
Th_c=Fp+Gp (22)

As described above, the threshold setting unit 108 can set the reference threshold Th_be based on one or more of the measurement information, the imaging information, and the image information. By determining the reference threshold Th_be, the threshold setting unit 108 can determine the threshold Th in more detail according to the magnitude of the error in the position of the subject, thereby further reducing the error in determining the shadowed area.

In addition, the threshold setting unit 108 can determine the reference threshold Th_be based on the imaging information, so it is possible to determine the reference threshold Th_be before imaging. Furthermore, the threshold setting unit 108 can determine the reference threshold Th_be based on the measurement information and image information, so it is possible to determine the reference threshold Th_be even if the imaging information is unknown.

The imaging device 111 can be applied to digital cameras or video cameras, as well as smartphones, tablets, industrial cameras, medical cameras, or vehicle-mounted cameras.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

The above embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted in a limiting manner. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

100 Image processing device, 101 Control unit, 102 Memory, 103 Imaging unit, 104 Light emission unit, 105 Image information input unit, 106 Parameter setting unit, 107 Point cloud acquisition unit, 108 Threshold setting unit, 109 Shadow region calculation unit, 110 Shadow image generation unit, 111 Imaging device

Claims (12)

a point cloud acquisition means for acquiring a set of points having information on a position of an object and a surface inclination of a local area at the position of the object as a point cloud based on an image;
a threshold setting means for setting a threshold value such that a difference between a surface gradient of the local region of each point of the first point group and a surface gradient of the local region of each point of the second point group is larger, the threshold value being smaller; and
and an area setting means for setting, among the second point cloud, points where the difference in distance between the light source and each point of the first point cloud and the distance from the light source to each point of the second point cloud is equal to or greater than the threshold value as a shadowed area. The image processing device according to claim 1, further comprising an image generating means for generating a shaded image based on the shaded area. The image processing device according to claim 1 or 2, characterized in that the point cloud acquisition means calculates a distance to a subject based on a pair of images based on light beams of the subject passing through different pupil regions of an optical system, and acquires a position of the subject based on the distance to the subject. 4. The image processing device according to claim 1, wherein the point cloud acquisition means acquires normal information as a surface inclination of the local region by a photometric stereo method. 4. The image processing device according to claim 1 , wherein the point cloud acquisition means acquires a tilt of a surface of the local region based on an amount of displacement between positions of each point adjacent to a position of the subject. The image processing device according to claim 5 , wherein the point cloud acquisition means calculates a corrected position by averaging positions of a point cloud included in the local region , and calculates the amount of displacement using the corrected position. 7. The image processing apparatus according to claim 1 , wherein the threshold setting means sets the threshold based on at least one of measurement information, imaging information, and image information. 8. The image processing apparatus according to claim 7 , wherein the measurement information is at least one of a standard deviation of a position of a local region and a defocus amount of an image capturing apparatus. 8. The image processing device according to claim 7 , wherein the imaging information is at least one of an aperture value of the imaging device, a distance to a focus position of the imaging device, and a focal length of the imaging device. 8. The image processing apparatus according to claim 7 , wherein the image information is at least one of information on contrast of the image and information on an S/N ratio of the image. a point cloud acquisition step of acquiring, as a point cloud, a set of points having information on the position of an object and the inclination of a surface of a local area at the position of the object based on an image;
a threshold setting step of setting a threshold value such that a difference between a surface gradient of the local region of each point of the first point group and a surface gradient of the local region of each point of the second point group is larger, the threshold value being smaller; and
and a region setting step of setting, among the second point cloud, points where the difference in distance between the light source and each point of the first point cloud and the distance from the light source to each point of the second point cloud is equal to or greater than the threshold value as a shadowed region. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 10 .

Images