JP6137929B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to an information processing device and an information processing method, and more specifically, relates to an information processing device and an information processing method for determining imaging conditions of a plurality of imaging devices that image a subject from different viewpoint positions.

  Conventionally, for example, techniques disclosed in Patent Document 1 and Patent Document 2 are known as techniques for a plurality of imaging devices to focus on the same subject. The technique described in Patent Document 1 calculates the imaging conditions so that the in-focus positions of all the imaging devices are within the depth of field. Further, the technique described in Patent Document 2 performs face recognition processing on images captured by the respective imaging devices and focuses on the same face.

JP 2012-44540 A JP 2010-114712 A

  However, in the technique described in Patent Document 1, when there are a plurality of subjects, each imaging apparatus calculates a focus position on a separate subject by AF (autofocus) processing, and all of those focus positions are the depth of field. An imaging condition with a large F value is calculated so as to fall within the range. If the F value is large, it is necessary to increase the ISO sensitivity or slow down the shutter speed in order to make the exposure appropriate. If the ISO sensitivity is increased, there is a problem that noise increases, and if the shutter speed is decreased, there is a problem that subject blurring occurs. Further, the technique disclosed in Patent Document 2 determines where to focus when an imaging device that does not capture a face from the front cannot perform face recognition when each imaging device captures images from different directions. There is a problem that you can not.

  Therefore, an object of the present invention is to allow each imaging device to focus on one subject among the plurality of subjects even when the plurality of imaging devices capture a plurality of subjects from different directions.

The present invention is an information processing apparatus that determines imaging conditions of a plurality of imaging apparatuses that capture an image of a subject from different viewpoint positions, an acquisition unit that acquires three-dimensional shape information of the subject, and 3 of the acquired subject. Determining means for determining the imaging conditions of each of the plurality of imaging devices based on the dimensional shape information , wherein the imaging conditions include an in-focus distance, an aperture value, ISO sensitivity, and a shutter speed. To do.

  According to the present invention, even when a plurality of imaging devices capture a plurality of subjects from different directions, each imaging device can focus on one of the plurality of subjects.

It is a figure showing an example of composition of an imaging system. It is a figure which shows the internal structure of information processing apparatus. It is a figure which shows the function structure of information processing apparatus. 3 is a flowchart illustrating a flow of the entire process in the first embodiment. It is a figure which shows the relationship between the point on a captured image, and a to-be-photographed point. It is a figure which shows the example of the captured image acquired from the imaging parts 101-108. 10 is a flowchart showing a flow of processing of an attention subject shape information extraction unit 303. FIG. 10 is a diagram for explaining processing of a target subject shape information extraction unit 303. It is a figure which shows the example of the shape information in Example 1. FIG. 6 is a diagram illustrating an example of target subject shape information in Embodiment 1. FIG. 3 is a flowchart illustrating a processing flow of an imaging condition determination processing unit 304 according to the first exemplary embodiment. It is a figure for demonstrating a depth value. FIG. 6 is a diagram showing target subject points arranged in order of depth value. 10 is a diagram illustrating an example of subject shape information in Embodiment 2. FIG. 10 is a diagram illustrating an example of target subject shape information in Embodiment 2. FIG.

[Example 1]
<Imaging system>
FIG. 1 is a diagram illustrating a configuration example of an imaging system according to the present embodiment. The imaging system includes imaging units 101 to 108 and an information processing device 109. The imaging units 101 to 108 are imaging devices that receive optical information of a subject with a sensor and obtain digital data (captured image data) of a captured image by performing A / D conversion. When the imaging units 101 to 108 capture a plurality of subjects, the information processing device 109 captures the imaging units 101 to 108 so that each imaging device focuses on one subject among the plurality of subjects. The imaging condition is determined. And the imaging parts 101-108 are controlled according to the determined imaging condition.

<Internal configuration of information processing apparatus>
FIG. 2 is a block diagram showing an internal configuration of the information processing apparatus 109. The information processing apparatus 109 includes a CPU 201, a RAM 202, a ROM 203, an HDD I / F 204, an HDD 205, an input I / F 206, an output I / F 207, and a system bus 208.

  The CPU 201 executes a program stored in the ROM 203 using the RAM 202 as a work memory, and comprehensively controls each component to be described later via the system bus 208. Thereby, various processes described later are executed.

  The HDD I / F 204 is an interface such as serial ATA (SATA), for example, and connects the HDD 205 as a secondary storage device. The CPU 201 can read data from the HDD 205 and write data to the HDD 205 via the HDD I / F 204. Further, the CPU 201 can expand the data stored in the HDD 205 to the RAM 202, and can similarly store the data expanded in the RAM 202 in the HDD 205. The CPU 201 can regard the data expanded in the RAM 202 as a program and execute it. The secondary storage device may be a storage device such as an optical disk drive in addition to the HDD.

  The input I / F 206 is a serial bus interface such as USB or IEEE1394. The CPU 201 via the input I / F 206 from the imaging units 101 to 108, the operation unit 209 (for example, mouse and keyboard), the external memory 211 (for example, hard disk, memory card, CF card, SD card, USB memory) and the like. Get the data. What data is acquired will be described later.

  The output I / F 207 is a video output interface such as DVI or HDMI (registered trademark). The CPU 201 displays captured images of the imaging units 101 to 108 and an arbitrary viewpoint image described later on the display unit 210 (various output devices such as a display) via the output I / F 207. Note that a touch panel display may serve as the operation unit 209 and the display unit 210.

<Functional configuration of information processing apparatus>
FIG. 3 is a block diagram illustrating a functional configuration of the information processing apparatus 109 according to the present embodiment. The information processing apparatus 109 includes a shape information acquisition unit 301, an attention subject designation information acquisition unit 302, an attention subject shape information extraction unit 303, and an imaging condition determination unit 304. The shape information acquisition unit 301 includes a captured image data acquisition unit 305, an imaging unit position and orientation information acquisition unit 306, and a shape information estimation unit 307.

  FIG. 4 is a flowchart illustrating a flow of a series of processes according to the present embodiment. Hereinafter, a flow of a series of processes according to the present embodiment will be described with reference to FIG.

<Acquisition of subject shape information>
In step S401, the shape information acquisition unit 301 acquires subject shape information (three-dimensional shape information). Hereinafter, the process of step S401 will be described in detail.

  The captured image data acquisition unit 305 acquires captured image data from the imaging units 101 to 108 via the input I / F 206. Here, the acquired captured image data is image data obtained by capturing the subject from the respective viewpoint positions of the imaging units 101 to 108.

  The imaging unit position / orientation information acquisition unit 306 acquires imaging unit position / orientation information from the imaging units 101 to 108 or the external memory 211 via the input I / F 206. The image capturing unit position and orientation information includes position information and orientation information when the image capturing units 101 to 108 image the subject. The position information can be three-dimensional coordinates representing the position of the imaging unit. The posture information can be a three-dimensional vector that represents the imaging direction of the imaging unit, and can include, for example, a three-dimensional vector that represents the front and a three-dimensional vector that represents the top.

  The shape information estimation unit 307 estimates the shape information of the subject from the captured image data acquired by the captured image data acquisition unit 305 and the imaging unit position and orientation information acquired by the imaging unit position and orientation information acquisition unit 306. In this embodiment, as shown in FIG. 9, the shape information is obtained when each subject point associated with an identification number of a point on the subject (hereinafter referred to as a subject point) is viewed from the imaging units 101 to 108. Is the depth value. As a method for calculating the shape information from the captured image data and the imaging unit position and orientation information, for example, the shape estimation method described in US Patent Application Publication No. 2009/0052796 can be used. That is, the subject shape is estimated by generating points on the subject so that the pixel values of the images captured by the plurality of imaging devices can be matched. Hereinafter, an outline of a method for calculating shape information will be described with reference to FIG. 5 showing a relationship between a point on a captured image and a subject point.

  First, corresponding point matching between captured images is performed. Corresponding point matching is a process for determining corresponding points on different captured images showing the same subject point and calculating the two-dimensional coordinates of these corresponding points. For example, the two-dimensional coordinates of the corresponding points 511 and 512 corresponding to the subject point 510 and the corresponding points 521 and 522 corresponding to the subject point 520 are calculated. In order to determine corresponding points, for example, feature detection using operators such as Harris and DOC (Difference-of-Gaussian) can be performed, and feature matching can be performed between captured images.

  Next, the three-dimensional coordinates of the subject points 510 and 520 are calculated from the calculated two-dimensional coordinates of the corresponding points 511, 512, 521, and 522 and the imaging unit position and orientation information using the principle of projective geometry.

  Finally, the depth values of the subject points 510 and 520 are calculated for all combinations of the imaging units 101 to 108 and the subject points 510 and 520 from the calculated three-dimensional coordinates of the subject points 510 and 520 and the imaging unit position and orientation information. To do. As shown in FIG. 12, the depth value refers to the distance (distance in the optical axis direction) from the intersection of the perpendicular line drawn from the subject point to the optical axis of the imaging unit and the optical axis to the optical center of the imaging unit. And can be calculated using the principle of projective geometry. Although an arbitrary unit may be used as the unit of the depth value, in the following description, the unit of the depth value is assumed to be cm.

  In this embodiment, the image capturing unit position / orientation information is acquired from the outside by the image capturing unit position / orientation information acquisition unit 306, but the present invention is not limited to this. The imaging unit position / orientation information acquisition unit 306 may calculate the imaging unit position / orientation information from the captured image data acquired from the captured image data acquisition unit 305 by an existing method such as structure from motion.

  In the present embodiment, the shape information acquisition unit 301 calculates the three-dimensional coordinates of each subject point from the captured image data and the imaging unit position / orientation information, and calculates the calculated three-dimensional coordinates of each subject point and the imaging unit position. Although the depth value of each subject point is calculated from the posture information, the present invention is not limited to this. The shape information acquisition unit 301 may acquire the three-dimensional coordinates of each subject point from the outside, and calculate the depth value of each subject point from the acquired three-dimensional coordinates of each subject point and the imaging unit position and orientation information. . Alternatively, the shape information acquisition unit 301 may acquire the depth value of each subject point from the external memory 211 via the input I / F 206 instead of calculating the depth value of each subject point internally.

<Acquisition of target subject specification information>
In step S <b> 402, the target subject designation information acquisition unit 302 acquires target subject designation information from the operation unit 209 via the input I / F 206. The subject of interest designation information is information for designating which subject is to be focused, and is, for example, two-dimensional coordinates designated by the user on the captured image. The subject of interest can be specified by the subject of interest designation information. Hereinafter, a method for acquiring attention subject designation information will be described with reference to FIG. 6 illustrating an example of a captured image acquired from the imaging units 101 to 108.

  First, the captured image is displayed on the display unit 210 via the output I / F 207. Then, the two-dimensional coordinates of the position designated by the user on the captured image (the user designated position in FIG. 6) are acquired from the operation unit 209. Here, the display unit 210 and the operation unit 209 are shown separately in FIG. 2, but may be the same. The image displayed on the display unit 210 may be an image captured by any of the image capturing units 101 to 108. The display unit 210 may display an arbitrary viewpoint image generated using the captured images of the imaging units 101 to 108 and the shape information of the subject acquired in step S401.

<Extracting subject shape information of interest>
In step S403, the subject-of-interest shape information extraction unit 303 extracts shape information of the subject to be focused on (target subject shape information) from the shape information acquired in step S401 and the subject-of-interest designation information acquired in step S402. .

  FIG. 7 is a flowchart showing the processing flow of the target subject shape information extraction unit 303. FIG. 8 is a diagram for explaining the processing of the target subject shape information extraction unit 303. Hereinafter, the details of the processing of the target object shape information extraction unit 303 will be described with reference to FIGS. 7 and 8.

  In step S701, the subject point (shape information) acquired in step S401 is polygonized. For example, the subject point shown in FIG. 8A can be polygonized (subdivided into a plurality of polygons) as shown in FIG. 8B. For example, an existing method such as Poisson Surface Reconstruction can be used as a method for forming a polygon.

  In step S702, the target subject polygon is extracted from the polygons generated in step S701 using the target subject specifying information acquired in step S402. FIG. 8C shows the user specified position (target subject specifying information) superimposed on FIG. 8B. A polygon designated by the subject-of-interest designation information, a peripheral polygon sharing a side with the designated polygon, and a polygon group sharing a side with the peripheral polygon are extracted as a subject of interest. In the example of FIG. 8C, the subject 1 is extracted as the subject of interest.

  In step S703, shape information of the subject of interest (target subject shape information) is extracted. For example, when the shape information acquired in step S401 is shown in FIG. 9, and the subject points with identification numbers 2, 4, 5, 9, 11, and 12 are extracted as the subject of interest, the subject shape of interest shown in FIG. Information is extracted.

  Through the above processing, the target subject shape information is extracted based on the shape information and the target subject designation information.

<Determination of imaging conditions>
In step S404, the imaging condition determination unit 304 determines the imaging condition from the target subject shape information extracted in step S403.

  FIG. 11 is a flowchart illustrating a flow of processing in which the imaging condition determination unit 304 according to the first embodiment determines the imaging conditions of one of the imaging units 101 to 108. Hereinafter, with reference to FIG. 11, the details of the processing of the imaging condition determination unit 304 in the first embodiment will be described.

  In step S1101, the imaging condition determination unit 304 determines the in-focus distance. Hereinafter, the determination of the in-focus distance will be described with reference to FIG. 13 in which the target subject points are arranged in order of depth values. In this embodiment, it is desirable to focus on the entire subject of interest. Therefore, with respect to each imaging unit, the focusing distance of each imaging unit can be around the average value of the minimum depth value and the maximum depth value of the subject of interest from the imaging unit. Hereinafter, the in-focus distance is represented by d. For example, in the example of the imaging unit 101 in FIG. 10, since the minimum depth value is 5 cm and the maximum depth value is 22 cm, the focusing distance d of the imaging unit 101 is (5 + 22) /2=13.5 (cm). can do. However, the method for determining the focus distance d is not limited to this, and in step S1102 described later, the lens aperture value (F value) such that both the minimum depth value and the maximum depth value are included in the depth of field. Can be set to an arbitrary distance.

  In step S1102, the imaging condition determination unit 304 determines the F value. Below, the determination method of F value is demonstrated using FIG.

  In order to focus on the entire subject of interest, it is necessary to satisfy the condition that both the minimum depth value and the maximum depth value of the subject of interest are included in the depth of field. The depth of field is a range from a near point distance (distance from the imaging unit to the near depth of field depth) to a far point distance (distance from the imaging unit to the depth side depth of field). . If the near point distance is represented by Df and the far point distance is represented by Db, the above condition is that the near point distance Df is not more than the minimum depth value and the far point distance Db is not less than the maximum depth value. Consider equivalent. The F value determined in this step is represented by α. For the determination of the F value α, the following formulas (1) and (2) that define the relationship between the near point distance Df and the far point distance Db, the focusing distance d, and the F value α are used.

Here, f represents the focal length of the lens. ε represents the diameter of an allowable circle of confusion, and is a value indicating how much blur is allowed. As the diameter ε of the allowable circle of confusion, for example, a value of 0.02 mm can be used.

  As another condition, it is desirable that the F value is as small as possible. The reason is that as the F value is smaller, the imaging condition determined in step S1103, which will be described later, has a lower ISO sensitivity and a faster shutter speed, and a captured image with less noise and subject blur can be obtained.

  An F value that satisfies these conditions is determined. Specifically, formulas (1) and (2) are transformed into the following formulas (3) and (4).

From Equation (3), the minimum F value that makes the near point distance Df less than or equal to the minimum depth value is α calculated by substituting the minimum depth value into the near point distance Df of Equation (3). I understand. On the other hand, from Equation (4), the minimum F value that makes the far point distance Db equal to or greater than the maximum depth value is α calculated by substituting the maximum depth value into the far point distance Db in Equation (4). I understand that. Therefore, the minimum F value at which the near point distance Df is equal to or less than the minimum depth value and the far point distance Db is equal to or greater than the maximum depth value is large among α calculated from the equations (3) and (4). Α of the direction. Further, the F value calculation method is not limited to this, and the minimum F value satisfying the expressions (1) and (2) may be calculated using an existing optimization technique.

  In step S1103, the imaging condition determination unit 304 determines other imaging conditions. Other imaging conditions include ISO sensitivity and shutter speed. The ISO sensitivity and shutter speed can be determined by existing AE (automatic exposure) processing. Note that it is arbitrary whether to preferentially determine the ISO sensitivity or the shutter speed. For example, one value of the ISO sensitivity and the shutter speed may be set in advance by the user, and the other value may be determined so that an appropriate exposure is obtained in the setting. Alternatively, the imaging condition determination unit 304 may determine both ISO sensitivity and shutter speed in consideration of the balance between noise and subject blur.

  The imaging condition determination unit 304 performs the processing of steps S1101 to S1103 for each of the imaging units 101 to 108. Thereby, an imaging condition is determined so that each of the imaging units 101 to 108 focuses on the entire target subject specified by the target subject specifying information.

  By performing the processing control described above, even when a plurality of imaging devices capture a plurality of subjects from different directions, each imaging device can focus on the entire subject of interest in the plurality of subjects. . In addition, since the imaging condition is determined based on the shape information of the subject of interest, it is possible to determine an F value so that the entire subject of interest falls within the depth of field. As a result, an imaging condition with a low ISO sensitivity and a high shutter speed is determined, so that an imaging process with less noise and subject blur can be performed.

[Example 2]
In the first embodiment, the depth value when each subject point is viewed from the imaging units 101 to 108 is acquired as the shape information of the subject, and the imaging conditions of the imaging units 101 to 108 are determined using the acquired depth values. In this embodiment, a depth value and visibility information described later are acquired as the shape information of the subject, and the imaging conditions of the imaging units 101 to 108 are determined using the acquired depth value and visibility information. Note that description of the same processing as in the first embodiment is omitted.

  FIG. 14 is a diagram illustrating an example of the shape information of the subject acquired by the shape information acquisition unit 301 in the present embodiment in step S401. As shown in FIG. 14, the shape information in this embodiment includes an identification number of a subject point, a depth value when each subject point is viewed from the imaging units 101 to 108, and visibility information. Visibility information is information indicating whether or not each subject point is visible from the imaging unit. Visibility information can be calculated during the shape estimation process in the shape information estimation unit 307. For example, in the example of FIG. 5, the three-dimensional coordinates of the subject point 510 are calculated from the correspondence between the corresponding point 511 and the corresponding point 512, and it can be seen that the subject point 510 is reflected in the captured image 501 and the captured image 502. That is, it can be seen that the subject point 510 is visible from the imaging unit that captured the captured image 501 and the imaging unit that captured the captured image 502.

  Details of the processing of the target object shape information extraction unit 303 in the present embodiment will be described with reference to FIG. In addition, since the process of step S701 and S702 is the same as Example 1, description is abbreviate | omitted.

  In step S703, the target subject shape information extraction unit 303 extracts target subject shape information. In the present embodiment, only the shape information of the subject point that is visible from each imaging unit (visibility is ON in FIG. 14) among the target subject is extracted as the target subject shape information. For example, when the shape information acquired in step S401 is shown in FIG. 14, and the subject points of identification numbers 2, 4, 5, 9, 11, and 12 are extracted as the subject of interest, the subject shape of interest shown in FIG. Information is extracted. As shown in FIG. 15, the shape information of subject points with identification numbers 5, 9, and 11 is extracted with respect to the imaging unit 101, and the shape information of subject points with identification numbers 2, 4, and 5 is extracted with respect to the imaging unit 102. Regarding the imaging unit 103, the shape information of the subject points with the identification numbers 2 and 9 is extracted.

  As described above, it is possible to focus only on subject points that are visible from each imaging unit among subject points that constitute the subject of interest. By removing the invisible subject point from the depth of field, each imaging unit can determine an imaging condition with a smaller F value while focusing on the entire visible portion of the subject of interest. By reducing the F value, the ISO sensitivity is smaller and the shutter speed is faster, so that a captured image with less noise and subject blur can be obtained.

(Other examples)
The present invention is not limited to the image forming apparatus having the conveyance path form described in each embodiment. In other words, any image forming apparatus can be used as long as it can read a document image with a backing paper as a document, determine a margin area based on the read document image data, and form an image corresponding to the print content in the margin area. The present invention can be applied even in the form.

  In the present embodiment, the backing paper printing in the on-print processing has been described. However, the present invention can be applied to other forms of printing processing. For example, a configuration in which an image reading unit is provided in the vicinity of a paper feeding unit that feeds recording paper may be employed. Then, the back paper may be stored in the paper supply unit.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (12)

An information processing apparatus that determines imaging conditions of a plurality of imaging apparatuses that image a subject from different viewpoint positions,
Obtaining means for obtaining three-dimensional shape information of the subject;
Determining means for determining each imaging condition of the plurality of imaging devices based on the acquired three-dimensional shape information of the subject ;
The information processing apparatus , wherein the imaging condition includes an in-focus distance, an aperture value, ISO sensitivity, and a shutter speed .   The information processing apparatus according to claim 1, wherein the acquisition unit estimates three-dimensional shape information of the subject using a plurality of images obtained by imaging the subject from different viewpoint positions by the plurality of imaging devices. apparatus.   The information processing according to claim 1, wherein the three-dimensional shape information includes a depth value indicating a distance in an optical axis direction from each of the plurality of imaging devices to each point constituting the subject. apparatus.   The determination unit is configured to capture the imaging condition so that both the minimum depth value point and the maximum depth value point are included in the depth of field among the points constituting the subject for each of the plurality of imaging devices. The information processing apparatus according to claim 3, wherein the information processing apparatus is determined.   The determination unit is configured to determine, for each of the plurality of imaging apparatuses, a minimum depth value such that both the minimum depth value point and the maximum depth value point are included in the depth of field among the points constituting the subject. The information processing apparatus according to claim 4, wherein an aperture value is determined. A means for identifying the subject of interest;
Extraction means for extracting the three-dimensional shape information of the subject of interest specified by the specifying means from the three-dimensional shape information of the subject; and the determining means includes the extracted three-dimensional shape information of the subject of interest. The information processing apparatus according to claim 1, wherein an imaging condition for each of the plurality of imaging apparatuses is determined based on the information processing apparatus.   The extraction means subdivides the subject into a plurality of polygons based on the three-dimensional shape information of the subject, the polygon specified by the specifying means, a peripheral polygon sharing an edge with the specified polygon, and the peripheral The information processing apparatus according to claim 6, wherein a polygon group sharing a side with a polygon is extracted as the subject of interest. The three-dimensional shape information includes visibility information indicating whether or not each point constituting the plurality of subjects is visible from each imaging device,
The extraction means, for each of the plurality of imaging devices, based on the visibility information, the three-dimensional shape information of only the visible points among the points constituting the target subject as the three-dimensional shape information of the target subject. The information processing apparatus according to claim 6, wherein the information processing apparatus is extracted.   The information processing apparatus according to claim 8, wherein the acquisition unit acquires the visibility information using a plurality of images obtained by imaging the subject from different viewpoint positions by the plurality of imaging apparatuses. An imaging apparatus that images a subject from different viewpoint positions together with another imaging apparatus,
Acquisition means for acquiring three-dimensional shape information of the subject; and determination means for determining an imaging condition of the imaging device based on the acquired three-dimensional shape information of the subject, wherein the imaging condition is a focusing distance , aperture value, ISO sensitivity, and shutter including speed, imaging apparatus characterized by imaging using the imaging conditions determined by the information processing apparatus having a determining means. An information processing method for determining imaging conditions of a plurality of imaging devices that image a subject from different viewpoint positions,
An acquisition step of acquiring three-dimensional shape information of the subject;
Determining each imaging condition of the plurality of imaging devices based on the acquired three-dimensional shape information of the subject ,
The information processing method , wherein the imaging condition includes an in-focus distance, an aperture value, ISO sensitivity, and a shutter speed . Program for functioning as an information processing apparatus according to computer to any one of claims 1 to 9.

Images