JP6498429B2 - Distance image generation device, distance image generation method and program - Google Patents

Description

  The present invention relates to a distance image generation technique using a time-of-flight distance image sensor, and more particularly to a distance value calculation technique.

  There is a distance image generation device that generates a distance image having a pixel value of a distance from an object in an imaging target space using the optical flight distance image sensor. Among them, the time-of-flight (TOF method: Time Of Flight method) distance image sensor includes a light source that emits modulated light and a light receiving element that receives the modulated light, emits modulated light, and receives the light for each pixel. A distance value is calculated based on the reflected light (see, for example, Patent Document 1). Such an optical flight-type distance image sensor includes a sort-type TOF type in which charges generated by reflected light of received modulated light are synchronized with the modulation frequency of the modulated light and the charges are distributed (for example, Patent Documents). 2).

  As a performance index of the distance image sensor, there are an absolute distance accuracy indicating how much the actual distance and the measurement distance coincide with each other, and a repeat distance accuracy indicating a measurement variation. As a technique for improving the absolute distance accuracy, for example, there is a technique for removing measurement errors using pixel values calculated in environments of different brightness (see, for example, Patent Document 3). Further, as a technique for improving the repeat distance accuracy, for example, there are a technique for taking a spatial average of pixel values (for example, see Patent Document 4) and a technique for taking a time average.

  On the other hand, in order to reduce the noise of the distance image, there is a technique of going back to the past frame and adding the pixel value of the corresponding pixel to the pixel value of the current frame until the pixel value becomes a predetermined value or more (for example, patents) Reference 5).

JP 2005-235893 A JP 2011-179925 A JP 2014-70936 A Japanese Patent No. 3,906,859 JP 2011-122913 A

  When the repeat distance accuracy is low, the distance value frequently fluctuates greatly, and the reliability of measurement also decreases. For example, when a distance image sensor is used for object recognition or the like, an object may be misrecognized due to a large variation in distance value, or a recognized object may not be recognized suddenly. When a distance image sensor is used for a motion gesture or the like, erroneous recognition of the gesture may occur due to a large variation in the distance value, and an unexpected operation may be induced.

  Since the method of Patent Document 3 aims at improving the absolute distance accuracy as described above, only the absolute distance accuracy is improved. Further, according to the method of Patent Document 4 and the technique for calculating the time average, the effective number of pixels and the frame rate are reduced.

  Furthermore, in the technique disclosed in Patent Document 5, addition of past frames must be continued until a predetermined intensity value is reached or the set number of frames is reached, and processing such as comparison and addition for each pixel must be repeated. For this reason, in the case of a pixel having a low intensity value, addition is required up to the set number of frames, which is the upper limit of addition, and the amount of processing becomes enormous. Reduction of noise results in reduction of repetition error. However, for example, in a situation where the intensity value of the entire screen is low, enormous processing is required to reduce noise.

  The present application has been made in view of the above circumstances, and in a distance image generation apparatus using an optical time-of-flight distance image sensor, variation in calculation distance of measurement can be performed without reducing the effective number of pixels and frame rate. It aims at providing the technique which suppresses and improves a repetition distance precision.

  The present invention corrects the amount of charge (charge amount) accumulated in each charge accumulation unit that is a source of distance value calculation. The correction is performed using an intermediate value calculated from the maximum value and the minimum value of the charge amount of a predetermined number of frames that are the latest. The correction amount is determined so as to increase the correction amount so as to approach the intermediate value and the difference between the maximum value and the minimum value is large, and correction is performed using the correction amount.

  Specifically, a light source unit including a light source that irradiates modulated light modulated into the target space, and light including reflected light that is irradiated from the light source unit and reflected by the surface of the target in the target space is received. A photoelectric conversion element provided for each pixel to be converted into a charge, a plurality of charge storage units provided for each photoelectric conversion element, and a charge converted by the photoelectric change element in synchronization with modulation of the modulated light A charge distribution unit that distributes the plurality of charge storage units, and a distance image generation unit that generates a distance image whose pixel value is a distance value using the amount of charge stored in each of the charge storage units. The distance image generation unit corrects the charge amount accumulated in each of the charge accumulation units, and a distance value calculation unit calculates the distance value using the charge amount corrected by the correction unit. The correction unit includes a predetermined number of frames. Using the intermediate value calculated from the maximum and minimum values of each of the charge storage unit charges accumulated amount, to provide a distance image generation apparatus characterized by correcting each said charge amount.

  In addition, a light source unit including a light source that irradiates modulated light that is modulated into the target space, and light including reflected light that is irradiated from the light source unit and reflected from the surface of the target object in the target space is received and converted into charges. A plurality of photoelectric conversion elements provided for each pixel; a plurality of charge storage units provided for each of the photoelectric conversion elements; and the plurality of charges converted by the photoelectric conversion elements in synchronization with modulation of the modulated light. A distance image generation method in a distance image generation device comprising a charge distribution unit that distributes the charge storage units, wherein the maximum value and the minimum value of the amount of charge stored in each of the charge storage units during a predetermined frame The intermediate value is calculated from the above, the charge amount of the charge storage unit is corrected using the intermediate value, and the distance image whose pixel value is the distance value is generated using the corrected charge amount. A distance image generation method is provided.

  According to the present invention, in a distance image generation apparatus using a time-of-flight distance image sensor, the measurement repeatability accuracy is improved without reducing the effective number of pixels and the frame rate.

It is a block diagram of the distance image generation device of a first embodiment. It is explanatory drawing for demonstrating the relationship between the modulation signal of 1st embodiment, and a gate signal. It is explanatory drawing for demonstrating the principle of the distance image generation by the optical time-of-flight distance image generation apparatus. It is explanatory drawing for demonstrating a repetition distance error generation factor. It is explanatory drawing for demonstrating a repetition distance error generation factor. It is explanatory drawing for demonstrating the database of 1st embodiment. It is a flowchart of the distance image generation process of 1st embodiment. It is a flowchart of the correction process of 1st embodiment. It is a block diagram of the distance image generation device of a second embodiment. It is explanatory drawing for demonstrating the database of 2nd embodiment. It is a flowchart of the distance image generation process of 2nd embodiment. It is a flowchart of the fluctuation | variation detection process of 2nd embodiment.

<< First Embodiment >>
Hereinafter, a first embodiment of the present invention will be described. Hereinafter, in all the drawings for explaining the embodiments of the present invention, those having the same function are denoted by the same reference numerals, and repeated explanation thereof is omitted.

<Distance image generator configuration>
First, the configuration of the distance image generation device of this embodiment will be described. FIG. 1 is a functional block diagram of the distance image generation apparatus of the present embodiment.

  As shown in the figure, the distance image generating apparatus 100 of the present embodiment generates a distance image using a phase difference between irradiated modulated light 111 and incident light 112, and includes a light source unit 110, photoelectric conversion, and the like. Unit 120, charge distribution unit 130, control unit 140, charge storage unit 150, and distance image generation unit 170.

  Note that the distance image generation device 100 of this embodiment includes a CPU, a memory, and a storage device. The charge distribution unit 130, the control unit 140, and the distance image generation unit 170 are realized by a CPU loading a program stored in a storage device in advance into a memory and executing the program.

  The light source unit 110 includes a light source that irradiates modulated light (for example, infrared light or visible light modulated at high speed with a sine wave or a rectangular wave) 111 in a target space under the control of the control unit 140 described later. A device capable of high-speed modulation such as an LED is used as the light source.

  The photoelectric conversion unit 120 includes light (incident light) including reflected light, which is reflected by the object (object) 113 in the target space (the field of view of the distance image generation device 100) of the modulated light 111 emitted from the light source of the light source unit 110. ) 112 is received and converted into electric charge. The photoelectric conversion unit 120 includes a plurality of photoelectric conversion elements that convert charges according to the amount of received light. For this reason, each photoelectric conversion element is provided for each pixel, and is regularly arranged in association with each pixel of the intensity image and the distance image. A lens may be disposed in front of the photoelectric conversion unit 120.

  The charge distribution unit 130 distributes the charge converted by the photoelectric conversion element to the charge storage unit 150 (to be described later) under the control of the control unit 140 (to be described later). The sorting is performed for each photoelectric conversion element. In the present embodiment, in synchronization with the modulation of the modulated light 111, the charges converted by the photoelectric conversion elements are distributed to a plurality of charge storage units 150 provided in association with the photoelectric conversion elements.

  The charge storage unit 150 stores the charge distributed by the charge distribution unit 130. The phase difference between the modulated light 111 and the incident light 112, which is distance information, is calculated using the charges accumulated in the charge accumulation unit 150. In order to calculate this phase difference, at least three or more pieces of phase information (charges accumulated in the charge accumulation unit) are necessary.

  Hereinafter, in the present embodiment, description will be made assuming that there are four charge storage units 150. That is, in the present embodiment, this set of four charge storage units 150 is provided for each pixel. When distinguishing each charge storage part, it is set as 151, 152, 153, 154, respectively. The charge distribution unit 130 distributes the charges to the four charge storage units 151, 152, 153, and 154 in a predetermined order.

  The charge storage unit 150 may store the charge itself, or may store data after AD conversion of this charge amount. Hereinafter, in this embodiment, both the charge itself and the data after AD conversion are stored in the charge storage unit 150 without distinction, and are referred to as charges, and the amount is referred to as a charge amount.

  These photoelectric conversion elements, the charge distribution unit 130, and the charge storage unit 150 form a pixel.

  The control unit 140 controls the light source unit 110 and the charge distribution unit 130 in synchronization. Control is performed by the modulation signal and the gate signal.

  The modulation signal is a signal that instructs modulation and output timing of the modulated light, and is output toward the light source unit 110. The light source unit 110 generates modulated light 111 modulated by the modulation signal and irradiates the target space. The irradiated modulated light 111 is reflected by the object 113 and becomes reflected light. The reflected light enters the photoelectric conversion unit 120 as incident light 112, is converted into charges, and is sent to the charge distribution unit 130.

  The gate signal is a signal for instructing the charge distribution to each charge storage unit 150 and is output toward the charge distribution unit 130. The charge distribution unit 130 distributes the charge to each charge storage unit 150 according to the gate signal.

  The control unit 140 generates a gate signal that is synchronized with the modulation period of the light source unit 110 and is shifted by a time corresponding to the modulation period and the number of the charge distribution units 130. For example, if the modulation period is 10 MHz and there are four charge storage units, the gate signal is generated at a timing shifted by a time (25 ns) obtained by dividing the modulation period by four. As a result, the charges are distributed to the charge storage units 151, 152, 153, and 154 according to the gate signal.

  An example of the output timing of the modulation signal and the gate signal of this embodiment is shown in FIG. Here, the modulation signal 114 and the gate signal 115 are shown in the case where charges are sequentially distributed to the four charge storage units 151, 152, 153, and 154 in a period obtained by dividing one period Tq of the modulation signal into four equal parts.

  In addition, the control unit 140 transmits a read signal to the distance image generation unit 170 to read out the charges accumulated in the charge accumulation units 151, 152, 153, and 154, respectively. Note that the modulation frequency of the modulated light 111 is several tens of MHz. Therefore, one modulation period is about several tens of ns. In order to obtain a distance image, a charge accumulation time Tf of several hundred to several hundred thousand cycles is required. This charge accumulation time Tf is determined in advance.

  The control unit 140 of the present embodiment outputs a read signal so that the distance image generation unit 170 reads out charges from the charge storage units 151, 152, 153, and 154 every charge storage time Tf. One distance image obtained at every charge accumulation time Tf interval is called a frame. The charge accumulation time Tf is referred to as 1 frame time Tf.

  The modulated signal 114, the gate signal 115, and the readout signal are collectively referred to as a control signal.

  The distance image generation unit 170 reads the charges accumulated in the charge accumulation units 151, 152, 153, and 154 according to the readout signal of the control unit 140, calculates a distance value for each pixel from each charge (charge amount), and calculates A distance image having the distance value thus obtained as a pixel value is generated. The distance value is a value of the distance to the object 113 in the visual field. Details of the calculation will be described later.

[Principle of TOF type distance image generation]
Here, the principle of distance image generation in the time-of-flight (TOF method) distance image generation apparatus 100 will be described. FIG. 3 is a diagram for explaining the principle of distance image generation.

  When the intensity of the modulated light 111 emitted from the light source unit 110 changes so as to draw a sine curve as shown in the figure, the intensity of the incident light 112 to the photoelectric conversion unit 120 also changes so as to draw a sine curve. At this time, the modulated light 111 and the incident light 112 have a phase delay (phase difference φ) due to the time of flight of the light traveling back and forth to the object.

Since the speed c of light is known, the calculated distance value D to the object 113 is calculated by the following equation (1) using the phase difference φ and the modulation frequency f.

  Therefore, if the phase difference φ is known, the calculated distance value D can be obtained. In the present embodiment, as described above, one period Tp of the modulated light is equally divided into four, and the charge amounts acquired in the respective periods are stored in the four charge storage units 151, 152, 153, and 154, respectively.

なお、１周期Ｔｐを４等分した各期間は、例えば、０度から９０度の間、９０度から１８０度の間、１８０度から２７０度の間、２７０度から０度の間とする。 When the charge amounts accumulated in the charge storage units 151, 152, 153, and 154 in the respective periods obtained by dividing one period of the modulated light 111 into four are C1, C2, C3, and C4, the modulated light 111 and the incident light The phase difference φ from 112 is expressed by the following equation (2).

Each period obtained by equally dividing one cycle Tp into four is, for example, between 0 degrees and 90 degrees, between 90 degrees and 180 degrees, between 180 degrees and 270 degrees, and between 270 degrees and 0 degrees.

[Distance image generator]
Next, the distance image generation unit 170 of this embodiment will be described. In the present embodiment, it is assumed that the target unit 113 is stationary.

  The distance image generation unit 170 uses the charge amounts C1, C2, C3, and C4 accumulated in the charge accumulation units 151, 152, 153, and 154 during the above-described charge accumulation time Tf, and the pixel value is a distance value. Generate a distance image. The distance value D is calculated for each pixel according to the above equation (1) by obtaining the phase difference φ according to the above equation (2) and using the obtained phase difference φ.

  As described above, the charge distribution unit 130 distributes charges to the charge storage units 151, 152, 153, and 154 in synchronization with the modulation of the modulated light 111. Even if the object 113 is stationary, since a semiconductor switch is used for sorting, random charges such as switching noise and ktc noise cause the charge storage units 151, 152, 153, and 154 to be charged at each charge storage time Tf. The amount of charge distributed is different. If the distance value is calculated using this charge amount as it is, an error occurs in the obtained distance value D between frames. This error is called a repetition distance error.

  In order to suppress this repeated distance error, the distance image generation unit 170 of the present embodiment corrects the charge (charge amount) read from the charge storage unit 150, and uses the corrected charge amount to calculate the distance value. calculate. Therefore, the distance image generation unit 170 according to the present embodiment corrects the charge amounts C1, C2, C3, and C4 accumulated in the charge accumulation units 151, 152, 153, and 154, as shown in FIG. 171, a database 172 that stores data used for correction, and a distance value calculation unit 173 that calculates a distance value D using the corrected charge amount.

[Principle of repeated distance error generation]
The occurrence of the repetition distance error will be described with reference to FIGS. FIG. 4 is a diagram for explaining how the measurement distance varies for each frame in the charge distribution type TOF type distance image generation apparatus 100.

  As shown in this figure, due to the noise of the semiconductor switch that performs the distribution, even if the object 113 is at the same distance, the charge amount C1 that is distributed to each of the charge storage units 151, 152, 153, and 154 for each frame , C2, C3, and C4 are different.

  The phase φ is the difference (C1-C3, C2-C4; difference between the charge amounts C1, C2, C3, C4 stored in the charge storage units 151, 152, 153, 154 as shown in the equation (2). (Hereinafter referred to as differential charge amount). FIG. 5 shows the relationship between the variation in the differential charge amount and the phase variation. In this figure, the horizontal axis indicates the value of the differential charge amount C1-C3, and the vertical axis indicates the value of the differential charge amount C2-C4. As shown in FIG. 5, the variation in the differential charge amount C1-C3 and the differential charge amount C2-C4 is a variation in the phase φ.

  The correction unit 171 of the present embodiment uses the magnitude of the variation in the differential charge amount C1-C3 and the differential charge amount C2-C4 due to the distributed charge variation, and the differential charge amount C1-C3 and the difference. The charge amount C2-C4 is corrected.

[Correction method by correction unit]
The correction unit 171 includes charge amounts C1, C2, C3, and C4 accumulated in the charge accumulation units 151, 152, 153, and 154 of a predetermined number of frames (in this embodiment, differential charge amounts C1-C3, C2- Each charge amount (differential charge amount C1-C3, C2-C4) is corrected using an intermediate value calculated from the maximum value and the minimum value of C4).

  In other words, the correction unit 171 sets the maximum value Cmax and the minimum value of the difference charge amounts C1-C3 and C2-C4 of the most recent and predetermined number of frames for each of the charge storage units 151, 152, 153, and 154. Cmin is specified. Then, using the intermediate value (intermediate value) between the maximum value Cmax and the minimum value Cmin, the differential charge amount is corrected. Hereinafter, the predetermined number of frames is F (F is an integer of 1 or more).

In the present embodiment, for example, a median value (median value) Cmid between the maximum value Cmax and the minimum value Cmin is used as the intermediate value. The median value Cmid is expressed by the following formula (3).
Cmid = (Cmax + Cmin) / 2 (3)

  The correction unit 171 according to the present embodiment is configured so that each charge amount (difference charge amount C1-C3, C2-C4) approaches each intermediate value (median value Cmid) (difference charge amounts C1-C3, C2). -C4) is corrected.

  When the number of frames is very large and the target portion 113 is not displaced, the variation in the differential charge amounts C1-C3 and C2-C4 follows a normal distribution. For this reason, it is considered that the true values of the differential charge amounts C1-C3 and C2-C4 substantially coincide with the median value Cmid of the maximum value Cmax and the minimum value Cmin. Note that the true values of the differential charge amounts C1-C3 and C2-C4 do not include noise due to the semiconductor switch, and the charges obtained by the photoelectric conversion unit 120 are accurately stored in the charge storage units 151, 152, 153, 154. It means the value obtained when it is assigned.

  When the number of frames is small, it cannot be said that the median value Cmid substantially coincides with the true values of the differential charge amounts C1-C3 and C2-C4. However, it is considered to be closer to the true values of the differential charge amounts C1-C3 and C2-C4 than the maximum value Cmax or the minimum value Cmin. Therefore, by correcting the measured differential charge amounts C1-C3, C2-C4 using the median value Cmid obtained from the maximum value Cmax and the minimum value Cmin, a value closer to the true value of the differential charge amount And the variation of the differential charge amounts C1-C3 and C2-C4 can be reduced.

In addition, the correction unit 171 of the present embodiment increases the correction amount of each of the charge amounts (difference charge amounts C1-C3, C2-C4) as the difference between the maximum value Cmax and the minimum value Cmin increases. That is, the correction unit 171 calculates the magnitudes of variations of the differential charge amounts C1-C3 and the differential charge amounts C2-C4 from the maximum value Cmax and the minimum value Cmin, and increases the correction amount as the variation increases. . The variation Cband is calculated by, for example, the following formula (4).
Cband = Cmax−Cmin (4)

The correction unit 171 corrects each of the differential charge amounts C1-C3 and the differential charge amounts C2-C4 so as to approach the respective median values Cmid and increase the correction amount as the variation Cband increases. Specifically, for example, the correction is made by the following equation (5).
C ′ = C− (C−Cmid) × (Cband / A) (5)
Here, C is the differential charge amount C1-C3 or the differential charge amount C2-C4, A is a constant set for each pixel from time to time according to the surrounding environment, etc., and C ′ is the corrected charge amount. The difference charge amount C1-C3 or the difference charge amount C2-C4.

When it is difficult to set the constant A for each pixel, the constant A is, for example, the charge amounts C1, C2, C3, and C4 (differential charges) of the charge storage units 151, 152, 153, and 154 of the pixel. You may calculate by the following formula | equation (6) using quantity C1-C3, C2-C4).

Alternatively, the constant A may be calculated by the following equation (7) using the maximum values Cmax and Cmin of the pixel.
A = | Cmax | + | Cmin | (7)

  The maximum value Cmax and the minimum value Cmin are determined by comparing the magnitudes of the difference charge amounts of the latest F frames.

  The correction of the differential charge amount C1-C3 and the differential charge amount C2-C4 by the correction unit 171 of the present embodiment is not limited to the above method. The maximum value Cmax and the minimum value Cmin of the difference charge amount of the latest F frame determined in advance may be used to correct the difference charge amounts closer to the true values of the difference charge amounts C1-C3 and C2-C4. For example, the calculated intermediate value may be used as a correction value as it is.

  Further, the intermediate value is not limited to the median value Cmid. Any value that is close to the true value of the varying charge amount and that can be calculated based on the maximum value Cmax and the minimum value Cmin may be used.

[Distance value calculator]
The distance value calculation unit 173 of the present embodiment replaces the differential charge amount C1-C3 and the differential charge amount C2-C4 with the corrected differential charge amount (C1-C3) ′ and the differential charge amount (C2-C4) ′. Is substituted into the above equation (2) to calculate the distance value of each pixel.

[Database]
The database 172 accumulates the respective differential charge amounts C1-C3 and differential charge amounts C2-C4 of the latest predetermined number of frames F that the correction unit 171 uses for correction.

  The database 172 has a capacity capable of storing the values of the differential charge amounts C1-C3 and the differential charge amounts C2-C4 for a preset number of storage frames (F frame in the above example) with each pixel position as an address.

  For example, it is assumed that the distance image generated by the distance image generating apparatus 100 is 128 × 128 pixels, and the charge amount for the three most recent frames is stored for correction processing. In this case, a memory having a storage capacity of 3 frames (16384 × 3 frames) of 128 × 128 = 16384 addresses is prepared.

  FIG. 6 shows an image of the memory structure of the database 172 of this embodiment. As shown in this figure, the value Af (m, n) of the differential charge amount C1-C3 and the differential charge amount C2-C4 for each database 172, each pixel (m, n), and each frame f of the present embodiment. Value Bf (m, n) is stored. Here, the total number of pixels is M × N, and the total number of frames is F. M, N, and F are integers of 1 or more. M, n, and f are integers that satisfy 1 ≦ m ≦ M, 1 ≦ n ≦ N, and 1 ≦ f ≦ F, respectively.

  In the database 172, the frame numbers f are assigned in order from the newest in time. That is, when the differential charge amounts C1-C3 and C2-C4 of a new frame are acquired, the differential charge amounts C1-C3 and C2-C4 are stored in association with the frame 1 and are already stored. The frame number associated with is updated to a larger value by one. Then, the differential charge amounts C1-C3 and C2-C4 stored in association with the frame f are deleted.

  If there is a margin in memory size, the oldest differential charge amounts C1-C3 and C2-C4 need not be deleted. However, the difference charge amounts C1-C3 and C2-C4 for the most recent F frames are used for correction.

  In addition, the number of past frames stored in the database 172 may be arbitrarily set, or may be configured to be changeable by the user.

  Actually, it is practical to determine the maximum number of frames to be stored in advance, prepare a memory having a capacity corresponding to the maximum value, and change the storage amount corresponding to the set number of frames.

[Flow of distance image generation processing]
Next, the flow of the distance image generation process of this embodiment will be described. FIG. 7 is a processing flow of distance image generation processing for generating a distance image for one frame according to the present embodiment. Here, the total number of pixels is P (P is an integer equal to or greater than 1), and the pixel to be processed is p (an integer satisfying 1 ≦ p ≦ P).

  When charges for one frame time Tf are accumulated in the charge accumulation units 151, 152, 153, and 154 of each pixel (step S1101), the distance image generation unit 170 performs the following processing (steps S1103 to S1105) for each pixel. ) Is performed for all pixels (steps S1102, S1106, and S1107).

  The distance image generation unit 170 calculates the differential charge amount C1-C3 and the differential charge amount C2-C4 (step S1103). Then, the correction unit 171 corrects the differential charge amount C1-C3 and the differential charge amount C2-C4, respectively, to obtain a corrected differential charge amount (C1-C3) ′ and a differential charge amount (C2-C4) ′. (Step S1104). Then, the distance value calculation unit 173 calculates a distance value using the corrected differential charge amount (C1-C3) 'and differential charge amount (C2-C4)' (step S1105).

  The distance image generation unit 170 generates a distance image using the calculated distance value as a pixel value (step S1108), and ends the process.

  Next, the flow of correction processing by the correction unit 171 in step S1104 will be described with reference to FIG.

  The correcting unit 171 acquires the differential charge amounts C1-C3 and the differential charge amounts C2-C4 of the latest past F frame from the database 172 (step S1201), and calculates the maximum value Cmax and the minimum value Cmin, respectively. (Step S1202).

  From the obtained maximum value Cmax and minimum value Cmin, the correction unit 171 determines the median value Cmid and variation of each of the differential charge amounts C1-C3 and the differential charge amounts C2-C4 according to the above formulas (3) and (4). Cband is calculated (step S1203).

  Then, the correction unit 171 calculates correction values (corrected differential charge amounts) (C1-C3) ′ and (C2-C4) ′ from the median value Cmid and the variation Cband according to the above equation (5). (Step S1204).

  The correcting unit 171 updates the database 172 (step S1205) and ends the process.

  Note that the difference charge amount for the most recent F frame is not stored in the database 172 immediately after the start of measurement until the (F-1) th frame. In such a case, the maximum value Cmax and the minimum value Cmin are specified among the difference charge amounts for all the stored frames, and the median value Cmid and the variation Cband are calculated. Alternatively, in such a case, the distance image generation unit 170 may be configured to calculate the distance value using the difference charge amount calculated from the acquired charge amount without performing the correction process.

  As described above, the distance image generating apparatus 100 according to the present embodiment includes the light source unit 110 including the light source that irradiates the modulated light 111 that is modulated in the target space, and the target in the target space that is irradiated from the light source unit 110. The photoelectric conversion element (photoelectric conversion unit 120) provided for each pixel that receives the light 112 including the reflected light reflected from the surface of the object 113 and converts it into charges, and a plurality of charges provided for each of the photoelectric conversion elements In synchronization with the modulation of the accumulating unit 150, the modulated light 111, the charge distributing unit 130 that distributes the charges converted by the photoelectric change element to the plurality of charge accumulating units 150, and each of the charge accumulating units 150 A distance image generation unit 170 that generates a distance image whose pixel value is a distance value using the accumulated charge amount, and the distance image generation unit 170 is accumulated in each of the charge accumulation units 150. A correction unit 171 that corrects the charge amount, and a distance value calculation unit 173 that calculates the distance value using the charge amount corrected by the correction unit 171. The correction unit 171 includes a predetermined number of frames. The charge amount is corrected using an intermediate value calculated from the maximum value and the minimum value of the charge amount stored in each charge storage unit 150.

  At this time, the correction unit 171 corrects the charge amount so that each charge amount approaches the intermediate value. Further, the correction unit 171 increases the correction amount of each charge amount as the pixel has a larger difference between the maximum value and the minimum value.

  The object of the present embodiment is to suppress variations in calculated distance values (repetition distance errors) caused by variations in the amount of accumulated charges in the charge storage units 151, 152, 153, and 154. Then, it is considered that the intermediate value calculated from the maximum value and the minimum value of the frames in the past predetermined range is closest to the true values of the differential charge amounts C1-C3 and C2-C3.

  In the present embodiment, by using this, for each frame so as to approach the intermediate value calculated from the maximum value and the minimum value of each of the differential charge amounts C1-C3 and C2-C4 in the past predetermined range of frames, Each differential charge amount C1-C3, C2-C4 of each pixel is corrected. The correction is performed using the difference charge amount of the latest predetermined number of frames for each frame. Therefore, it is possible to suppress the repetition error with a small processing amount. At this time, there is no reduction in spatial resolution or temporal resolution.

  For this reason, according to the present embodiment, it is possible to suppress variation in the calculated distance value without lowering the frame rate and the number of effective pixels, improve the repeat distance accuracy, and obtain a more accurate distance value.

<< Second Embodiment >>
Next, a second embodiment of the present invention will be described. In the first embodiment, it is assumed that the object 113 is stationary. In the present embodiment, the case where the object 113 moves is also considered.

  The distance image generation apparatus of the present embodiment basically has the same configuration as that of the first embodiment. However, in the present embodiment, it is determined whether or not the object 113 has moved, and correction is performed only when the object 113 is stationary. For this reason, the distance image generation unit 170 of the present embodiment further includes a configuration for determining whether or not the object 113 has moved. Hereinafter, the present embodiment will be described focusing on the configuration different from the first embodiment.

  As a configuration for determining whether or not the object 113 has moved, the distance image generation unit 170 according to the present embodiment uses an uncorrected charge accumulated in the charge accumulation unit 150 as shown in FIG. An intensity image generation unit 174 that generates an intensity image that is a value, and a variation determination unit 175 that determines whether or not the distance of the measurement target (object 113) varies for each pixel using the intensity image.

  Then, the correction unit 171 of the present embodiment performs the same correction as that of the first embodiment on the pixels that the variation determination unit 175 has determined that there is no variation. On the other hand, the distance value calculation unit 173 calculates the distance value using the uncharged charge amount (difference charge amount) for the pixels that the variation determination unit 175 determines as having displacement.

[Intensity image generator]
The intensity image generation unit 174 of the present embodiment reads out the charge amounts C1, C2, C3, and C4 from the charge storage units 151, 152, 153, and 154, respectively, and calculates the intensity value of each pixel according to the following equation (8). P is calculated, and an intensity image using the calculated value as a pixel value is generated.

[Fluctuation discrimination unit]
As can be seen from Equation (8), the intensity value P depends on the intensity of the reflected light (incident light 112) from the object 113. For this reason, when the object 113 approaches or moves away and the distance value fluctuates, or when another object enters the imaging range, the reflected light (incident light 112) also increases or decreases, and the intensity value P is Change.

  The change determination unit 175 uses this to determine whether or not there is a change in the distance of the object 113 in the field of view of each pixel. For example, the change determination unit 175 sets a predetermined threshold value Pth in advance, and when the intensity value P fluctuates by more than Pth from the intensity value of the previous frame, the object 113 to be imaged by the pixel. It is determined that there is movement.

[Database]
In the present embodiment, as shown in FIG. 10, the database 172 stores the difference charge amount C1-C3 and the difference charge amount C2-C4 of the latest predetermined number of frames F used by the correction unit 171 for correction. In addition, the intensity value P of the latest one frame (frame 1) is accumulated. The intensity value P is updated each time a new frame intensity value is calculated.

[Flow of distance image generation processing]
The flow of the distance image generation process of this embodiment will be described. FIG. 11 is a processing flow of distance image generation processing for generating a distance image for one frame according to the present embodiment. Also in this embodiment, the total number of pixels is P (P is an integer of 1 or more), and the processing target pixel is p (an integer satisfying 1 ≦ p ≦ P).

  When charges for one frame time Tf are accumulated in the charge accumulation units 151, 152, 153, and 154 of each pixel (step S2101), the distance image generation unit 170 performs the following processing (step S2103 to step S2103) for each pixel. S2111) is performed for all pixels (steps S2102, S2111, and S2112).

  The distance image generation unit 170 calculates the differential charge amount C1-C3 and the differential charge amount C2-C4 (step S2103). Then, the intensity image generation unit 174 calculates an intensity value using the difference charge amount C1-C3 and the difference charge amount C2-C4 (step S2104).

  The change determination unit 175 determines whether or not the pixel p has changed (step S2105).

  If it is determined in step S2105 that there is no variation, the correction unit 171 corrects the differential charge amount C1-C3 and the differential charge amount C2-C4, respectively, and corrects the differential charge amount after correction, as in the first embodiment. (C1-C3) ′ and differential charge amount (C2-C4) ′ are obtained (step S2106). Then, the distance value calculation unit 173 calculates the distance value D using the corrected differential charge amount (C1-C3) 'and differential charge amount (C2-C4)' (step S2107).

  On the other hand, when a change is detected in step S2105, that is, when it is determined that there is a change, the distance image generation unit 170 relates to the differential charge amounts C1-C3 and C2-C4 of the pixel p in the database 172. The accumulated data is deleted (step S2108), and the differential charge amount C1-C3 and the differential charge amount C2-C4 calculated in step S2103 are newly accumulated in the database 172 (step S2109).

  Further, the distance value calculation unit 173 calculates the distance value D using the differential charge amount C1-C3 and the differential charge amount C2-C4 calculated in step S2103 (step S2110).

  The distance image generation unit 170 generates a distance image using the calculated distance value as a pixel value (step S2113), and ends the process.

  It should be noted that the most recent F frame is between the start of measurement and the (F-1) th frame, and after the object 113 fluctuates and the data in the database 172 is deleted and until the (F-1) th frame. The difference charge amount of minutes is not stored in the database 172. In such a case, the maximum value Cmax and the minimum value Cmin are specified among the difference charge amounts for all the latest frames, and the median value Cmid and the variation Cband are calculated. Alternatively, in such a case, the distance image generation unit 170 may be configured to calculate the distance value using the difference charge amount calculated from the acquired charge amount without performing the correction process.

  In addition, either the processing of steps S2108 and 2109 or the processing of step S2110 may be performed first.

  Note that the correction processing by the correction unit 171 in step S2106 is the same as in the first embodiment.

  Next, the flow of fluctuation detection processing by the fluctuation determination unit 175 in step S2105 will be described with reference to FIG.

  The variation determining unit 175 extracts the intensity value (comparative intensity value) of the previous frame of the processing target pixel p from the database 172 (step S2201). Then, a difference ΔP from the calculated intensity value P is calculated (step S2202). The calculated difference ΔP is compared with a predetermined threshold value Pth (step S2203). If the difference ΔP is equal to or larger than the threshold value, it is assumed that the variation is detected (step S2204). It is assumed that no change has been detected (step S2205).

  Finally, the variation determination unit 175 updates the intensity value (comparison intensity value) in the database 172 to the intensity value calculated in step S2104 (step S2206), and ends the process.

  Note that the update process of the intensity value may be performed any time after step S2203.

  As described above, the distance image generating apparatus 100 according to this embodiment has the same configuration as that of the first embodiment, and the pixel value is an intensity value using the uncorrected charges accumulated in the charge accumulation unit. An intensity image generation unit 174 that generates a certain intensity image; and a variation determination unit 175 that determines whether or not the measurement target varies for each pixel using the intensity image, and the correction unit 171 includes the variation determination The respective charge amounts are corrected for pixels that the unit 175 has determined to have no variation, and the distance value calculation unit 173 uses the uncorrected charge amount for the pixels that the variation determination unit 175 has determined to have variation. To calculate the distance value.

  As described above, also in the present embodiment, as in the first embodiment, it is possible to improve the repeat distance accuracy with a small amount of processing, without a decrease in spatial resolution and a decrease in temporal resolution.

  Furthermore, according to the present embodiment, using the difference charge amount C1-C3 and the difference charge amount C2-C4 used for the distance image generation, an intensity image is calculated prior to the distance image generation, and whether or not the object 113 has moved. Is determined. Then, only when there is no movement of the object 113 in the field of view of the pixel, the differential charge amount C1-C3 and the differential charge amount C2-C4 used for the distance image generation are corrected by the same method as in the first embodiment. To do. On the other hand, when there is a movement, a distance image is generated using the acquired differential charge amount C1-C3 and differential charge amount C2-C4 as they are.

  Therefore, even when the object 113 may move, the differential charge amount C1-C3 and the differential charge amount C2-C4 can be corrected only when the object 113 is stationary. Accordingly, it is possible to eliminate the correction of the difference in the accumulation of the charge amount due to the movement, and it is possible to accurately correct the variation in the accumulation of the charge amount that causes the repeated error regardless of the movement of the object 113. .

  Therefore, according to the present embodiment, it is possible to obtain a highly accurate distance image that eliminates the influence of repeated errors regardless of the environment of the subject to be photographed.

  For example, in a surveillance camera that basically performs fixed-point observation, there is little change in the shooting range over time. For this reason, when there is a change such as the passage of a person or an object, it is difficult for the surveillance camera using the conventional distance image generation device to determine whether the change in the image is due to these or the background varies. However, according to the distance image generation device of each of the embodiments described above, the distance accuracy of the background image in a steady state can be improved and an image with less repetition error can be obtained. .

  In each of the above embodiments, the values of the differential charge amount C1-C3 and the differential charge amount C2-C4 are corrected, but the correction target is not limited to this. You may correct | amend each electric charge amount C1, C2, C3, and C4, respectively.

  In this case, the database 172 stores the charge amounts C1, C2, C3, and C4 of each pixel in the past frame for each of the charge storage units 151, 152, 153, and 154.

  Furthermore, in each of the above-described embodiments, the case where a plurality of (four) charge storage units 150 are provided for each photoelectric conversion unit (pixel) 120 has been described as an example, but the number of charge storage units 150 is not limited thereto. It is not limited. What is necessary is just three or more.

  Further, the number of charge storage units 150 may be one. In this case, for example, when the distance value is calculated using the four charge amounts C1, C2, C3, and C4, the distance image generation unit 170 divides one cycle Tq of the modulation signal into four equal intervals at Tq / 4 intervals. The charge stored in the charge storage unit 150 is read and read as charge amounts C1, C2, C3, and C4 at intervals of Tq / 4.

  DESCRIPTION OF SYMBOLS 100 ... Distance image generation apparatus, 110 ... Light source part, 111 ... Modulated light, 112 ... Incident light, 113 ... Target part, 114 ... Modulation signal, 115 ... Gate signal, 120 ... Photoelectric conversion part, 130 ... Charge distribution part, DESCRIPTION OF SYMBOLS 140 ... Control part, 150 ... Charge storage part, 151 ... Charge storage part, 152 ... Charge storage part, 153 ... Charge storage part, 154 ... Charge storage part, 170 ... Distance image generation part, 171 ... Correction part, 172 ... Database , 173 ... Distance value calculation unit, 174 ... Intensity image generation unit, 175 ... Variation determination unit

Claims (10)

A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
A charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light;
A distance image generation unit that generates a distance image whose pixel value is a distance value using the amount of charge stored in each of the charge storage units,
The distance image generation unit
A correction unit that corrects the amount of charge accumulated in each of the charge accumulation units;
A distance value calculation unit that calculates the distance value using the charge amount corrected by the correction unit,
The correction unit corrects each charge amount so as to approach an intermediate value calculated based on the maximum value and the minimum value of the charge amount accumulated in each charge accumulation unit in a predetermined number of frames. A distance image generation device characterized by the above. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
A charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light;
A distance image generation unit that generates a distance image whose pixel value is a distance value using the amount of charge stored in each of the charge storage units,
The distance image generation unit
A correction unit that corrects the amount of charge accumulated in each of the charge accumulation units;
A distance value calculation unit that calculates the distance value using the charge amount corrected by the correction unit,
The correction unit corrects each charge amount using an intermediate value calculated based on a maximum value and a minimum value of the charge amount accumulated in each charge accumulation unit of a predetermined number of frames; and The larger the difference between the maximum value and the minimum value, the larger the correction amount of each charge amount.
A distance image generation device characterized by the above. The distance image generating device according to claim 1 or 2 ,
An intensity image generation unit that generates an intensity image in which a pixel value is an intensity value using the charge amount before correction accumulated in the charge accumulation unit;
Using the intensity image, further comprising a fluctuation determination unit that determines the presence or absence of fluctuation of the object for each pixel,
The correction unit corrects each charge amount for the pixel that the variation determination unit determines to have no variation,
The distance value calculation unit is configured to calculate the distance value using the uncorrected charge amount for a pixel that is determined to have variation by the variation determination unit. The distance image generating device according to any one of claims 1 to 3 ,
Four charge storage units are provided for each photoelectric conversion element,
The charge distribution unit distributes the charges in the predetermined order to the four charge storage units,
The correction unit is configured to calculate a difference between charge amounts accumulated in the charge accumulation unit that distributes charges to the first and third, and a difference between charge amounts accumulated in the charge accumulation unit that distributes charges to the second and fourth. A distance image generation device characterized by correcting each of them. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
A distance image generation method in a distance image generation apparatus comprising: a charge distribution unit that distributes charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with modulation of the modulated light,
An intermediate value is calculated based on the maximum value and the minimum value of the amount of charge accumulated in each of the charge accumulation units during a predetermined frame,
Correct the amount of charge in the charge storage unit so as to approach the intermediate value,
A distance image generation method characterized by generating a distance image whose pixel value is a distance value using the corrected charge amount. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
A distance image generation method in a distance image generation apparatus comprising: a charge distribution unit that distributes charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with modulation of the modulated light,
An intermediate value is calculated based on the maximum value and the minimum value of the amount of charge accumulated in each of the charge accumulation units during a predetermined frame,
Using the intermediate value, the charge amount of the charge storage unit is corrected so that the correction amount increases as the difference between the maximum value and the minimum value increases .
A distance image generation method characterized by generating a distance image whose pixel value is a distance value using the corrected charge amount. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
In a computer of a distance image generation device comprising: a charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light,
A procedure for calculating an intermediate value based on a maximum value and a minimum value of the amount of charge accumulated in each of the charge accumulation units during a predetermined frame;
A procedure for correcting the charge amount of the charge storage section so as to approach the intermediate value;
A program for executing a procedure for generating a distance image in which a pixel value is a distance value using a corrected charge amount. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
A plurality of charge storage units provided for each photoelectric conversion element;
In a computer of a distance image generation device comprising: a charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light,
A procedure for calculating an intermediate value based on a maximum value and a minimum value of the amount of charge accumulated in each of the charge accumulation units during a predetermined frame;
Using the intermediate value, a procedure for correcting the charge amount of the charge storage unit so that the correction amount increases as the difference between the maximum value and the minimum value increases .
A program for executing a procedure for generating a distance image in which a pixel value is a distance value using a corrected charge amount. A light source unit including a light source that emits modulated light that is modulated into a target space;
A photoelectric conversion element provided for each pixel that receives light including reflected light that is irradiated from the light source unit and reflected by the surface of the object in the target space, and converts the light into charges;
  A plurality of charge storage units provided for each photoelectric conversion element;
A charge distribution unit that distributes the charges converted by the photoelectric conversion elements to the plurality of charge storage units in synchronization with the modulation of the modulated light;
A distance image generation unit that generates a distance image whose pixel value is a distance value using the amount of charge stored in each of the charge storage units,
The distance image generation unit
A correction unit that corrects the amount of charge accumulated in each of the charge accumulation units;
A distance value calculation unit that calculates the distance value using the charge amount corrected by the correction unit,
Four charge storage units are provided for each photoelectric conversion element,
The charge distribution unit distributes the charges in the predetermined order to the four charge storage units,
The correction unit stores, for each of the charge storage units, the difference in the amount of charge stored in the charge storage unit that distributes the charge first and third, and the charge storage unit that distributes the charge second and fourth Identifying the maximum and minimum values of the difference in the amount of charge to be performed, and correcting each differential charge amount using an intermediate value calculated based on the maximum and minimum values
A distance image generation device characterized by the above. The distance image generating device according to claim 9, wherein
The correction unit corrects the charge amount so that each charge amount approaches the intermediate value.
A distance image generation device characterized by the above.

Images