JP7758938B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

In recent years, the event camera (event-based camera) disclosed in Patent Document 1 below has become known as a technology for generating images of a captured object faster. The event camera is a brightness value differential output camera, developed inspired by the retinal structure of living organisms. It is configured to sense changes in brightness for each pixel and output the coordinates, time, and polarity of the brightness change. This configuration allows the event camera to avoid outputting pixel information without brightness changes, i.e., redundant data, as in conventional cameras. This not only enables the event camera to generate images of the captured object faster, but also realizes cameras with advantages such as high temporal resolution, high dynamic range, reduced data consumption, and reduced power consumption. In particular, Patent Document 1 below reconstructs an image of an information code by estimating the brightness of each cell that makes up the information code based on events output in response to pixels where brightness changes occur when the event camera and the information code are moved relative to each other.

Japanese Patent Application Laid-Open No. 2021-082273

For example, when an event camera captures a QR code (registered trademark) passing by at close range, the two-dimensional black-and-white changes result in the instantaneous output of a large amount of event data due to the event camera's high temporal resolution. This creates a problem when inexpensive event cameras that cannot instantaneously output large amounts of event data reach their limit and become saturated with the event data they should output, resulting in some data being lost and not output. This problem is not limited to the above-mentioned case of capturing QR codes, but can also occur, for example, when capturing images in connection with SLAM (Simultaneous Localization and Mapping: simultaneous self-localization and environmental map creation).

The present invention was made to solve the above-mentioned problems, and its purpose is to provide an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, reduced data consumption, and reduced power consumption, while suppressing output saturation of event data.

In order to achieve the above object, the imaging device (10) according to claim 1 of the claims comprises:
an imaging element (13a) that generates event data including two-dimensional point data that corresponds to a pixel where a luminance change has occurred and identifies the position of the pixel;
an image generating unit (11) that generates a captured image of a predetermined imaging area (S) using event data output from the imaging element;
an output limiting unit (11) capable of limiting the output of event data generated by the imaging element by outputting event data of pixels that have undergone a luminance change only for limited areas (Sa to Sf) that limit the predetermined imaging area to a narrow area, and not outputting event data of pixels in areas other than the limited areas of the predetermined imaging area, regardless of whether there is a luminance change ;
Equipped with
the output limiting unit sequentially switches the limited area so as to cover the predetermined imaging area;
The image generation unit is characterized in that, when the output restriction unit restricts the output of event data, it generates an image of the specified imaging area by aggregating the event data that is output each time the limited area is switched.
The symbols in parentheses above indicate the correspondence with the specific means described in the embodiments to be described later.

In the invention of claim 1, the output limiting unit limits the predetermined imaging area to a narrow area and outputs event data of pixels that have experienced a luminance change , and does not output event data of pixels in the predetermined imaging area excluding the limited area, regardless of whether there is a luminance change, thereby limiting the output of event data generated by the imaging element, and sequentially switches the limited area so as to cover the predetermined imaging area. When the output limiting unit limits the output of event data, the image generating unit generates an image of the predetermined imaging area by aggregating the event data output each time the limited area is switched.

As a result, even if a QR code crosses the specified imaging area at close range, for example, event data is output for the part of the QR code included in the limited area, and no event data is output for the remaining part of the QR code that is not included in the limited area, thereby preventing saturation of event data output. Furthermore, by aggregating the event data output each time the limited area is switched, it is possible to generate a captured image of the specified imaging area, thereby preventing saturation of event data output while realizing an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, low data consumption, and low power consumption.

In the invention of claim 2, the output limiting unit temporarily disables the functions of pixels that do not correspond to the limited area, thereby outputting event data for pixels that have experienced a change in brightness only in the limited area, allowing for accurate and fast switching of the limited area. In particular, the range of each limited area within a specified imaging area can be easily adjusted, such as by dividing it into four parts or by spacing it out.

In the invention of claim 3, the output restriction unit has a shielding unit whose surface facing the image sensor is formed in a single color and which covers pixels different from those corresponding to the limited area. By moving the shielding unit in response to sequential switching of the limited area, event data for pixels in the limited area that have experienced a change in brightness can be output. In this way, the limited area can be switched in response to movement such as rotation or sliding of the shielding unit.

In the invention of claim 4, the output limiting unit has an infrared-transmitting section that covers the light-receiving side of the image sensor, and an infrared irradiating section that irradiates infrared rays from the image sensor side onto a portion of the surface of the infrared-transmitting section that corresponds to a predetermined image area. By moving the irradiation range of infrared rays irradiated from the infrared irradiating section in response to sequential switching of the limited area, event data for pixels that have experienced a change in luminance in only the limited area is output. In this way, the limited area can be switched in response to the movement of the infrared irradiation range.

In the invention of claim 5, when the output restriction unit restricts the output of event data, the image generation unit generates a captured image of the specified imaging area by aggregating the event data output each time the limited area is switched, taking into account the movement speed of the imaging target estimated by the estimation unit. As a result, even if the imaging target, such as a QR code, is moving at high speed and the position of the imaging target in the specified imaging area shifts each time the limited area is switched, the event data can be aggregated to correct the positional shift, and a captured image of the specified imaging area can be generated with high accuracy.

In the invention of claim 6, the output limiting unit limits the output of event data when the amount of event data output from the imaging element per unit time exceeds a predetermined amount. This prevents the output of event data from being limited even when only a small amount of event data is output, thereby preventing the output of unnecessary event data from being limited.

1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment. FIG. 3 is an explanatory diagram illustrating the relationship between an imaging area and each limited area in the first embodiment. FIG. 10 is an explanatory diagram illustrating a state in which a QR code enters an imaging area. Figure 4(A) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(A), Figure 4(B) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(B), Figure 4(C) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(C), and Figure 4(D) is an explanatory diagram illustrating a portion of the QR code in Figure 3 that is the target of event output in the limited area shown in Figure 2(D). 10A and 10B are explanatory diagrams illustrating a method of switching a limited area relative to an imaging area in the second embodiment. 13A and 13B are explanatory diagrams illustrating a method of switching a limited area relative to an imaging area in the third embodiment.

[First embodiment]
Hereinafter, a first embodiment of an imaging device according to the present invention will be described with reference to the drawings.
The imaging device 10 according to this embodiment functions as a so-called event camera. The imaging device 10 outputs event data that includes two-dimensional point data that identifies the position of a pixel that has experienced a brightness change, as well as time and the polarity of the brightness change, and generates image data of the imaging target by plotting the two-dimensional point data of multiple event data output within a certain period of time as points on a predetermined plane.

As shown in FIG. 1, the imaging device 10 includes a control unit 11 consisting of a CPU or the like and a storage unit 12 consisting of semiconductor memory or the like, as well as an imaging unit 13 and a display unit 14 controlled by the control unit 11, an operation unit 15 that outputs operation signals to the control unit 11 in response to input operations, and a communication unit 16 for communicating with external devices, etc.

The imaging unit 13 is configured to include an imaging element 13a and a light-receiving lens, and the imaging element 13a is configured to output to the control unit 11 event data including two-dimensional point data that identifies the position of pixels that have experienced a change in luminance when light is received through the light-receiving lens. In other words, the imaging element 13a outputs to the control unit 11 event data (two-dimensional point data, time, polarity of luminance change) corresponding to pixels that have experienced a change in luminance, and functions not to output data for pixels that have not experienced a change in luminance.

In particular, in this embodiment, the image sensor 13a of the imaging unit 13 is configured to be switchable on a pixel-by-pixel basis in response to output restriction processing by the control unit 11 between pixels that output event data when there is a change in luminance (hereinafter also referred to as output-enabled pixels) and pixels that do not output event data regardless of whether there is a change in luminance by temporarily disabling the pixel function (hereinafter also referred to as output-restricted pixels).

For example, by designating a predetermined imaging area S using all pixels of the imaging element 13a such that only pixels corresponding to the limited area Sa shown in FIG. 2(A) are outputtable and the other pixels are output-restricted, the amount of event data output can be reduced compared to when all pixels are outputtable. The amount of event data output can also be reduced in a similar manner when only pixels corresponding to the limited area Sb shown in FIG. 2(B) are outputtable and the other pixels are output-restricted, when only pixels corresponding to the limited area Sc shown in FIG. 2(C) are outputtable and the other pixels are output-restricted, or when only pixels corresponding to the limited area Sd shown in FIG. 2(D) are outputtable and the other pixels are output-restricted.

In this embodiment, the combined area of each limited area Sa-Sd is set to match the imaging area S, and under specified imaging conditions, each limited area Sa-Sd is switched sequentially to cover the imaging area S. Under such imaging conditions, the amount of event data output is reduced as described above, and the event data output each time each limited area Sa-Sd is switched is aggregated to generate an image of the imaging area S. Note that the control unit 11 that performs the output restriction process described above can serve as an example of an "output restriction unit" that can restrict the output of event data generated by the image sensor 13a by outputting event data for pixels that have experienced luminance changes only for limited areas (Sa-Sd) that limit the imaging area S to a small area. Furthermore, each limited area Sa-Sd may be set so that the combined area of each limited area Sa-Sd includes the imaging area S.

For example, under the above-mentioned specified imaging conditions, when QR code C is held over the imaging device 10, and the QR code C enters the imaging area S as shown in FIG. 3, first, event data for generating the image shown in FIG. 4(A) when switching to limited area Sa is output from the imaging element 13a to the control unit 11. Next, event data for generating the image shown in FIG. 4(B) when switching to limited area Sb is output from the imaging element 13a to the control unit 11, and then event data for generating the image shown in FIG. 4(C) when switching to limited area Sc is output from the imaging element 13a to the control unit 11. Finally, event data for generating the image shown in FIG. 4(D) when switching to limited area Sd is output from the imaging element 13a to the control unit 11.

The control unit 11 then performs an image generation process that generates a captured image of the specified imaging area S by aggregating the event data output each time the limited area is switched. As a result, the event data obtained when switching to limited area Sa, the event data obtained when switching to limited area Sb, the event data obtained when switching to limited area Sc, and the event data obtained when switching to limited area Sd are aggregated to generate a captured image corresponding to QR code C as shown in FIG. 3. The control unit 11 that performs the image generation process can be an example of an "image generation unit" that generates a captured image of the specified imaging area S by aggregating the event data output each time the limited area is switched.

As described above, in the imaging device 10 according to this embodiment, the output restriction process by the control unit 11 restricts the output of event data generated by the image sensor 13a by outputting event data of pixels (output-enabled pixels) that have experienced a change in brightness only in a limited area (Sa-Sb) that limits the specified imaging area S to a small area, and sequentially switches the limited area to cover the specified imaging area S. When the output restriction process restricts the output of event data, the image generation process by the control unit 11 generates a captured image of the specified imaging area S by aggregating the event data output each time the limited area is switched.

As a result, even if, for example, a QR code C enters the specified imaging area S by crossing it at close range, event data is output for the portion of the QR code C that is included in the limited area, and event data is not output for the remainder of the QR code C that is not included in the limited area, thereby preventing saturation of the event data output. Furthermore, even if, for example, the surrounding environment enters the specified imaging area S, event data is output for the portion of the surrounding environment that is included in the limited area, and event data is not output for the remainder of the surrounding environment that is not included in the limited area, thereby preventing saturation of the event data output. Furthermore, by aggregating the event data output each time the limited area is switched, it is possible to generate a captured image of the specified imaging area S, thereby preventing saturation of the event data output, and realizing an imaging device that has the advantages of an event camera, such as high temporal resolution, high dynamic range, low data consumption, and low power consumption.

In particular, the output restriction process by the control unit 11 temporarily disables the functions of pixels that do not correspond to the limited area (output-restricted pixels), thereby outputting event data for pixels that have experienced a change in brightness only in the limited area (output-enabled pixels), allowing for accurate and fast switching of limited areas. Furthermore, the range of each limited area within the specified imaging area S can be easily adjusted, such as by dividing it into four parts or making it sparse.

Second Embodiment
Next, an imaging device according to a second embodiment of the present invention will be described with reference to the drawings.
The second embodiment differs from the first embodiment mainly in that the limited area is physically switched. Therefore, the same components as those in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.

In the imaging device 10 according to this embodiment, as illustrated in FIG. 5, a shielding portion 20 is rotatably arranged on the light-receiving side of the imaging unit 13 (image sensor 13a). The surface of this shielding portion 20 facing the image sensor is a single color (e.g., gray, which is an intermediate color between white and black), has a fan-shaped cutout 21 formed therein, and is arranged so that its center of rotation coincides with the imaging center of the image sensor 13a. When viewed from the image subject side, the shielding portion 20 is formed so that it covers the image sensor 13a except for the cutout 21 when rotated, and the central angle of the cutout 21 is set to an angle (e.g., 90°) according to the switching frequency.

The output restriction process performed by the control unit 11 drives and controls a motor or the like that rotates the shielding unit 20 to switch between the range of the image sensor 13a covered by the shielding unit 20 and the range of the image sensor 13a exposed through the cutout 21. The pixels of the image sensor 13a covered by the shielding unit 20 become the output-restricted pixels because they receive light from the single-color surface of the shielding unit 20 and do not experience a change in brightness, while the pixels of the image sensor 13a exposed through the cutout 21 become the output-enabled pixels.

For this reason, the limited area can be switched by rotating the shielding unit 20. The control unit 11 then performs an image generation process in which, for example, each time the shielding unit 20 is rotated by the central angle of the cutout 21, the limited area is considered to have been switched, and the event data is aggregated to generate a captured image of the specified imaging area S. Note that in Figure 5, the limited area switched by rotating the shielding unit 20 is indicated by the symbol Se. The control unit 11 and shielding unit 20 that perform the output restriction process can be considered an example of an "output restriction unit."

Note that the limited area may not necessarily be physically switched by rotating the shielding portion 20 in which the notch 21 is formed. For example, the limited area may be physically switched by sliding the shielding portion in a predetermined direction (e.g., up and down).

[Third embodiment]
Next, an imaging device according to a third embodiment of the present invention will be described with reference to the drawings.
The third embodiment differs from the first embodiment mainly in that the limited area is switched using infrared rays. Therefore, the same components as those in the first embodiment are denoted by the same reference numerals and their description will be omitted.

As shown in FIG. 6, the imaging device 10 according to this embodiment is provided with an infrared-transmitting section 31 that covers the light-receiving side of the imaging section 13 (image sensor 13a), and an infrared irradiating section 32 that irradiates infrared light from the image sensor side onto a portion of the surface of the infrared-transmitting section 31 that corresponds to a predetermined imaging area. The infrared-transmitting section 31 is, for example, an infrared-transmitting filter that functions to transmit infrared light while blocking visible light.

As a result, some pixels of the image sensor 13a that receive infrared light reflected from the image subject or the like and transmitted through the infrared-transmitting section 31 become output-enabled pixels. On the other hand, light reflected from the image subject or the like that is not irradiated with infrared light cannot pass through the infrared-transmitting section 31, so the remaining pixels of the image sensor 13a whose light-receiving range is an area that is not irradiated with infrared light become output-limited pixels.

The infrared irradiation unit 32 is configured so that the irradiation range (irradiation direction) of its infrared rays can be controlled by the control unit 11. Therefore, in the output limiting process performed by the control unit 11, the limited area can be switched by moving the infrared irradiation range of the infrared irradiation unit 32. Then, each time the infrared irradiation range of the infrared irradiation unit 32 changes, the control unit 11 performs image generation process to generate a captured image of the specified imaging area S by aggregating event data as if the limited area had been switched. Note that in Figure 6, the limited area switched by the irradiation of infrared rays from the infrared irradiation unit 32 is indicated by the symbol Sf. Furthermore, the control unit 11, infrared transmission unit 31, and infrared irradiation unit 32 that perform the output limiting process can correspond to an example of an "output limiting unit."

The infrared irradiation unit 32 is not limited to being configured to change the irradiation range of the infrared transmission unit 31 by changing the irradiation direction. For example, it may be configured to have a corresponding infrared light for each limited area, and to change the irradiation range of the infrared transmission unit 31 by switching the infrared light that is turned on.

[Fourth embodiment]
Next, an imaging device according to a fourth embodiment of the present invention will be described.
The fourth embodiment differs from the first embodiment in that event data is aggregated taking into account the estimated moving speed of the imaging target. Therefore, components that are substantially the same as those in the first embodiment are denoted by the same reference numerals, and descriptions thereof will be omitted.

If the relative movement speed of the imaging target relative to the imaging device 10 is relatively fast or the limited area switching speed is relatively slow, the imaging target may move while the limited area is being switched, causing a positional shift, making it impossible to generate an accurate image of the imaging target.

For this reason, in this embodiment, the movement speed of the imaging subject is estimated, and the event data output each time the limited area is switched is aggregated taking into account the estimated movement speed of the imaging subject.

Specifically, first, the control unit 11 performs a movement speed estimation process, generating multiple images of the object to be imaged using event data in the image capture area or a single limited area, and then detecting the amount of positional deviation of the object between each image, thereby estimating the movement speed of the object. Note that the control unit 11 that performs the movement speed estimation process can be considered an example of an "estimation unit" that estimates the movement speed of the object to be imaged.

Next, in the output restriction process performed by the control unit 11, the limited area is switched as described above, and event data is acquired for each limited area. Then, in the image generation process performed by the control unit 11, the position of the acquired event data is corrected using the movement direction of the imaging target calculated from the movement speed estimated as described above and the movement distance at the limited area switching interval, and the position-corrected event data is aggregated to generate an image of the imaging target.

For example, when four limited areas Sa to Sd are used as in the first embodiment, first, the event data output when limited area Sa is switched is aggregated with the event data output when limited area Sb is switched, with the position corrected using the movement distance, etc., of the event data output. Next, the event data aggregated for the two limited areas as described above is further aggregated with the position corrected using the movement distance, etc., and the event data output when limited area Sc is switched. Then, the event data aggregated for the three limited areas as described above is further aggregated with the position corrected using the movement distance, etc., and the event data output when limited area Sd is switched. In this way, a captured image of the specified imaging area S is generated based on the event data aggregated so that position misalignment is corrected.

As described above, in the imaging device 10 according to this embodiment, when the output of event data is restricted by the output restriction process, the image generation process performed by the control unit 11 generates an image of the specified imaging area S by aggregating the event data output each time the limited area is switched, taking into account the estimated movement speed of the imaging subject.

As a result, even if the position of the imaging target in the specified imaging area S shifts each time the limited area is switched due to the imaging target, such as a QR code, moving at high speed, event data can be aggregated to correct the positional shift, and an image of the specified imaging area S can be generated with high accuracy.

Note that the characteristic configuration of this embodiment, which aggregates event data taking into account the estimated movement speed of the imaging subject, can also be applied to other embodiments, etc.

The present invention is not limited to the above-described embodiments, and may be embodied as follows, for example.
(1) In the output limiting process performed by the control unit 11, the output of event data may be limited when the amount of event data output from the image sensor 13a per unit time is equal to or greater than a predetermined amount as the predetermined imaging condition, and may not be limited when the amount of event data output is less than the predetermined amount. This prevents the output of event data from being limited even when only a small amount of event data is output, thereby preventing unnecessary output restrictions from occurring. Furthermore, the output of event data may be limited when, for example, a predetermined operation is performed on the operation unit 15 or a predetermined instruction is received from the outside via the communication unit 16.

(2) The image generation process performed by the control unit 11 is not limited to generating an image of the specified imaging area S by aggregating all event data output each time the limited area is switched, but may also generate, for example, an image of a portion of the imaging target or an image of a portion of the specified imaging area S.

10...imaging device 11...control unit (output limiting unit, image generating unit, estimation unit)
13: Imaging unit 13a: Imaging element S: Imaging area Sa to Sf: Limited area

Claims (6)

an image sensor that generates event data including two-dimensional point data that identifies the position of a pixel that has experienced a luminance change;
an image generation unit that generates a captured image of a predetermined imaging area using the event data output from the imaging element;
an output limiting unit that limits the output of event data generated by the image sensor by outputting event data of pixels that have undergone a luminance change only in a limited area that limits the predetermined image capture area to a narrow area, and not outputting event data of pixels that have undergone a luminance change in an area other than the limited area of the predetermined image capture area, regardless of whether or not there is a luminance change;
Equipped with
the output limiting unit sequentially switches the limited area so as to cover the predetermined imaging area;
The imaging device is characterized in that, when the output restriction unit restricts the output of event data, the image generation unit generates an image of the specified imaging area by aggregating the event data that is output each time the limited area is switched. The imaging device described in claim 1, characterized in that the output restriction unit temporarily disables the functions of pixels that do not correspond to the limited area, thereby outputting event data for pixels that have experienced a luminance change only in the limited area. The imaging device described in claim 1, characterized in that the output restriction unit has a shielding section whose surface facing the imaging element is formed in a single color and covers pixels different from those corresponding to the limited area, and by moving the shielding section in response to sequential switching of the limited area, event data of pixels that have experienced a change in brightness is output only in the limited area. The imaging device described in claim 1, characterized in that the output limiting unit includes an infrared-transmitting section that covers the light-receiving side of the imaging element, and an infrared irradiating section that irradiates infrared rays from the imaging element side onto a portion of the surface of the infrared-transmitting section that corresponds to the specified imaging area, and by moving the irradiation range of infrared rays irradiated from the infrared irradiating section in accordance with the sequential switching of the limited area, event data of pixels that have experienced a change in brightness is output only in the limited area. an estimation unit that estimates a moving speed of an imaging target;
The imaging device according to any one of claims 1 to 4, characterized in that, when the output limiting unit limits the output of event data, the image generating unit generates an image of the specified imaging area by aggregating the event data output each time the limited area is switched, taking into account the movement speed of the imaging target estimated by the estimation unit. The imaging device described in any one of claims 1 to 5, characterized in that the output limiting unit limits the output of the event data when the amount of event data output from the imaging element per unit time exceeds a predetermined amount.