JP7770596B2 - Imaging device - Google Patents

Description

The present disclosure relates to an imaging device.

Generally, image processing techniques performed within imaging devices have been proposed that effectively correct images affected by optical aberrations generated by the optical system (also referred to as the imaging optical system) within the imaging device (see, for example, Patent Document 1). The imaging device described in Patent Document 1 accurately corrects images whose image quality has been degraded by the effects of the optical system based on a first image affected by the aberrations of the optical system, a second image from which the effects of the aberrations have been removed, positional information for each pixel obtained from information on these images, and a distortion aberration correction table.

International Publication No. 2017/158690

However, with conventional technology, imaging performance is highest at the center of the light-receiving surface of the imaging device, and decreases the further away from the center of the light-receiving surface. As a result, the peripheral area located outside the center of the light-receiving surface is significantly affected by optical system aberrations. Furthermore, the peripheral area receives information outside the image range as stray light. In other words, while such conventional technology can effectively correct pixels near the center of the light-receiving surface, its correction performance is limited for pixels near the outer edges of the light-receiving surface. This issue becomes more pronounced when using optical systems with simple configurations, such as single lenses or spherical lenses.

The purpose of this disclosure is to solve the above-mentioned conventional problems and to provide an imaging device that can remove noise from the entire captured image with high precision, even when using an optical system with a simple configuration.

An imaging device according to one aspect of the present disclosure includes:
an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount,
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The feature extraction unit
a first image storage unit in which the ideal image is stored;
a second image storage unit in which the noise image is stored;
a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image;
and
The first image storage unit stores an image having a wider range than the range captured by the imaging optical system .
An imaging device according to another aspect of the present disclosure includes:
an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface ,
The feature extraction unit
a first image storage unit in which the ideal image is stored;
a second image storage unit in which the noise image is stored;
a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image;
and
The second image storage unit stores an image having a wider range than the range captured by the imaging optical system.
Imaging device.
An imaging device according to yet another aspect of the present disclosure includes:
an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface.
Imaging device.
An imaging device according to yet another aspect of the present disclosure includes:
an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or a portion of the light receiving surface other than the outermost edge portion.
Imaging device.

This disclosure makes it possible to provide an imaging device that can remove noise from the entire captured image with high accuracy, even when using an optical system with a simple configuration.

FIG. 1 is a block diagram illustrating a schematic configuration of an imaging device. FIG. 2 is a block diagram illustrating an example of a feature extraction unit. 10A and 10B are block diagrams illustrating the configuration of an image acquisition unit. FIG. 2 illustrates an example of a hardware configuration of a main control unit. FIG. 10 is a diagram illustrating another example of the hardware configuration of the main control unit. FIG. 1 is a diagram schematically illustrating an example of an imaging optical system.

The imaging device 200 according to an embodiment of the present disclosure will be described below with reference to the drawings. The embodiment described below is merely an example, and various modifications are possible.

FIG. 1 is a block diagram showing a schematic configuration of an imaging device 200 according to the present embodiment.
The imaging device 200 includes an imaging optical system 100 , a feature extraction unit 31 , an image correction unit 41 , and an image display unit 51 .

The imaging optical system 100 captures an image. The imaging optical system 100 has an imaging element 1 with a light receiving surface 10. The image captured by the imaging optical system 100 is also referred to as the "captured image."

The feature extraction unit 31 extracts features from the noise image 21, which contains noise due to the imaging optical system 100, and the ideal image 11, in which the noise has been removed from the noise image 21. The feature is the difference between the ideal image 11 and the noise image 21. In other words, the feature extraction unit 31 extracts the noise (also referred to as the noise component) from the noise image 21.

FIG. 2 is a block diagram showing an example of the feature extraction unit 31. As shown in FIG.
In the example shown in FIG. 2 , the feature extraction unit 31 includes a first image storage unit 12 , a second image storage unit 22 , and a machine learning processing unit 32 .

The machine learning processing unit 32 performs machine learning using a machine learning model such as a genetic algorithm or a neural network. By using a machine learning model such as a neural network, it is possible to correct various aberrations, such as image distortion or blur correction, all at once.

Neural networks extract features by convolution processing (also called convolution operations) of two-dimensional information using, for example, a Convolutional Neural Network (CNN). The data input to these machine learning models is two-dimensional information of the ideal image 11 and noise image 21, or multidimensional information including RGB information.

The first image memory unit 12 can store images with a wider range than the range captured by the imaging optical system 100. Similarly, the second image memory unit 22 can store images with a wider range than the range captured by the imaging optical system 100. For example, if the imaging device 100 can capture an image in a range of ±30 degrees, a pair of a wide-angle ideal image 11 and a wide-angle noise image 21 can be obtained by performing calculations using ray tracing of ±35 degrees. If pixel loss occurs due to noise or distortion caused by stray light of ±30 degrees or more, the noise or pixel loss is estimated and corrected using the wide-angle pair of images.

Instead of using only one image, multiple images may be used. When multiple images are used, the ideal image 11 is stored in the first image storage unit 12, and the noise image 21 is stored in the second image storage unit 22. In this case, a tag is attached to each ideal image 11 so that each ideal image 11 is associated with the noise image 21.

When a neural network is used as the machine learning model, the machine learning processing unit 32 extracts features by performing convolution processing on the ideal image 11 and the noise image 21.

In the example shown in FIG. 2, when the ideal image 11 and the noise image 21 are input to the machine learning processing unit 32, the machine learning processing unit 32 can estimate the noise components contained in the noise image 21. The machine learning processing unit 32 repeatedly adjusts the weights of the machine learning model so that the value of each pixel of the noise image 21 approaches that of the ideal image 11. In this way, the machine learning model learns to restore the noise image 21 to the ideal image 11.

The ideal image 11 is an image from which noise due to the imaging optical system 100 has been removed. The ideal image 11 is, for example, a real image captured by an imaging device having an imaging optical system with higher precision than the imaging optical system 100 of the imaging device 200. The ideal image 11 may also be an image generated by computer graphics (CG). The ideal image 11 may be stored in advance in a memory within the imaging device 200 (for example, the first image storage unit 12).

The noise image 21 is an image that includes noise caused by the imaging optical system 100. That is, the noise image 21 is an image (i.e., a real-life image) output from the imaging optical system 100. Noise refers to, for example, image distortion or image blur caused by optical aberrations of the imaging optical system 100, or stray light generated by the optical components or housing of the imaging optical system 100.

The noise image 21 may be an image generated by data processing such as ray tracing. In this case, the ray tracing can be calculated using, for example, a point spread function (PSF) or an optical transfer function (OTF).

The image correction unit 41 acquires the features from the feature extraction unit 31. The image correction unit 41 corrects the captured image captured by the imaging optical system 100 using the features. The captured image captured by the imaging optical system 100 is an image that contains noise equal to the noise in the noise image 21. For example, the image correction unit 41 removes the features from the captured image to output an image equal to the ideal image 11.

The image output from the image correction unit 41 is displayed on the image display unit 51. The image displayed on the image display unit 51 is a clear image in which noise has been removed by the image correction unit 41. The image display unit 51 is, for example, a device such as an LCD display or a projector.

3A and 3B are block diagrams showing the configurations of the image acquisition units 110 and 111. FIG.
The imaging device 200 may include an image acquisition unit 110 or an image acquisition unit 111 .

The image acquisition unit 110 has an optical calculation unit 120. The ideal image 11a is an image that does not contain noise caused by the imaging optical system 100. The optical calculation unit 120 calculates the noise and stray light that occurs when the ideal image 11a is input to the imaging optical system 100, and generates a noise image 21a.

The image acquisition unit 111 has an imaging optical system 100 and an imaging optical system 101. The imaging optical system 101 is, for example, an imaging optical system with higher accuracy than the imaging optical system 100 of the imaging device 200. In this case, the ideal image 11b is an image captured by the imaging optical system 101. The noise image 21b is an image captured by the imaging optical system 100. In other words, the ideal image 11b is an image captured with higher accuracy than the noise image 21b.

FIG. 4 is a diagram showing an example of the hardware configuration of the main control unit 42.
FIG. 5 is a diagram showing another example of the hardware configuration of the main control unit 42. In FIG.

The feature extraction unit 31 and the image correction unit 41 are configured, for example, by the main control unit 42. In this case, the main control unit 42 is configured, for example, by at least one processor 42a and at least one memory 42b. In this case, the first image storage unit 12 and the second image storage unit 22 shown in FIG. 2 may be memory 42b.

The processor 42a is, for example, a central processing unit (CPU) that executes programs stored in the memory 42b. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by software, firmware, or a combination of software and firmware. The software and firmware can be stored as programs in the memory 42b. With this configuration, the programs for realizing the functions of the main control unit 42 are executed by the computer.

Memory 42b is a computer-readable recording medium, such as volatile memory, non-volatile memory, or a combination of volatile and non-volatile memory, such as Random Access Memory (RAM) or Read Only Memory (ROM).

The main control unit 42 may have multiple processors 42a and multiple memories 42b. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by these multiple processors 42a and multiple memories 42b.

The main control unit 42 may be configured as a processing circuit 42c as dedicated hardware, such as a single circuit or a composite circuit. The processing circuit 42c may be, for example, a system LSI. In this case, the functions of the feature extraction unit 31 and the image correction unit 41 are realized by the processing circuit 42c.

FIG. 6 is a diagram schematically illustrating an example of the imaging optical system 100. As shown in FIG.
In the example shown in FIG. 6, the imaging optical system 100 includes an imaging element 1 and an optical member 2 .

In Figure 6, the X-axis direction is the horizontal direction of the imaging device 200, the Y-axis direction is the vertical direction of the imaging device 200, and the Z-axis direction is the optical axis direction of the optical element 2. In the example shown in Figure 6, the subject is located on the -Z side of the optical element 2, and the imaging element 1 is located on the +Z side of the optical element 2.

In Figure 6, the direction along the Y-axis direction is, for example, the vertical direction. In this case, the +Y direction is the upward direction in the vertical direction, and the -Y direction is the downward direction in the vertical direction.

The image sensor 1 is an element capable of converting light intensity into an electrical signal. The image sensor 1 is composed of, for example, multiple light-receiving elements arranged two-dimensionally. The image sensor 1 converts imaging information (e.g., light) emitted from a subject area into an electrical signal and can generate an image of the subject area based on the converted electrical signal.

The imaging element 1 may be any element capable of converting received light into an electrical signal. The imaging element 1 may be, for example, an image sensor such as a Charge Coupled Device (CCD) or a Complementary Metal Oxide Semiconductor (CMOS).

The optical element 2 is, for example, a single spherical lens or multiple spherical lenses. In the example shown in Figure 6, the optical element 2 is a single spherical lens.

In the example shown in Figure 6, the imaging optical system 100 has a range of angle of view that can be received by the imaging element 1 of ±30 degrees in the X-axis direction and ±30 degrees in the Y-axis direction. In Figure 6, light rays r0, r10, r20, r30, and r31 represent light rays with angle of view of 0 degrees, 10 degrees, 20 degrees, 30 degrees, and 31 degrees, respectively.

In an imaging optical system, imaging performance is typically highest at the position on the light-receiving surface of the image sensor where light rays with a 0-degree angle of view reach, and imaging performance decreases as the angle of view increases. In other words, the imaging performance of an imaging optical system decreases the further away from the center of the light-receiving surface of the image sensor it is. Imaging performance is expressed, for example, by an index such as the Modulation Transfer Function (MTF), which indicates the degree of contrast reproduction as a spatial frequency characteristic. A high MTF value indicates high imaging performance. Furthermore, for example, imaging performance can be said to be high when the amount of optical aberration or stray light generated in the imaging optical system is small.

On the other hand, image processing techniques that use an ideal image and a noise image generate an image within the range of the image sensor's light-receiving surface. For example, in an imaging optical system with a field of view of ±30 degrees, it is possible to effectively correct the image by using image processing only within the range of ±30 degrees.

However, if the imaging performance of the imaging optical system is poor in the peripheral area outside the center of the image sensor's light-receiving surface, light rays outside the ±30-degree range will be received by the image sensor as stray light. This stray light is information that differs from the known noise components that can be predicted using an ideal image and a noise image. Therefore, in the range where this stray light is received by the image sensor, it is difficult to restore a clear image using image processing techniques.

In contrast, in the imaging device 200 according to this embodiment, the imaging performance of the imaging optical system 100 in the peripheral portion 1B located outside the central portion 1A of the light receiving surface 10 is higher than the imaging performance of the imaging optical system 100 in the central portion 1A. That is, in the example shown in FIG. 6, the imaging performance in the peripheral portion 1B where the light ray r10 or the light ray reaches is higher than the imaging performance in the central portion 1A where the light ray r0 reaches. The central portion 1A of the light receiving surface 10 is the position where the optical axis of the optical member 2 and the imaging element 1 intersect.

In the example shown in Figure 6, the imaging performance of the imaging optical system 100 is highest at the outermost edge 1C of the light receiving surface 10. The outermost edge 1C is located at the outermost part of the light receiving surface 10. In other words, in the example shown in Figure 6, the imaging performance at the outermost edge 1C where light ray r30 reaches is higher than the imaging performance at the center 1A where light ray r0 reaches.

The imaging performance of the imaging optical system 100 may be highest in a portion of the light receiving surface 10 other than the central portion 1A or in a portion of the light receiving surface 10 other than the outermost edge portion 1C. In this case, the imaging performance of the imaging optical system 100 may be highest, for example, in the peripheral portion 1B of the light receiving surface 10.

This prevents light rays outside the angle of view of the imaging optical system 100, such as light ray r31, from being received by the image sensor 1 as stray light. Furthermore, although the imaging performance in the central portion 1A is degraded compared to the imaging performance in the peripheral portion 1B, information about the light ray is retained within the image sensor 1. Therefore, noise such as the amount of blur caused by degradation of imaging performance can be removed as a known noise component.

The optical design of the single lens or spherical lens used as the optical element 2 ensures that the imaging performance of the imaging optical system 100 in the peripheral portion 1B is higher than the imaging performance of the imaging optical system 100 in the central portion 1A. For example, if an imaging optical system has the highest imaging performance in the central portion and the lowest imaging performance in the peripheral portion on the light receiving surface, the imaging optical system 100 can be obtained by translating the optical lens in the optical axis direction so that the imaging performance in the peripheral portion is highest.

Noise components may be removed, for example, using image processing technology. In this case, each aberration, such as image distortion or blur correction, may be corrected individually. The method of aberration correction using image processing technology is not limited to a specific method.

As described above, according to this embodiment, an imaging device 200 can be provided that can remove noise from the entire captured image with high precision, even when using an optical system with a simple configuration.

According to this embodiment, an imaging device 200 can be provided that can correct the entire captured image acquired by the imaging optical system 100 with high precision.

The features of the embodiments and variants described above can be combined with each other.

1 Image sensor, 1A Central portion, 1B Peripheral portion, 1C Outermost edge portion, 2 Optical member, 10 Light receiving surface, 11 Ideal image, 12 First image memory unit, 21 Noise image, 22 Second image memory unit, 31 Feature extraction unit, 32 Machine learning processing unit, 41 Image correction unit, 51 Image display unit, 100 Imaging optical system, 200 Imaging device.

Claims (9)

an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount,
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The feature extraction unit
a first image storage unit in which the ideal image is stored;
a second image storage unit in which the noise image is stored;
a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image;
and
The first image storage unit stores an image having a wider range than the range captured by the imaging optical system.
Imaging device. The imaging device according to claim 1 , wherein the second image storage unit stores an image having a wider range than the range captured by the imaging optical system. an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface ,
The feature extraction unit
a first image storage unit in which the ideal image is stored;
a second image storage unit in which the noise image is stored;
a machine learning processing unit that extracts the feature amounts by performing convolution processing on the ideal image and the noise image;
and
The second image storage unit stores an image having a wider range than the range captured by the imaging optical system.
Imaging device. The imaging device according to claim 1 , wherein the imaging optical system includes a single spherical lens or a plurality of spherical lenses. The imaging device according to claim 1 , wherein the imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface. an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The imaging performance of the imaging optical system is highest at the outermost edge of the light receiving surface.
Imaging device. The imaging device according to claim 1 , wherein the imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or in a portion of the light receiving surface other than the outermost edge portion. an imaging optical system that has an imaging element having a light receiving surface and performs imaging;
a feature extraction unit that extracts feature amounts from a noise image including noise due to the imaging optical system and an ideal image obtained by removing the noise from the noise image;
an image correction unit that corrects an image captured by the imaging optical system using the feature amount;
Equipped with
the imaging performance of the imaging optical system in a peripheral portion located outside the center portion of the light receiving surface is higher than the imaging performance of the imaging optical system in the center portion of the light receiving surface,
The imaging performance of the imaging optical system is highest in a portion of the light receiving surface other than the central portion or a portion of the light receiving surface other than the outermost edge portion.
Imaging device. The imaging device according to claim 1 , further comprising an image display unit for displaying the image output from the image correction unit.