JP6819352B2 - Imaging equipment, imaging methods and programs - Google Patents

Description

The present invention relates to imaging devices, imaging methods and programs.

For example, there are known image pickup devices that generate one image by connecting images captured by each of a plurality of image pickup elements (image sensors), such as a spherical image. In such an imaging apparatus, there is known a technique of aligning the scanning directions and adjusting the exposure timing in order to improve the quality of the joint (see, for example, Patent Document 1).

However, for example, in order to physically align the scanning directions, it is conceivable to prepare a plurality of different image sensor boards (boards on which the image sensor is mounted), but in this case, there is a problem that the cost increases. When one type of substrate (one substrate on which a plurality of image sensors are mounted) is used in order to reduce the cost, the reading direction from the image sensor (the direction in which the image of each line is read) is operated in the opposite direction. It is possible to align the scanning directions, but in this case, the read timing between the image sensors may be shifted by several lines. That is, in the past, there is a problem that the quality of the joint cannot be guaranteed while suppressing the increase in cost.

The present invention has been made in view of the above, and an object of the present invention is to ensure the quality of joints while suppressing an increase in cost.

In order to solve the above-mentioned problems and achieve the object, the present invention relates to an image pickup device that generates one image by connecting images taken by each of a plurality of image pickup elements corresponding to a plurality of optical systems. each said imaging device is a rolling shutter system for reading the exposure and an image line by line, so as to correct the deviation of the exposure timing of each of the image pickup device, a correction control section for controlling the imaging device, wherein The correction control unit is an image pickup device that crops and reads out the deviation of the exposure timing with respect to the image sensor having the earlier exposure timing .

According to the present invention, the quality of the joint can be ensured while suppressing the increase in cost.

FIG. 1 is a diagram showing an example of the hardware configuration of the image pickup apparatus of the embodiment. FIG. 2 is a diagram showing an example of a detailed configuration of the imaging unit of the embodiment. FIG. 3 is a diagram for explaining the arrangement and scanning direction of the image sensor. FIG. 4 is a diagram for explaining the influence of the image due to the deviation of the exposure timing. FIG. 5 is a diagram for simply explaining how to connect two captured images to one spherical image. FIG. 6 is a diagram for explaining the deviation of the exposure timing. FIG. 7 is a diagram showing an example of the functions of the image pickup apparatus of the embodiment. FIG. 8 is a diagram for explaining a method for correcting the deviation of the exposure timing. FIG. 9 is a diagram showing an example in which the input of the horizontal synchronization signal is delayed by the amount of deviation.

Hereinafter, embodiments of the imaging apparatus, imaging method, and program according to the present invention will be described in detail with reference to the accompanying drawings. In the following embodiment, an image pickup device that generates a spherical image in which two hemispherical images are joined will be described as an example, but the present invention is not limited thereto. In short, the image pickup device may be any image pickup device that generates one image by connecting the images captured by each of the plurality of image pickup elements corresponding to the plurality of optical systems. The spherical image is an image captured over the entire field of view with a horizontal angle of 360 degrees and a vertical angle of 180 degrees.

FIG. 1 is a diagram showing an example of the hardware configuration of the image pickup device 1 (imaging device 1 that generates a spherical image) of the present embodiment. Hereinafter, the optical systems 10b and 10c constituting the fisheye lens and the image pickup devices 13 and 14 will be collectively referred to as an image pickup unit 10. The optical system 10b corresponds to the image sensor 13, and the optical system 10c corresponds to the image sensor 14. In FIG. 1, the image sensors 13 and 14 are image sensors such as CMOS sensors and CCD sensors that convert optical images by the optical systems 10b and 10c into image data of electrical signals and output them, and horizontal / vertical synchronization of the image sensors. It has a timing generation circuit that generates signals, pixel clocks, and the like, and a group of registers in which various commands and parameters necessary for the operation of the image sensor are set. It is necessary that the synchronization signals of these two image sensors 13 and 14 are synchronized with each other so that they can be imaged at the same timing. The method for that will be described later.

Each of the image sensors 13 and 14 of the image pickup unit 10 is connected to the image processing unit 20 via a parallel I / F bus. Further, each of the image pickup elements 13 and 14 of the image pickup unit 10 is separately connected to the image pickup control unit 30 via a serial I / F bus (I2C bus or the like). The image processing unit 20 and the image pickup control unit 30 are connected to the CPU 40 via the bus 100. Further, the ROM 50, the SRAM 60, the DRAM 70, the operation unit 80, the external I / F circuit 90, and the like are connected to the bus 100. The image processing unit 20 takes in the image data output from the image sensors 13 and 14 through the parallel I / F bus, performs predetermined processing on each image data, and then synthesizes these image data. Generate an all-sky image.

The image pickup control unit 30 generally uses the image pickup control unit 30 itself as a master device and the image pickup elements 13 and 14 as slave devices, and sets commands and the like in the register group of the image pickup elements 13 and 14 by using the I2C bus. Necessary commands and the like are received from the CPU 40. Further, the image pickup control unit 30 also uses the I2C bus to take in the status data and the like of the register group of the image pickup elements 13 and 14 and send them to the CPU 40. Further, the image pickup control unit 30 instructs the image pickup devices 13 and 14 to output image data at the timing when the shutter button of the operation unit 80 is pressed. In the present embodiment, the CPU 40 and the image pickup control unit 30 cooperate to control the image pickup element 13 or 14 so as to correct the deviation of the exposure timing (image readout timing) of the image pickup elements 13 and 14. (Correction control unit) is realized. This function is realized by the CPU 40 executing a predetermined program to operate the imaging control unit.

Depending on the type of the image pickup device 1 that generates the spherical image, it may have a preview display function on the display and a function corresponding to the moving image display. In that case, the image data output from the image sensors 13 and 14 is continuously performed at a predetermined frame rate (frames / minute). The CPU 40 controls the overall operation of the image pickup apparatus 1 and executes necessary processing. The ROM 50 stores various programs for the CPU 40. The SRAM 60 and the DRAM 70 are work memories, and store programs executed by the CPU 40, data in the middle of processing, and the like. Here, the DRAM 70 is also used to store image data in the process of being processed by the image processing unit 20 and data of the processed spherical image.

The operation unit 80 is a general term for various operation buttons, a power switch, a shutter button, a touch panel having both display and operation functions, and the like. By operating the operation buttons, the user inputs various shooting modes and shooting conditions. The external I / F circuit 90 is a general term for an interface circuit (USB I / F, etc.) with an external memory (SD card, flash memory, etc.) or a personal computer. Further, the external I / F circuit 90 may be a network interface regardless of whether it is wireless or wired. The spherical image stored in the DRAM 70 is stored in the external memory via the external I / F circuit 90, and if necessary, the personal computer via the external I / F circuit 90 that serves as the network I / F. Or, it is transferred to a smartphone or the like.

FIG. 2 is a diagram showing an example of a detailed configuration of the imaging unit 10. As shown in FIG. 2, the image pickup unit 10 of the present embodiment includes an optical system 10b, a cover glass CGb, an image pickup element 13, an optical system 10c, a cover glass CGc, and an image pickup element 14. .. The optical systems 10b and 10c both form an equidistant projection method in order to form a fisheye lens. The optical system 10b includes a lens group (first lens group) as a front group, an aperture 4b, and a lens group (second lens group) as a rear group. The optical system 10c includes a lens group (first lens group) as a front group, an aperture 4c, and a lens group (second lens group) as a rear group.

The lens group, which is the front group of the optical system 10b, is arranged on the object side of the optical system 10b. This lens group captures light rays with a wide angle of view exceeding 180 degrees with a negative refractive power as a whole. This lens group is configured such that the lens L1b, the lens L2b, and the right-angle prism PSb are arranged in series in this order from the object side to the image side. The right-angle prism PSb has a mirror surface MSb on which a reflective film is formed by coating aluminum on the slope portion. The right-angle prism PSb internally reflects light rays from the front lens group toward the rear lens group by the mirror surface MSb. That is, the optical axis OA2b of the optical system 10b is bent 90 degrees at the mirror surface MSb of the right-angle prism PSb, passes through the lens group that is the rear group, and heads toward the image sensor 13.

The lens group, which is the rear group of the optical system 10b, is arranged on the image side of the optical system 10b. This lens group has a positive refractive power as a whole and mainly corrects aberrations in the captured image. This lens group is configured such that the lens L3b, the lens L4b, the lens L5b, the lens L6b, and the lens L7b are arranged in series in this order from the object side to the image side.

The lens group, which is the front group of the optical system 10c, is arranged on the object side of the optical system 10c. This lens group captures light rays with a wide angle of view exceeding 180 degrees with a negative refractive power as a whole. This lens group is configured such that the lens L1c, the lens L2c, and the right-angle prism PSc are arranged in series in this order from the object side to the image side.

The right-angle prism PSc has a mirror surface MSc on which a reflective film is formed by coating aluminum on the slope portion. The right-angle prism PSc internally reflects light rays from the front lens group toward the rear lens group by the mirror surface MSc. That is, the optical axis OA2c of the optical system 10c is bent 90 degrees on the mirror surface MSc of the right-angle prism PSc, passes through the lens group that is the rear group, and heads toward the image sensor 14.

The lens group, which is the rear group of the optical system 10c, is arranged on the image side of the optical system 10c. This lens group has a positive refractive power as a whole and mainly corrects aberrations in the captured image. This lens group is configured such that the lens L3c, the lens L4c, the lens L5c, the lens L6c, and the lens L7c are arranged in series in this order from the object side to the image side.

Further, the outer surface on which the mirror surface MSb of the right-angle prism PSb of the optical system 10b is formed and the outer surface on which the mirror surface MSc of the right-angle prism PSc of the optical system 10c is formed are adhesively fixed to each other. That is, since it has a configuration in which two imaging systems are combined, it is possible to realize a miniaturization of the entire imaging unit 10 and to realize a handheld system. Further, since the maximum angles of view of the optical system 10b and the optical system 10c are each 190 degrees or more, these are combined as shown in FIG. 2 to capture an image of 360 degrees vertically and horizontally. The unit 10 can be configured.

FIG. 3 is a diagram for explaining the arrangement and scanning direction of the image sensor. FIG. 3A is an image diagram in which the configuration of FIG. 2 is simplified. In the imaging device 1 that generates a spherical image, it is possible to reduce the region where imaging is impossible (the region where imaging is not possible 6) when the intervals between the optical systems 10b and 10c are narrow. Many image sensors, such as CMOS image sensors generally used in cameras, have a 3: 2 or 4: 3 horizontally long shape. In order to narrow the distance between the optical systems 10b and 10c, it is advantageous to arrange the image pickup elements 13 and 14 vertically as shown in FIG. 3B. Further, in general, since the column ADC of the CMOS image sensor is often provided on the long side side, the scanning direction is the horizontal direction as shown in 131a and 131b shown in FIG. 3B. When the scanning directions 131a and 131b of the image pickup devices 13 and 14 are aligned as shown in FIG. 3B, the scanning directions 132a and 132b in space are different because the light passes through the lens optical system and the mirror of the prism. The direction is as shown in FIG. 3B. As will be described in detail later, as shown in FIG. 3B, the scanning directions in space should be aligned, and the reading of images (images in line units) from each of the two image sensors 13 and 14 should be synchronized. For example, by matching the exposure timings of the contact portions connecting the images captured by the two image pickup devices 13 and 14, the quality of the joints with respect to the moving object can be improved. If a mechanical shutter is mounted on the optical system, the influence of such exposure timing is almost eliminated in a still image, but there is no mechanical shutter and a rolling shutter in the image sensor may be used. In addition, a rolling shutter is used when shooting a moving image regardless of the presence or absence of a mechanical shutter.

On the other hand, with respect to the substrate on which the image pickup elements 13 and 14 are mounted, when one type of substrate is used, due to restrictions on the connection from the sensor (image pickup elements 13 and 14) to the main substrate, FIG. It becomes necessary to rotate it in the horizontal direction so that it faces each other. Then, as shown in FIG. 3C, the scanning directions 132a and 132b'in space are not aligned. Therefore, for example, it is possible to reverse the mounting directions of the image pickup devices 13 and 14 to align the scanning directions, but in that case, it is necessary to prepare two types of substrates, which is disadvantageous in terms of cost and quality. On the other hand, as a function of the image sensors 13 and 14, there is a function of reversing the reading direction. However, when this function is used, the read timing between the image sensors may be shifted by several lines.

Next, the influence of the image due to the deviation of the exposure timing will be described. In the present embodiment, each of the image pickup devices 13 and 14 uses a rolling shutter method in which exposure and image reading are performed in line units (line units). FIG. 4A is a diagram showing a subject to be imaged. In (A) of FIG. 4, it is assumed that the subject 8 and the subject 9 exist so as to straddle the joint 7 of the spherical images, the subject 8 is moving in the direction of the arrow, and the subject 9 is stationary. ..

FIG. 4B is a diagram showing an image obtained by capturing an image of FIG. 4A when the exposure timing is correct. It is assumed that the scanning 130 of the image sensor 13 and the scanning 140 of the image sensor 14 have the same direction and timing. The subjects 10 and 11 captured at that time can be connected without any problem. For convenience of explanation, the joint 7 and the scanning direction are shown, but they are not shown in the actual image.

FIG. 4C is a diagram showing an image obtained by capturing an image of FIG. 4A when the exposure timing is deviated. The scan 130 of the image sensor 13 and the scan 140'of the image sensor 14 have the same directions, but the timings are different. Since the subject 11 captured at that time is stationary, it can be connected without any problem regardless of the exposure timing, but the moving subject 10 has a time lag, so that it shifts as shown in the figure and is accurate. I can't connect.

Next, with reference to FIG. 5, a method of connecting two captured images to one spherical image will be briefly described. In FIG. 5, the solid line on the captured image 141 and the solid line on the captured image 150 indicate the image circle of the fisheye lens, and each of the optical systems 10b and 10c shown in FIG. 2 has an angle of view of 190 degrees or more. Actually, the spherical image 160 is generated by combining the images of 180 degrees, so the region to be used is shown by a dotted line in FIG.

In the example of FIG. 5, it is assumed that the captured image 141 is arranged in the center and the captured images 150 are connected to the left and right in half to form a spherical image 160. At that time, assuming that the points Pa1 to Pa4 and Pb1 to Pb4 on the joint generate a spherical image 160 in which Pa1 and Pb2, Pa2 and Pb1, Pa3 and Pb4, and Pa4 and Pb3 are connected at the time of connection. Continue the explanation of.

Next, with reference to FIG. 6, the deviation of the exposure timing (timing for reading out the image in line units) will be described. FIG. 6A shows a diagram imagining the reading of an ideal state in which the two image sensors 13 and 14 are synchronized and there is no timing deviation (assuming the image described in FIG. 5). In the following description, when the image sensors 13 and 14 are not distinguished from each other, they are simply referred to as “image sensors”. As described above, here, it is assumed that the image sensor adopts the rolling shutter method and reads out one line at a time from the top to the bottom in the arrow direction (exposures in line units and reads out the image).

The time for reading the points Pa1 and Pa2 on the captured image 141 is ta1, and the time for reading the points Pa3 and Pa4 is ta2. Similarly, the time for reading the points Pb1 and Pb2 on the captured image 150 is tb1, and the time for reading the points Pb3 and Pb4 is tb2. Since (A) in FIG. 6 is an ideal state with no deviation, ta1 = tb1 and ta2 = tb2, and as described in FIG. 5, the exposure timings of the connected points coincide with each other.

FIG. 6B shows a case where the readout time of the captured image 150 changes due to a synchronization shift of the image pickup device or a timing shift due to reverse readout. In this case, ta1 and tb1 are different, and ta2 and tb2 are also different, so that the exposure timings of the connected points are different from each other, resulting in the state described in FIG. 4C.

FIG. 6C shows a photographed image when the image circle does not come to the center due to mechanical restrictions or when the image circle shifts due to an assembly error. Also in this case, since ta1 and tb1 are different and ta2 and tb2 are also different, the exposure timings of the connected points are different from each other, and the state described in FIG. 4C is obtained. Such an unintended image circle misalignment can be softly corrected by the adjustment process, but the exposure timing misalignment is not corrected.

Therefore, the image pickup device 1 of the present embodiment has a function (correction control unit) of controlling the image pickup element so as to correct the deviation of the exposure timing of each image pickup element. As described above, this function is realized by the cooperation of the CPU 40 and the imaging control unit 30. Further, as described above, it is assumed that each image sensor uses a rolling shutter method in which exposure and image reading are performed in line units, and the scanning direction of each image sensor (direction of scanning light for each line) is It is assumed that they are in the same direction.

FIG. 7 is a diagram showing functions related to the present invention among the functions of the image pickup apparatus 1 of the present embodiment. For convenience of explanation, only the functions related to the present invention will be illustrated here, but the functions of the image pickup apparatus 1 are not limited to these. As shown in FIG. 7, the image pickup apparatus 1 includes a shift amount storage unit 201 and a correction control unit 202.

The deviation amount storage unit 201 stores in advance the amount of deviation between the exposure timing of the image pickup device 13 and the exposure timing of the image pickup element 14. This amount of deviation can be grasped for each individual in the adjustment process.

The correction control unit 202 controls the image sensor so as to correct the deviation amount stored in advance in the deviation amount storage unit 201. In the present embodiment, the correction control unit 202 crops the image sensor with the earlier exposure timing (in this example, any of the image sensors 13 and 14) by the amount of deviation (exposure timing deviation). And read it out.

FIG. 8A corresponds to the state of FIG. 6C. In FIG. 8B, the correction control unit 202 crops and reads out the image sensor (the image sensor having the earlier exposure timing) that images the image 150 by the amount of the deviation of the exposure timing (line). It is shown that it is possible to make the reading time uniform (ta1 = tb1 and ta2 = tb2) by exposing the image in units and reading the image).

The deviation amount correction method is not limited to the above, and various correction methods can be adopted. For example, the correction control unit 202 may read out a blank line or a dummy line by the amount of deviation for the image sensor having an earlier exposure timing. In FIG. 8C, the correction control unit 202 has a blank line or a dummy line for the image sensor (the image sensor having the earlier exposure timing) that images the image 150. It is shown that it is possible to make the read time uniform (ta1 = tb1 and ta2 = tb2) by reading.

Further, for example, the correction control unit 202 may delay the input of the horizontal synchronization signal for reading the image in line units by the amount of deviation for the image sensor having the earlier exposure timing. For example, as shown in FIG. 9, the horizontal synchronization signal 170 for reading the image of the captured image 141 in line units is compared with the horizontal synchronization signal 180 for reading out the image of the captured image 150 in line units (the one with the earlier exposure timing). By delaying the horizontal synchronization signal 180) input to the image sensor of the above by the correction time 190 (time ta1-tb1 corresponding to the number of correction lines), it is possible to make the reading times uniform (ta1 = tb1, ta2 = tb2). become.

As described above, in the present embodiment, by controlling the image pickup element so as to correct the deviation of the exposure timing of each image pickup element having the same scanning direction, the image pickup image captured by each image pickup element is obtained. When a single image is generated by joining them, it is possible to ensure the quality of the joint between the captured images while suppressing an increase in cost (because it is not necessary to provide a plurality of imaging substrates).

Although the embodiments according to the present invention have been described above, the present invention is not limited to the above-described embodiments as they are, and at the implementation stage, the components can be modified and embodied without departing from the gist thereof. In addition, various inventions can be formed by an appropriate combination of the plurality of components disclosed in the above-described embodiment. For example, some components may be removed from all the components shown in the embodiments.

Further, the program executed by the image pickup apparatus 1 of the above-described embodiment is a file in an installable format or an executable format, and is a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versaille Disk), or a file. It may be configured to be recorded and provided on a computer-readable recording medium such as USB (Universal Serial Bus), or may be provided or distributed via a network such as the Internet. In addition, various programs may be configured to be provided by incorporating them into a ROM or the like in advance.

1 Image sensor 10 Image sensor 10b Optical system 10c Optical system 13 Image sensor 14 Image sensor 20 Image processing unit 30 Image control unit 40 CPU
50 ROM
60 SRAM
70 DRAM
80 Operation unit 90 External I / F circuit 100 Bus 201 Deviation amount storage unit 202 Correction control unit

JP 2015-61125

Claims (3)

In an image pickup device that generates one image by stitching together images taken by each of a plurality of image pickup elements corresponding to a plurality of optical systems.
Each of the image sensors is a rolling shutter system that performs exposure and image readout on a line-by-line basis.
So as to correct the deviation of the exposure timing of each of said image pickup element, e Bei correction control section for controlling the imaging device,
The correction control unit crops and reads out the image sensor having the earlier exposure timing by the amount of the deviation of the exposure timing.
Imaging device. This is an imaging method using an imaging device that generates one image by joining images captured by each of a plurality of imaging elements corresponding to a plurality of optical systems.
Each of the image sensors is a rolling shutter system that performs exposure and image readout on a line-by-line basis.
A correction control step that controls the image sensor so as to correct the deviation of the exposure timing of each image sensor ,
The image sensor having an earlier exposure timing has a step of cropping and reading out by the amount of the deviation of the exposure timing .
Imaging method. For an image pickup device that generates one image by connecting images taken by each of a plurality of image pickup elements, which is a rolling shutter system that supports multiple optical systems and performs exposure and image readout in line units .
A correction control step that controls the image sensor so as to correct the deviation of the exposure timing of each image sensor ,
A program for executing a step of cropping and reading out the difference of the exposure timing for the image sensor having the earlier exposure timing .

Images