JP7630292B2 - Image capture device and control method thereof - Google Patents

Description

The present invention relates to an imaging device and a control method thereof.

Conventionally, in imaging devices, it is common to drive and control the frame rate of the display device (hereinafter referred to as the "display frame rate") and the frame rate when recording in a memory or the like (hereinafter referred to as the "recording frame rate") in synchronization with the set frame rate during shooting (hereinafter referred to as the "imaging frame rate"). For example, when the imaging frame rate is 30p, the display frame rate and recording frame rate are also driven and controlled at 30p.

On the other hand, in recent years, display devices (OLED displays) using organic light-emitting diodes (OLEDs) have been widely used. OLED displays have the characteristic of having a faster response speed for switching between frames than liquid crystal display devices (LCDs), which have been commonly used in the past. With LCDs, which have a slow response speed for switching between frames, when shooting at a low imaging frame rate (e.g., 30p) while moving the angle of view by panning, etc., blurring of the captured image is recognized as an afterimage when displayed, and it is perceived as if interpolation is being performed between frames. This reduces the deterioration of visibility. However, with OLED displays, the response speed for switching between frames is fast, so when the image is displayed at the same display frame rate as the imaging frame rate, it is recognized as blurring as it is, resulting in a problem of worse visibility compared to LCDs.

Therefore, in order to improve visibility on an OLED display, it is conceivable to speed up the display update cycle by setting a high display frame rate capable of driving the OLED display, and to speed up the imaging frame rate to match the display frame rate. However, if the imaging frame rate becomes faster than the recording frame rate at which recording can be performed, it is not possible to record images of all captured frames, as shown in FIG. 12.

Patent Document 1 describes the following regarding the imaging frame rate, display frame rate, and recording frame rate. That is, when the imaging frame rate is faster than the display frame rate and the recording frame rate, the captured images of each frame are temporarily written to memory. Then, for display, the frame rate is converted by thinning out and reading the images from memory at the display frame rate and displaying them. Also, for recording, the read frame rate of the captured images from memory is controlled to the recording frame rate, so that the captured images are recorded without thinning out.

Patent Document 2 also discloses that when shooting at a high frame rate and performing long-exposure live view at a low frame rate, the number of shot images required to achieve appropriate brightness are sequentially synthesized and displayed according to the frame rate of the display device. It also discloses that when recording video, as with live view, the number of shot images required to achieve appropriate brightness are sequentially synthesized and the synthesized images are sequentially recorded.

Patent No. 4424174 JP 2020-10140 A

However, the method described in Patent Document 1 has the problem that if the subject is dark and sufficient charge accumulation time (exposure time) cannot be ensured for each frame, the resulting image will be dark.

The method described in Patent Document 2 is based on the premise that the display frame rate and the recording frame rate are the same. Therefore, when a display device capable of updating the display at a high display frame rate, such as an OLED display, is used, the recording may not be able to keep up, or if the display frame rate is matched to the recording frame rate, it is not possible to expect any improvement in visibility due to blurring.

The present invention was made in consideration of the above problems, and aims to capture, display, and record images at frame rates appropriate for each, and to obtain images appropriate for both display and recording.

In order to achieve the above-mentioned object, an imaging device of the present invention is characterized in having a setting means for setting a first charge accumulation time and an exposure , a control means for controlling an imaging frame rate for driving an imaging element, a display frame rate of a display means, and a recording frame rate of a recording means, a driving means for controlling a second charge accumulation time in each frame of the imaging frame rate based on the first charge accumulation time and the imaging frame rate to drive the imaging element, a first generation means for generating and outputting a display image equivalent to the exposure set by the setting means at the display frame rate from an image output from the imaging element at the imaging frame rate, based on the imaging frame rate, the display frame rate, the first charge accumulation time, and the second charge accumulation time, when the recording frame rate is equal to or lower than the imaging frame rate, and a second generation means for generating and outputting a recording image equivalent to the exposure set by the setting means at the recording frame rate, based on the imaging frame rate, the recording frame rate, the first charge accumulation time, and the second charge accumulation time, when the recording frame rate is equal to or lower than the imaging frame rate.

According to the present invention, images can be captured, displayed, and recorded at frame rates appropriate for each, and images appropriate for both display and recording can be obtained.

1 is a block diagram showing a schematic configuration of an imaging apparatus according to a first embodiment of the present invention. FIG. 4 is a conceptual diagram for explaining accumulation control in the first embodiment. FIG. 4 is a conceptual diagram for explaining synthesis control in the first embodiment. 5 is a flowchart showing synthesis control in the first embodiment. FIG. 11 is a block diagram showing a schematic configuration of an imaging apparatus according to a second embodiment. FIG. 11 is a conceptual diagram for explaining accumulation control in the second embodiment. FIG. 11 is a conceptual diagram for explaining synthesis control in the second embodiment. 10 is a flowchart showing synthesis control in the second embodiment. FIG. 13 is a conceptual diagram for explaining accumulation control in a modified example of the second embodiment. FIG. 13 is a conceptual diagram for explaining gain control and synthesis control in a modified example of the second embodiment. 10 is a flowchart showing gain control and synthesis control in a modified example of the second embodiment. FIG. 13 is a conceptual diagram showing timings of imaging, display, and recording for explaining a conventional example.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
A first embodiment of the present invention will now be described.
Fig. 1 is a block diagram showing an example of a schematic configuration of an image capturing apparatus 100 according to a first embodiment. As shown in Fig. 1, the image capturing apparatus 100 shows the configuration of an electronic device equipped with a camera, such as a so-called digital still camera, a digital video camera, or a mobile terminal with a camera.

The amount of light incident from the outside through the lens 101 is controlled by the light-reducing unit 102 and enters the imaging unit 103. In FIG. 1, the lens 101 is shown as a single lens, but it is usually made up of multiple lenses. The light-reducing unit 102 is made up of aperture blades, an ND (Neutral Density) filter, etc.

The imaging unit 103 is composed of an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) type or a CCD, a driver for driving the imaging element, a timing generation circuit, a CDS/AGC circuit, an A/D converter, etc. The driver and timing generation circuit supply driving pulses and the like for driving the imaging element to the imaging element, and read out the image captured by the imaging element and adjust the exposure time. The imaging element captures an image of a subject by photoelectrically converting incident light whose amount is controlled by the light reduction unit 102, and the CDS/AGC circuit samples and amplifies image information based on the charge (image signal) stored in each pixel of the imaging element. Note that correlated double sampling (CDS) is performed for sampling, and automatic gain control (AGC) is performed for amplification. The A/D converter converts the image signal (analog signal) output from the CDS/AGC circuit into a digital signal and outputs it.

The image processing unit 104 performs various signal processing such as AWB (Auto White Balance) and gamma control on the image signal (digital signal) output from the A/D converter, adjusts the color, brightness, etc., develops the image, and outputs the generated image.

The microcomputer 105 is a circuit also called a microcontroller, and performs overall control of the operations of the imaging device 100, including the above-mentioned controls.

The imaging device 100 of this embodiment also has a frame rate switching control unit 106, which determines whether to switch the imaging frame rate, the display frame rate, and the recording frame rate. If switching is necessary, the microcomputer 105 is notified of the switched frame rate. The microcomputer 105 controls the accumulation control unit 107, the first synthesis control unit 108, the second synthesis control unit 109, and the third synthesis control unit 112 in response to instructions from the frame rate switching control unit 106.

The accumulation control unit 107 calculates the charge accumulation time for each frame of the imaging unit 103 according to the imaging frame rate and notifies the imaging unit 103.

The display unit 110 displays the display image output from the first synthesis control unit 108 at the display frame rate of the display unit 110, and the external display unit 210 displays the display image output from the third synthesis control unit 112 at the display frame rate of the external display unit 210. In addition, the recording unit 111 records the display image output from the second synthesis control unit 109 at the recording frame rate of the recording unit 111.

Next, the control of the charge accumulation time in the accumulation control unit 107 and the specific operations in the first synthesis control unit 108, the second synthesis control unit 109, and the third synthesis control unit 112 will be described with reference to Figures 2 and 3.

For example, if the image capture frame rate is 30p and the charge accumulation time is 1/80 seconds, and the image capture frame rate is changed to 60p, a charge accumulation time of 1/80 seconds can be ensured for each frame. Therefore, the accumulation control unit 107 sets the charge accumulation time for each frame to 1/80 seconds (Figure 2(a)).

In addition, if the image frame rate is 30p and the charge accumulation time is 1/40 seconds, and the image frame rate is changed to 60p, it is not possible to take a charge accumulation time of 1/40 seconds for each frame. In such a case, the accumulation control unit 107 controls the charge accumulation time of each frame to 1/80 seconds, which is half of the charge accumulation time of 1/40 seconds, so that the charge accumulation time for two frames is 1/40 seconds (Figure 2 (b)). In this way, when one frame period (cycle) of the changed image frame rate is shorter than the charge accumulation time, the charge accumulation time is divided and allocated to multiple frames. Note that when this allocation is made, it is allocated to multiple frames in the number corresponding to one frame period of the image frame rate before the change.

The first synthesis control unit 108 synthesizes the required number of images as necessary based on the imaging frame rate notified by the microcomputer 105, the display frame rate of the display unit 110, and the charge accumulation time set by photometry or user instructions to generate a display image. Then, the first synthesis control unit 108 outputs the obtained display image to the display unit 110 at the display frame rate.

For example, when the imaging frame rate is 30p, the charge accumulation time is 1/80 seconds (corresponding to the state before the frame rate is changed in FIG. 2(a)), and the display frame rate of the display unit 110 is 30p, the first synthesis control unit 108 displays the image output in each frame from the image processing unit 104 as is on the display unit 110. Note that the same display is performed when the charge accumulation time is 1/40 seconds (corresponding to the state before the frame rate is changed in FIG. 2(b)).

Also, when the imaging frame rate is 60p, the charge accumulation time is 1/80 seconds (corresponding to the frame rate change in FIG. 2(a)), and the display frame rate of the display unit 110 is 60p, the first synthesis control unit 108 displays the image output from the image processing unit 104 for each frame as is on the display unit 110 (FIG. 3(a)). On the other hand, when the imaging frame rate is 60p, the charge accumulation time is 1/40 seconds (corresponding to the frame rate change in FIG. 2(b)), and the display frame rate of the display unit 110 is 60p, the first synthesis control unit 108 sequentially synthesizes the images two frames at a time while shifting, and displays the display image obtained by synthesizing on the display unit 110 (FIG. 3(b)). This makes the total charge accumulation time of each display image 1/40 seconds.

The second synthesis control unit 109 synthesizes the required number of images as necessary based on the imaging frame rate notified by the microcomputer 105, the recording frame rate of the recording unit 111, and the charge accumulation time set by photometry or user instructions to generate a recording image. Then, it outputs the obtained display image to the recording unit 111 at the recording frame rate.

For example, when the imaging frame rate is 30p, the charge accumulation time is 1/80 seconds (corresponding to the time before the frame rate is changed in FIG. 2(a)), and the recording frame rate of the recording unit 111 is 30p, the second synthesis control unit 109 outputs the images output for each frame from the image processing unit 104 as is to the recording unit 111. Note that the same output is performed when the charge accumulation time is 1/40 seconds (corresponding to the time before the frame rate is changed in FIG. 2(b)).

In addition, when the imaging frame rate is 60p, the charge accumulation time is 1/80 seconds (corresponding to the frame rate after change in FIG. 2(a)), and the recording frame rate of the recording unit 111 is 30p, the images output from the image processing unit 104 for each frame are thinned out and recorded by the recording unit 111 (FIG. 3(a)). On the other hand, when the imaging frame rate is 60p, the charge accumulation time is 1/40 seconds (corresponding to the frame rate after change in FIG. 2(b)), and the recording frame rate of the recording unit 111 is 30p, the second synthesis control unit 109 synthesizes images two frames at a time, and outputs the recorded image obtained by synthesis to the recording unit 111 (FIG. 3(b)). This makes the total charge accumulation time of each recorded image 1/40 seconds.

The third synthesis control unit 112 synthesizes the required number of images as necessary based on the imaging frame rate notified by the microcomputer 105, the display frame rate of the external display unit 210, and the charge accumulation time set by photometry or user instructions. The obtained display images are then output to the external display unit 210 at the display frame rate. Note that the processing in the third synthesis control unit 112 is similar to the processing in the first synthesis control unit 108, and therefore a detailed description thereof will be omitted.

Next, the control of the frame rate, charge accumulation time, and image synthesis in the imaging device 100 of this embodiment will be described with reference to the flowchart in FIG. 4. Here, the control when the external display unit 210 is connected to the imaging device 100 will be described, and it is assumed that the default imaging frame rate, display frame rate, and recording frame rate are the same when the external display unit 210 is not connected. It is also assumed that the recording frame rate cannot be changed from the default frame rate.

When the external display unit 210 is connected to the imaging device 100, in S101, it is determined whether the responsiveness of the external display unit 210 in switching between frames is fast. If it is determined that the responsiveness of the external display unit 210 in switching between frames is not fast, the process proceeds to S104, and images are displayed without changing either the imaging frame rate or the display frame rate. In this case, images captured at the default imaging frame rate are output to the external display unit 210 via the third synthesis control unit 112. Note that if the display frame rate of the external display unit 210 is slower than the default imaging frame rate, the third synthesis control unit 112 may thin out the images before outputting them.

In the next step S105, the recording unit 111 records images at the default recording frame rate.

On the other hand, if it is determined in S101 that the response of switching between frames on the external display unit 210 is fast, then in S102 the imaging device 100 determines whether or not the imaging element of the imaging unit 103 can be driven at the same imaging frame rate as the high-speed display frame rate of the external display unit 210. If it is determined that the imaging element cannot be driven at the same frame rate as the display frame rate of the external display unit 210, the imaging frame rate is not changed. Then, in S104, images are displayed at the default display frame rate, and in S105, images are recorded at the default recording frame rate. In this case, the image obtained at the default imaging frame rate is also output to the third synthesis control unit 112.

If it is determined in S102 that the imaging device 100 can drive the imaging element of the imaging unit 103 at the same imaging frame rate as the high-speed display frame rate of the external display unit 210, then in S103 the imaging frame rate is changed to match the high-speed display frame rate of the external display unit 210. Then, in S106, the charge accumulation time set in the imaging device 100 is compared with one frame period (cycle) of the changed imaging frame rate. If one frame period is equal to or greater than the charge accumulation time, proceed to S107, and if one frame period is less than the charge accumulation time, proceed to S109.

In S107, the display frame rate of the display unit 110 is set to the changed image capture frame rate. This allows the display unit 110 to quickly switch between images to be displayed at the same cycle as the high-speed image capture frame rate. In addition, in S108, the frames of the captured image are thinned out to match the default recording frame rate, and recorded in the recording unit 111 (control in FIG. 3(a)). Note that if the display frame rate of the display unit 110 cannot be increased to the image capture frame rate, the display frame rate of the display unit 110 is switched to a higher rate to the extent possible. Then, the frames of the captured image are thinned out and displayed to match the changed display frame rate.

On the other hand, in S109, as described with reference to FIG. 2B, the charge accumulation time is divided into a plurality of frames and allocated to the plurality of frames. Then, in S110, the display frame rate of the display unit 110 is set to the changed image capture frame rate. Also, in the first synthesis control unit 108 and the third synthesis control unit 112, as shown in FIG. 3B, the images of the plurality of frames are synthesized while being shifted sequentially, and are controlled to be output to the display unit 110 and the external display unit 210 at the display frame rate. This allows the display unit 110 and the external display unit 210 to display an image of the required charge accumulation time at a high display frame rate. Note that, if the display frame rate of the display unit 110 cannot be increased to the image capture frame rate, the display frame rate of the display unit 110 is switched to be as high as possible. Then, the images of the plurality of frames are added while being shifted, and output at the display frame rate.

Next, in S111, the second synthesis control unit 109 synthesizes an image of a number of frames according to the charge accumulation time, as shown in FIG. 3B, and outputs the images to the recording unit 111 at the default recording frame rate. This makes it possible to record an image with the same exposure as before the imaging frame rate was changed, at the recording frame rate.

In the control shown in FIG. 4, the process proceeds to S102 on the condition that the responsiveness of switching between frames on the external display unit 210 is determined to be fast in S101, but the present invention is not limited to this. For example, the process may proceed to S102 on the condition that the display frame rate of the set external display unit 210 or the display unit 110 is fast.

As described above, according to the first embodiment, imaging, display, and recording can be performed at a frame rate suitable for each device, and images with equivalent exposure can be displayed and recorded before and after changing the imaging frame rate.

Second Embodiment
Next, a second embodiment of the present invention will be described.
Fig. 5 is a block diagram showing an example of a schematic configuration of an image pickup device 200 according to an embodiment of the present invention. In Fig. 5, the same components as those in the image pickup device 100 shown in Fig. 1 are given the same reference numerals, and description thereof will be omitted. The difference from the configuration shown in Fig. 1 is that a first gain control unit 208 is provided instead of the first synthesis control unit 108, and a second gain control unit 209 is provided instead of the third synthesis control unit 112.

In the first embodiment described above, when one frame period (cycle) of the imaging frame rate is shorter than the charge accumulation time, the charge accumulation time is distributed so that charge accumulation is performed at equal timing in each frame. FIG. 6(a) shows the charge accumulation control in the first embodiment. Hereinafter, the charge accumulation time allocated to multiple frames is referred to as the "split accumulation time" to distinguish it from the charge accumulation time set by photometry or user instructions. Then, when generating a display image, multiple frames of images are synthesized while being shifted sequentially, so that the charge accumulation time of each display image is made equal before and after the change in the imaging frame rate, and control is performed so that there is no difference in the exposure of the display images.

However, with the above control, the timing for starting image readout for each frame (end of the split accumulation time) is the end position of each frame. In this case, because the split accumulation time At is shorter than the length of one frame, there is a period of time t during which no charge accumulation is performed between the start of the frame and the start of the split accumulation time (non-accumulation time t). There is concern that this non-accumulation time t occurring between frames may cause unnatural effects such as misalignment or blurring when photographing a fast-moving subject.

Therefore, in the second embodiment, when allocating charge accumulation time to multiple frames, the accumulation control unit 107 performs control so as to eliminate non-accumulation time t between composite frames.

Figure 6 (b) is a timing chart for explaining accumulation control in the second embodiment when one frame period of the changed imaging frame rate is shorter than the charge accumulation time. As shown in Figure 6 (b), the split accumulation time of the first frame is started at a timing that is shifted backward so that the end (readout start) timing of the split accumulation time of the first frame coincides with the start timing of the split accumulation time of the second frame. In other words, charge accumulation starts after a time 2t, which is the time for two frames minus the charge accumulation time 2At, has elapsed from the start of the first frame. In this way, charge accumulation can be performed in the same manner as normal charge accumulation, while preventing the non-accumulation time t that was between frames in the charge accumulation method of the first embodiment from being inserted between the first and second frames.

Next, the timing and method of generating a display image and a recording image from the images of each frame obtained by storing and reading out charge at the timing shown in FIG. 6(b) will be described with reference to FIG. 7(a). FIG. 7(a) shows, as an example, a case in which the changed display frame rate and imaging frame rate are 60p, the recording frame rate is 30p, and the charge storage time is longer than one frame period of the display frame rate and shorter than two frame periods.

As shown in FIG. 7(a), the recorded image is composed of two frames at a time and recorded at the recording frame rate. Here, the recording frame rate is 30p, so every two frames are composed by the second composition control unit 109 and recorded by the recording unit 111. In this second embodiment, charge accumulation and readout are performed as shown in FIG. 6(b), so that it is possible to reduce discomfort such as misalignment and blur when photographing a fast-moving subject.

On the other hand, if the display image is composed by shifting each frame as in the first embodiment, a display image will be generated in which the non-accumulation time 2t is sandwiched between the frames in which the images are composed, and a display image in which the non-accumulation time 2t is not sandwiched between the frames in which the images are composed. This will result in a noticeable sense of discomfort, such as misalignment and blur, when photographing a fast-moving subject.

Therefore, in the second embodiment, instead of synthesizing images of multiple frames, gain control is performed on the images obtained in each frame, so that the exposure of the displayed image does not change before and after the change in the image capture frame rate. In the example shown in FIG. 7(a), since the display frame rate is 60p, the first gain control unit 208 and the second gain control unit 209 perform gain-up control on the image read out at 60p, and output it to the display unit 110 and the external display unit 210. Note that the first gain control unit 208 and the second gain control unit 209 are switched to drive at the same frame rate as the display frame rate by the instruction of the frame rate switching control unit 106. This makes it possible to display an image with the same exposure before and after the change in the image capture frame rate for each frame. Note that the gain value applied to the image of each frame may be a value obtained by dividing the charge accumulation time 2At by the divided accumulation time of each frame.

In addition, while FIG. 7(a) shows the case where the recording frame rate is 30p, if the recording frame rate is lower (for example, 15p) and three or more frames of images need to be synthesized to maintain exposure, control is performed as follows. That is, if non-accumulation time t does not occur between frames of the imaging frame rate and multiple frames are allocated so that readout is performed at the end of a frame of the recording frame rate, for example, it will look like FIG. 7(b). In this case, too, since the length and timing of the divided accumulation time differ for each frame, control is performed so that exposure does not change by performing gain-up control according to the divided accumulation time of each frame on the image read out for display. Also, by bringing the charge accumulation time to the rear of the recording frame rate, if a frame to which the divided accumulation time is not allocated occurs, additional charge accumulation is performed to obtain a display image.

Next, the control of the frame rate, charge accumulation time, and image synthesis in the second embodiment will be described with reference to the flowchart in FIG. 8. Note that in the process shown in FIG. 8, the processes in S101 to S108 and S111 are similar to the processes shown in FIG. 4 described in the first embodiment, so the same step numbers are used and the description is omitted.

If it is determined in S106 that one frame period (cycle) of the changed imaging frame rate is shorter than the charge accumulation time, the process proceeds to S209. In S209, the accumulation control unit 107 starts reading out the accumulated charge at the end of each frame of the recording frame rate, and determines the start timing of the charge accumulation time so that no non-accumulation time occurs between frames of the imaging frame rate (FIG. 6(b)).

Next, in S210, it is determined whether or not there are frames at the imaging frame rate to which the divided accumulation time has not been allocated due to the slow recording frame rate, and if so, charge accumulation for display is performed in S211. This allows the image obtained in each frame to be displayed without interruption even when the recording frame rate is at a lower frame rate.

Next, in S212, the first gain control unit 208 and the second gain control unit 209 apply a gain to the image of each frame according to the divided accumulation time for each frame, and output the image to the display unit 110 and the external display unit 210 (FIG. 7(a)). This allows the display unit 110 and the external display unit 210 to display images with equivalent exposure before and after the change in the imaging frame rate at a high-speed display frame rate.

Then, in S111, multiple images captured for recording in each frame of the recording frame rate are composited and output to the recording unit 111 at the recording frame rate. This makes it possible to record images with equivalent exposure before and after the change in imaging frame rate at the recording frame rate.

As described above, the second embodiment achieves the same effect as the first embodiment, and also reduces the sense of incongruity that can be seen in images when a subject is moving at high speed.

<Modification>
Next, a modification of the second embodiment will be described.
In the second embodiment, as shown in FIG. 6(b), control is performed to bring the charge accumulation time to the end of each frame of the recording frame rate so that no non-accumulation time t occurs between frames of the imaging frame rate during the charge accumulation time spanning multiple frames.

In contrast, in this modified example, as shown in FIG. 9, the charge accumulation time is divided and allocated to multiple frames of the imaging frame rate so that the center of the charge accumulation time is the center of each frame of the recording frame rate.

Next, the timing and method for generating a display image and a recording image from the image obtained by storing and reading out charges at the timing shown in FIG. 9 will be described with reference to FIG. 10(a).

As shown in FIG. 10(a), the recorded image is composed of two frames at a time and recorded at the recording frame rate. Here, the imaging frame rate is 60p and the recording frame rate is 30p, so every two frames are composed by the second composition control unit 109 and recorded by the recording unit 111. As shown in FIG. 9, charge accumulation and readout are performed, so that it is possible to reduce discomfort such as misalignment and blur when photographing a fast-moving subject.

On the other hand, if the display image is synthesized while shifting each frame as in the first embodiment, the timing of the non-accumulation time t differs from frame to frame, resulting in noticeable misalignment and blurring when shooting a fast-moving subject.

In this modified example, instead of synthesizing images of multiple frames, gain control is performed on the images obtained in each frame, as in the second embodiment, so that the exposure of the displayed image does not change before and after changing the imaging frame rate. In the example shown in FIG. 10(a), since the display frame rate is 60p, the image read out at 60p is gain-up by the first gain control unit 208 and the second gain control unit 209, and is displayed every frame by the display unit 110 and the external display unit 210. The gain value applied to the image of each frame can be the charge accumulation time 2At divided by the divided accumulation time in each frame, and in the example shown in FIG. 10(a), the gain value is 2At/At=2.

In addition, while FIG. 10(a) shows the case where the recording frame rate is 30p, if the recording frame rate is lower (for example, 15p) and three or more frames of images need to be synthesized to maintain exposure, control is performed as follows. That is, if the charge accumulation time is divided and assigned to multiple frames of the imaging frame rate so that no non-accumulation time t occurs between frames of the imaging frame rate, it will look like FIG. 10(b), for example. In this case as well, since the length and timing of the divided accumulation time differ depending on the frame, the image to be displayed is controlled so that no change in exposure occurs by performing gain-up control according to the divided accumulation time of each frame on the image read out in each frame. Also, by assigning the charge accumulation time so that the center of the charge accumulation time coincides with the center of the recording frame rate, if a frame to which the divided accumulation time is not assigned occurs, additional charge accumulation is performed to obtain a display image.

Figure 11 is a flowchart showing control of the frame rate, charge accumulation, and image synthesis in a modified example. Note that in the process shown in Figure 11, instead of S209 in Figure 8, in S309, charge accumulation is started by allocating a divided accumulation time to multiple frames of the imaging frame rate so that the center of the charge accumulation time is aligned with the center of each frame period of the recording frame rate. The rest of the process is the same as the process shown in Figure 8, so the same step numbers are used and explanations are omitted.

As described above, the modified example can achieve the same effects as the second embodiment.

In the above-described embodiment, as shown in FIG. 1 and FIG. 5, frame rate control and image synthesis are performed by independent circuits. However, these configurations or parts thereof may be configured within the microcomputer 105.

In addition, the display frame rate of the display unit 110 can be switched between high and low speeds, and the above control may be performed when the default frame rate is low and the display frame rate is switched to high speed.

<Other embodiments>
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

101: lens, 102: dimming unit, 103: imaging unit, 104: image processing unit, 105: microcomputer, 106: frame rate switching control unit, 107: accumulation control unit, 108: first synthesis control unit, 109: second synthesis control unit, 110: display unit, 111: recording unit, 112: third synthesis control unit, 208: first gain control unit, 209: second gain control unit, 210: external display unit

Claims (14)

A setting means for setting a first charge accumulation time and an exposure ;
a control means for controlling an imaging frame rate for driving the imaging element, a display frame rate for the display means, and a recording frame rate for the recording means;
a driving means for controlling a second charge accumulation time in each frame of the imaging frame rate based on the first charge accumulation time and the imaging frame rate to drive the imaging element;
a first generating means for generating, when the display frame rate is equal to or lower than the imaging frame rate, a display image corresponding to the exposure set by the setting means at the display frame rate from an image output from the imaging element at the imaging frame rate, based on the imaging frame rate, the display frame rate , the first charge accumulation time , and the second charge accumulation time, and outputting the display image;
a second generating means for generating, when the recording frame rate is equal to or lower than the imaging frame rate , a recording image corresponding to the exposure set by the setting means at the recording frame rate from an image output from the imaging element at the imaging frame rate, based on the imaging frame rate, the recording frame rate , the first charge accumulation time , and the second charge accumulation time, and outputting the generated recording image;
An imaging device comprising : The imaging device according to claim 1, characterized in that, when the imaging frame rate is changed, the first generating means and the second generating means generate a display image and a recording image such that the display image and recording of each frame period have the same exposure before and after the change in the imaging frame rate. The imaging device according to claim 1 or 2, characterized in that, when the display frame rate is changed, the control means changes the imaging frame rate to the same frame rate as the display frame rate if the imaging element can be driven at the changed display frame rate. 4. The imaging device according to claim 1 , wherein when the first charge accumulation time is longer than a period of the imaging frame rate, the driving means divides the first charge accumulation time and allocates it to a plurality of frames to drive the imaging element. The second generating means is
when a period of the image capturing frame rate is equal to or longer than the first charge accumulation time and the recording frame rate is equal to the image capturing frame rate, an image output in each frame is directly used as the recording image;
when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the recording frame rate is slower than the imaging frame rate, an image output for each frame is thinned out in accordance with a ratio between the recording frame rate and the imaging frame rate to generate the recording image;
5. The imaging device according to claim 4 , wherein, when a period of the imaging frame rate is shorter than the first charge accumulation time and the recording frame rate is slower than the imaging frame rate, images of multiple frames are synthesized so that the sum of the second charge accumulation times assigned to each frame becomes the first charge accumulation time before division, thereby generating the recorded image. 6. The imaging device according to claim 4, wherein when the first charge accumulation time is longer than a period of the imaging frame rate, the driving means allocates the second charge accumulation time to the same timing of the plurality of frames. The first generating means is
when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is equal to the imaging frame rate, an image output in each frame is directly used as the display image;
when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is slower than the imaging frame rate, an image output in each frame is thinned out in accordance with a ratio between the display frame rate and the imaging frame rate to generate the display image;
7. The imaging device according to claim 6 , wherein, when a period of the imaging frame rate is shorter than the first charge accumulation time and the display frame rate is equal to or slower than the imaging frame rate, the display image is generated by sequentially synthesizing images of multiple frames so that the sum of the second charge accumulation times assigned to each frame becomes the first charge accumulation time before division. The imaging device described in claim 4 or 5, characterized in that when the period of the imaging frame rate is shorter than the first charge accumulation time and the period of the recording frame rate is longer than the first charge accumulation time, the driving means divides and allocates the first charge accumulation time so that the end timing of the first charge accumulation time coincides with the end timing of each frame of the recording frame rate and so that there is no time between the multiple frames when charge accumulation is not performed. The imaging device described in claim 4 or 5, characterized in that when the first charge accumulation time is longer than a period of the imaging frame rate and shorter than a period of the recording frame rate , the driving means divides and allocates the first charge accumulation time so that a center of the first charge accumulation time coincides with a center of the recording frame rate and so that no time is generated between the multiple frames during which charge accumulation is not performed. The first generating means is
when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is equal to the imaging frame rate, an image output in each frame is directly used as the display image;
when a period of the imaging frame rate is equal to or longer than the first charge accumulation time and the display frame rate is slower than the imaging frame rate, an image output in each frame is thinned out in accordance with a ratio between the display frame rate and the imaging frame rate to generate the display image;
10. The imaging device according to claim 8, wherein when a period of the imaging frame rate is shorter than the first charge accumulation time, the display image is generated by applying a gain to an image obtained from each frame in accordance with a ratio between the second charge accumulation time assigned to each frame and the first charge accumulation time before division. 11. The imaging device according to claim 8, wherein when allocating the first charge accumulation time to the plurality of frames, if a frame to which the second charge accumulation time is not allocated occurs, the driving means drives the imaging element to perform charge accumulation for a predetermined time in that frame. a setting step in which a setting means sets a first charge accumulation time and an exposure;
a control step in which a control means controls an imaging frame rate for driving the imaging element, a display frame rate of the display means, and a recording frame rate of the recording means;
a driving step of driving the image sensor by controlling a second charge accumulation time in each frame of the image capture frame rate based on the first charge accumulation time and the image capture frame rate;
a first generation step in which, when the display frame rate is equal to or lower than the imaging frame rate, a first generation means generates and outputs a display image corresponding to the exposure set in the setting step at the display frame rate from an image output from the imaging element at the imaging frame rate based on the imaging frame rate, the display frame rate, the first charge accumulation time, and the second charge accumulation time;
a second generation step in which, when the recording frame rate is equal to or lower than the imaging frame rate, a second generation step generates and outputs a recording image corresponding to the exposure set in the setting step at the recording frame rate from an image output from the imaging element at the imaging frame rate based on the imaging frame rate, the recording frame rate, the first charge accumulation time, and the second charge accumulation time;
13. A method for controlling an imaging apparatus comprising : A program for causing a computer to function as each of the means of the imaging apparatus according to any one of claims 1 to 11. A computer-readable storage medium storing the program according to claim 13.

Images