JP7561535B2 - Imaging device, control method thereof, program, and storage medium - Google Patents

Description

The present invention relates to a technology for suppressing subject blur when photographing a moving subject in an imaging device.

Conventionally, imaging devices are known that perform control to suppress subject blur when determining imaging conditions for a subject.

In Patent Document 1, the motion vector of the subject and the motion vector of the background are calculated using multiple captured images, the movement of the subject is calculated from the motion vector of the subject, and the movement of the imaging device is calculated from the motion vector of the background, and the imaging conditions are set based on these.

Patent No. 4379918

However, in the technology disclosed in the above-mentioned Patent Document 1, the speed of the subject is calculated from the amount of movement of the subject between multiple images. Therefore, when there is a sudden change in speed, such as when an animal pounces on its prey, it is difficult to accurately calculate the change in speed. In such cases, subject blur cannot be completely suppressed.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an imaging device that can suppress subject blur when photographing a moving subject.

The imaging device of the present invention is characterized by comprising: an imaging means for imaging a subject that may move within a real space at a predetermined speed ; an estimation means for estimating, from an image of the subject imaged by the imaging means, the speed of the subject at the time of capturing a subsequent image, using a trained model created by machine learning that associates posture information of the specific subject with the speed of the specific subject after a predetermined time; and a control means for performing control to determine an aperture value, a shutter speed, and an ISO sensitivity as imaging conditions for capturing the subsequent images by the imaging means, based on the estimated speed of the subject estimated by the estimation means.

The present invention makes it possible to reduce subject blur when photographing a moving subject.

1 is a block diagram showing the configuration of a digital camera that is a first embodiment of an image capturing apparatus of the present invention. FIG. 2 is a diagram showing a block configuration of an estimation unit. FIG. 4 is a diagram showing an example of an image input to an estimation unit. FIG. 4 is a diagram showing an example of a program diagram corresponding to the moving speed of a subject. 6 is a flowchart showing an operation of switching a program chart based on the speed of a subject in the first embodiment. FIG. 6 is a flowchart showing an operation of switching a program chart during panning in the first embodiment. 5 is a program diagram for performing panning in the first embodiment. 10 is a flowchart showing an operation of switching a frame rate of a live view in the second embodiment. FIG. 13 is a diagram showing the structure of an image sensor according to a third embodiment. FIG. 13 is a circuit diagram of a unit pixel according to a third embodiment. FIG. 13 is a circuit diagram of a unit column circuit according to a third embodiment. FIG. 13 is a diagram showing the timing of accumulation and readout in HDR using the accumulation time switching method. 13 is a timing chart of a readout operation of the entire pixel area in HDR using the accumulation time switching method. 13 is a timing chart of a readout operation of the entire pixel region in gain switching type HDR. 11 is a flowchart showing an operation for switching an imaging operation of an HDR image depending on an estimated speed of a subject. 13 is a timing chart showing the operation of live view shooting and subject speed estimation in the fourth embodiment. FIG. 13 is a diagram showing an example of obtaining an estimated speed at an arbitrary timing by interpolating the estimated speed. 6A and 6B are diagrams showing control of accumulation time based on the subject speed at the start of accumulation. FIG. 13 is a diagram showing control of accumulation time based on the subject speed at the center of accumulation. FIG. 23 is a block diagram showing the configuration of an estimation unit according to the sixth embodiment. FIG. 23 is a diagram for explaining the learning of a learning model in the sixth embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

(First embodiment)
FIG. 1 is a block diagram showing the configuration of a digital camera 100 which is a first embodiment of an image capturing apparatus of the present invention.

In FIG. 1, the lens unit 101 forms an optical image of a subject on the image sensor 105. In addition, in the lens unit 101, zoom control, focus control, aperture control, etc. are performed by the lens drive device 102. The mechanical shutter 103 is controlled by the shutter drive device 104. The image sensor 105 is made up of a CCD or CMOS sensor, etc., and photoelectrically converts the subject image formed by the lens unit 101 to output an image signal.

The signal processing circuit 106 performs various correction processes, data compression processes, development processes, etc. on the image signal output from the image sensor 105. The signal processing circuit 106 also includes a gain unit 106a that performs digital gain processing to apply a gain to the image signal. The photometry unit 107 obtains the subject brightness from the image signal. The estimation unit 108 estimates and learns the subject's speed from the image signal at the time the subject is captured (a time a predetermined time after the present).

The control unit 109 performs various calculations and controls the entire digital camera 100. The motion sensor 110 includes an angular velocity sensor (gyro sensor) that detects the angular velocity of the motion of the digital camera 100, and an acceleration sensor that detects acceleration. The memory unit 111 temporarily stores image data. The interface unit (I/F unit) 112 is an interface for recording data to the recording medium 113 or reading data from the recording medium 113. The recording medium 113 is composed of a removable semiconductor memory or the like for recording or reading image data.

The display unit 114 displays various information and captured images, and includes an electronic viewfinder (EVF) and a liquid crystal display (LCD). The electronic viewfinder is provided with an eyepiece through which the image displayed on the EVF panel can be viewed. The proximity sensor 115 is disposed near the eyepiece of the EVF, and detects the proximity of an object to the eyepiece.

The operation unit 116 includes a power switch, a shutter switch, and a touch panel. The shutter switch includes an instruction switch SW1 that starts a shooting preparation operation such as autofocus (AF) when the shutter button is pressed halfway, and an instruction switch SW2 that starts imaging when the shutter button is pressed all the way. The touch panel is layered on the LCD, and detects touch operations by the user.

Figure 2 is a diagram showing the block configuration of the estimation unit 108. The estimation unit 108 is configured to include a speed estimation unit 201 and a correction unit 202. The speed estimation unit 201 estimates and outputs the speed of the subject in the image signal at the time of capturing an image for recording a predetermined time later, in response to an input of a single image signal held in the memory unit 111. The timing of capturing the image after the predetermined time is notified by the control unit 109.

This estimation of the subject's speed can be achieved by using well-known deep learning technology, etc. to machine-learn the association between the subject in the input image and the speed of the subject's next movement, and then using the created trained model. Therefore, by using an input image that shows signs of a sudden change in movement in advance, it is possible to estimate the speed even if there is a sudden change in movement.

Figure 3 is a diagram showing an example of an image input to the estimation unit 201. When an image of a standing dog as shown in Figure 3(a) is input to the speed estimation unit 201, the speed estimation unit 201 outputs an estimated speed of 0 km/h. When an image of a running dog as shown in Figure 3(b) is input to the speed estimation unit 201, the speed estimation unit 201 outputs an estimated speed of, for example, 30 km/h.

The correction unit 202 corrects the estimated speed output from the speed estimation unit 201 according to the size of the subject on the image plane in the input image. Specifically, if the size of the subject on the image plane is small, the speed on the image plane is corrected to be slower, and conversely, if the size of the subject on the image plane is large, the speed on the image plane is corrected to be faster. The correction unit 202 may also correct the estimated speed output from the speed estimation unit 201 using the distance to the subject and the imaging magnification of the lens unit 101, rather than the size of the subject on the image plane.

The correction unit 202 sends the corrected estimated speed of the subject to the control unit 109. The control unit 109 selects a corresponding program diagram according to the estimated speed of the subject, and determines imaging conditions such as shutter speed, ISO sensitivity, and aperture based on the EV value (Exposure value) obtained by the photometry unit 107. The control unit 109 determines the type of movement, whether non-moving, slow, medium, or fast, based on the estimated speed of the subject described above. Here, if the estimated speed of the subject is less than a threshold, it is treated as a non-moving subject.

Figure 4 shows program diagrams for each type of movement. Figure 4(a) shows an example of a program diagram for a non-moving subject, Figure 4(b) shows an example of a program diagram for a slow-moving subject, Figure 4(c) shows an example of a program diagram for a medium-moving subject, and Figure 4(d) shows an example of a program diagram for a fast-moving subject. All program diagrams have a maximum aperture (aperture with the largest aperture diameter) of F1.4 and a minimum aperture (aperture with the smallest aperture diameter) of F22. The ISO sensitivity can be set in the range of ISO 100 to ISO 1638400 by the gain unit 106a applying a gain according to the ISO sensitivity. Gain may also be applied within the image sensor 105.

When a subject is classified as a non-moving subject, the program diagram for a non-moving subject in FIG. 4(a) is selected. In the program diagram for a non-moving subject in FIG. 4(a), aperture and shutter speed are controlled according to the EV value measured by the photometry unit 107 within a specified luminance range (EV4 to EV20 in the diagram). In the maximum aperture region 301 and the minimum aperture region 302, control is performed only with the shutter speed.

When a subject is classified as a slow-moving subject, an extremely slow shutter speed would cause subject blur, so the program diagram for slow-moving subjects in FIG. 4(b) is selected. In the program diagram for slow-moving subjects in FIG. 4(b), the ISO sensitivity is fixed within a specified brightness range (EV8 to EV20 in the diagram) and aperture and shutter speed control is performed, just like the program diagram for non-moving subjects. The program diagram for slow-moving subjects differs from the program diagram for non-moving subjects in that in region 303, where the shutter speed is 1/30 seconds, the shutter speed is fixed and aperture and ISO sensitivity control is performed. In region 304, where the aperture is at maximum, only ISO sensitivity control is performed.

When a subject is classified as moving at a medium speed, it is necessary to further increase the shutter speed to prevent subject blur, and the program diagram for a medium-speed moving subject in FIG. 4(c) is selected. In the program diagram for a medium-speed moving subject in FIG. 4(c), the ISO sensitivity is fixed within a predetermined brightness range (EV12 to EV20 in the diagram) and aperture and shutter speed control are performed, as in the program diagram for a slow-speed moving subject. In region 305 where the shutter speed is 1/125 seconds, the shutter speed is fixed and aperture and ISO sensitivity control is performed. In region 306 where the aperture is at maximum, only ISO sensitivity control is performed.

When a subject is classified as a fast-moving subject, it is necessary to further increase the shutter speed to prevent subject blur, and the program diagram for fast-moving subjects in FIG. 4(d) is selected. In the program diagram for fast-moving subjects in FIG. 4(d), the ISO sensitivity is fixed within a predetermined brightness range (EV16 to EV20 in the diagram) and aperture and shutter speed control is performed, as in the program diagram for slow-moving subjects. In region 307 where the shutter speed is 1/500 seconds, the shutter speed is fixed and aperture and ISO sensitivity control is performed. In region 308 where the aperture is at maximum, only ISO sensitivity control is performed.

The following describes the operation of so-called live view photography, in which still images are captured while the image output from the image sensor 105 is displayed on the display unit 114 in P (program) mode, where the digital camera 100 itself determines the imaging conditions such as shutter speed, ISO sensitivity, and aperture.

In this embodiment, the estimation unit 108 estimates the subject's speed using pre-captured images during live view shooting, and the control unit 109 switches the program diagram that determines the imaging conditions according to the estimated speed.

Figure 5 is a flowchart showing the operation of switching the program diagram depending on the estimated speed of the subject. When the digital camera 100 is turned on, the operation of this flowchart starts, and live view shooting begins.

In step S401, the control unit 109 uses the signal processing circuit 106 to perform image processing such as various correction processes, data compression processes, and development processes on the image signal output from the image sensor 105. The processed image data is stored in the memory unit 111 and then displayed on the display unit 114. At this time, the image signal output from the image sensor 105 may be resized in the signal processing circuit 106 because it is used for estimating the subject speed in the estimation unit 108 and for display on the display unit 114. In addition, images that have been subjected to different image processing may be input separately to the display unit 114 and the estimation unit 108.

In step S402, the control unit 109 uses the speed estimation unit 201 to estimate the speed of the subject after a predetermined time for one image stored in the memory unit 111. The predetermined time here is the time until the imaging time of the next frame in live view shooting (live view operation).

In step S403, the control unit 109 uses the correction unit 202 to correct the estimated speed of the subject according to the size of the subject on the image plane.

In step S404, the control unit 109 classifies the estimated speed of the subject corrected by the correction unit 202 into high speed, medium speed, slow speed, and non-moving. If the subject is a high speed moving subject, the process proceeds to step S405; if the subject is a medium speed moving subject, the process proceeds to step S406; if the subject is a slow speed moving subject, the process proceeds to step S407; and if the subject is a non-moving subject, the process proceeds to step S408. In steps S405, S406, S407, and S408, the control unit 109 selects a program diagram corresponding to each subject speed (controls the imaging operation).

In step S409, the control unit 109 determines the imaging conditions, such as the shutter speed, ISO sensitivity, and aperture, based on the selected program diagram and the EV value determined by the photometry unit 107.

In step S410, the control unit 109 determines whether or not the release button has been pressed. If the release button has been pressed, the process proceeds to step S411, where an image for recording is captured under the imaging conditions determined in step S409. The captured image signal is subjected to signal processing in the signal processing circuit 106, and then stored in the memory unit 111, and is recorded on the recording medium 113 via the I/F unit 112. If the release button has not been pressed in step S410, the process returns to step S401, and the series of operations is repeated.

Note that if the live view frame rate is high and the time between frames is short, the subject speed estimation process may not be completed by the time of the next live view frame. In this case, the estimation process may be performed once every few frames, and the timing of the estimated subject speed may be the imaging time of the next frame or later.

The above describes the live view shooting operation when the shooting mode is P mode, but it can also be applied to Av (aperture priority) mode in which the photographer sets the aperture as desired, and Tv (shutter speed priority) mode in which the photographer sets the shutter speed as desired. In P mode, a program diagram is selected that limits the shutter speed based on the estimated subject speed output by the estimation unit 108 so as not to set a shutter speed that would cause subject blur.

In contrast, in Tv mode, the shutter speed can be set arbitrarily. In this case, in Tv mode, the shutter speed that the photographer can set is not limited, and if a shutter speed that would cause subject blur is set, a display may be displayed on the display unit 114 to notify the user that subject blur may occur. Also, restrictions may be placed so that a shutter speed that would cause subject blur cannot be set.

As described above, according to this embodiment, by estimating the speed of the subject when capturing an image a predetermined time later and selecting a program diagram that corresponds to that speed, it is possible to set capturing conditions that are less likely to cause subject blur.

Next, as a modified example of the first embodiment, we will explain a case where information from the motion sensor 110, which includes an angular velocity sensor and an acceleration sensor, is referenced to set the imaging conditions.

For example, when taking a picture of a running dog with a panning shot, the shutter speed is controlled not to be slow according to the flowchart of the first embodiment. However, in the case of panning, as shown in FIG. 6, the subject does not move on the image plane, and in addition, the background needs to move. Therefore, the shutter speed cannot be too fast. In other words, even if a moving subject is detected, it is necessary to prevent the program diagram of the moving subject from being selected. Therefore, in this modified example, when performing panning, the motion sensor 110 detects the movement of the digital camera 100 tracking the subject, and determines whether panning is being performed.

Figure 7 is a flowchart showing the operation of the digital camera 100, including the determination of panning. Steps S601 to S603 and steps S609 to S611 are similar to steps S401 to S403 and steps S409 to S411 in Figure 5, respectively, and therefore will not be described.

In step S604, the control unit 109 refers to the subject's speed information estimated in step S602 and the angular velocity information and acceleration information from the motion sensor 110 to calculate the speed of the digital camera 100 and the relative speed between the subject and the digital camera 100.

In step S605, the control unit 109 determines whether or not panning is being performed based on the speed of the digital camera 100 calculated in step S604 and the relative speed between the subject and the digital camera 100. For example, if the relative speed is less than a threshold and the speed of the digital camera 100 detected by the motion sensor 110 is equal to or greater than the threshold, it can be determined that panning is being performed in which the digital camera 100 is following the subject. In that case, the control unit 109 selects a program diagram for panning in step S606.

If the relative speed is equal to or greater than the threshold, subject blur or image blur due to camera shake may occur, so in step S607, a program line for a moving subject is selected based on the subject speed classification made in steps S602 and S603. This operation is similar to the operations in steps S404 to S407 in FIG. 5. If both the relative speed and the speed of the digital camera 100 are less than the threshold, subject blur or image blur due to camera shake is unlikely to occur, so the control unit 109 selects a program line for a non-moving subject in step S608.

Figure 8 shows a program diagram for panning. In the program diagram for panning, the ISO sensitivity is fixed within a specified luminance range (EV2 to EV16 in the diagram), and the aperture and shutter speed are controlled. If the shutter speed is too fast, the background will not blur, so in area 701, where the shutter speed is 1/125 seconds and the aperture is at its minimum, the shutter speed is fixed and only the ISO sensitivity is controlled.

On the other hand, when the shutter speed is slow, image blur occurs due to camera shake caused by panning during panning, so in area 702 where the shutter speed is 1/2 second and the aperture is at maximum aperture, the shutter speed is fixed and only the ISO sensitivity is controlled.

As described above, in the modified version of the first embodiment, even if a moving subject is detected while performing a panning shot, a program diagram for the panning shot can be selected, making it possible to obtain an image in which the background flows in relation to the subject.

Second Embodiment
In the first embodiment, the imaging conditions during imaging using live view are described, but in the present embodiment, the operation of live view itself is described. Live view imaging is an operation for capturing a still image while displaying an image output from the image sensor 105 on the display unit 114.

When capturing a moving subject in live view, if the output from the image sensor 105 is at a low frame rate, the display unit 114 will show the subject's movement frame by frame. For this reason, it is desirable that the frame rate of the output from the image sensor 105 be a high frame rate that can display the moving subject smoothly.

However, outputting an image signal from the image sensor 105 at a high frame rate increases the power consumption of the digital camera 100. Therefore, when capturing an image of a stationary subject using live view shooting, it is desirable for the frame rate of the output from the image sensor to be a low frame rate.

Figure 9 is a flowchart showing the operation of switching the live view frame rate according to the estimated speed of the subject. Steps S801 to S803 are similar to steps S401 to S403 in Figure 5, so the explanation will be omitted.

In step S804, the control unit 109 determines whether the estimated speed of the subject is equal to or greater than a predetermined threshold. If the estimated speed of the subject is equal to or greater than the threshold, the process proceeds to step S805, where a high frame rate drive operation for live view is selected. If the estimated speed of the subject is less than the threshold, the process proceeds to step S806, where a low frame rate drive operation for live view is selected.

In step S807, the control unit 109 determines whether or not an operation to end live view has been performed. If an operation to end live view has been performed, the control unit 109 ends live view in step S808. If an operation to end live view has not been performed in step S807, the control unit 109 returns to step S801 and repeats the series of operations.

As described above, in this embodiment, when a moving subject is present in the live view, it is possible to display it smoothly at a high frame rate, and when a non-moving subject is present, it is possible to reduce power consumption by displaying it at a low frame rate.

In this embodiment, two types of frame rates, high and low, have been described, but it is also possible to have multiple frame rates and select one according to the estimated speed of the subject.

Third Embodiment
A third embodiment will be described below. In this embodiment, switching of the driving method of the image sensor when acquiring an HDR signal (high dynamic range signal) in which image signals of low exposure and high exposure are combined will be described.

First, we will explain the pixel structure of an image sensor that can acquire an HDR signal. A known pixel structure that can acquire an HDR signal is one in which two adjacent pixels have color filters of the same color. Methods for acquiring an HDR image with this pixel structure include an accumulation time switching method that differentiates the accumulation times of two adjacent pixels of the same color, and a gain switching method that differentiates the gains.

Figure 10 is a diagram showing the structure of the image sensor 105A of this embodiment. The image sensor 105A is configured with a pixel region 900, a vertical control circuit 902, a horizontal control circuit 903, a timing generator (hereinafter, TG) 904, row selection switches 905 and 906, a vertical transfer line 907, a column circuit 909, and a signal addition unit 910.

In the pixel region 900, unit pixels 901 are arranged in a matrix. As described above, each unit pixel 901 is composed of two adjacent pixels having color filters of the same color. Therefore, two vertical transfer lines 907 are connected to each unit pixel 901. In addition, each vertical transfer line 907 is connected to one unit column circuit 908.

For ease of understanding, FIG. 10 shows an example in which 4×4 unit pixels 901 are shown in the pixel region 900, but in reality many more pixels are arranged. The vertical control circuit 902 selects the pixels in the pixel region 900 row by row using row selection switches 905 and 906. The pixel signals read from the unit pixels 901 via the vertical transfer lines 907 are converted into digital signals by the column circuit 909, which is composed of multiple unit column circuits 908, and input to the signal synthesis unit 910.

The signal synthesis unit 910 synthesizes low and high exposure signals read from two adjacent pixels of the same color filter among the unit pixels 901 to generate an HDR signal. For example, a synthesis method may be a method in which, when the high exposure signal is saturated, an HDR signal is generated using the low exposure signal. Note that any synthesis method can be used as a method for generating an HDR signal, and the synthesis method is not limited.

TG904 sends control signals to the vertical control circuit 902 and the horizontal control circuit 903 to output pixel signals for each pixel. In addition, TG904 also sends control signals to the signal synthesis unit 910.

Figure 11 is a diagram showing the structure of a unit pixel 901 with the same color filter. The unit pixel 901 is configured with photodiodes 1001 (PDa), 1002 (PDb), transfer transistors 1003, 1004, FD capacitance additional switches 1005, 1006, additional FD capacitances 1007, 1008, FDs 1009, 1010, reset transistors 1011, 1012, and amplification transistors 1013, 1014.

Figure 12 is a diagram showing the structure of the unit column circuit 908. The unit column circuit 908 is configured with a comparator 1101, a counter 1102, an N memory 1103, an S memory 1104, and a CDS circuit 1105.

The comparator 1101 compares the pixel signal read from the unit pixel 901 with a ramp signal ramp (not shown). When the magnitude relationship between the pixel signal and the ramp signal is inverted, the output of the comparator 1101 is also inverted. The counter 1102 performs a counting operation based on the input clock signal clk, and when the output of the comparator 1101 is inverted, it stops the counting operation and performs AD conversion.

After AD conversion, the counter 1102 outputs the count value it holds to the N memory 1103 if it is the AD conversion result of the reset level of the FD (floating diffusion section), or to the S memory 1104 if it is the AD conversion result of the pixel signal. The CDS circuit 1105 subtracts the value held in the N memory 1103 from the value held in the S memory 1104 to obtain a signal with the reset noise component removed from the pixel signal.

Figure 13 is a diagram showing the accumulation operation and readout operation when the accumulation time switching method is used as the HDR method. PDa 1001 is reset during period P1, and accumulation is performed during period P2. Then, a readout operation is performed during period P3. Meanwhile, PDb 1002 is reset during period P1, and accumulation is performed during period P4. Then, a readout operation is performed during period P5. The series of operations is the same for both PDa 1001 and PDb 1002, but the lengths of periods P2 and P4, which are periods for accumulation, are different. This makes it possible to obtain signals of low exposure and high exposure. By combining these signals in the signal combining unit 910, an HDR signal can be obtained.

When using the gain switching method to obtain an HDR signal, the signal FDa (see FIG. 11) is set to Lo and the signal FDb (see FIG. 11) is set to Hi. This increases the FD capacitance on the PDb1002 side, lowering the gain of the pixel signal of PDb1002 relative to PDa1001. This makes it possible to obtain low exposure and high exposure signals. The HDR signal can be obtained by combining these signals in the signal combining unit 910.

Note that reading at different gains may be performed by changing the ramp signal supplied to the comparator 1101 or by changing the gain of the preamplifier in the unit column circuit 908 (not shown) rather than by changing the FD capacitance.

Figure 14 shows the readout operation of the entire pixel area 900 when the accumulation time switching method is used to obtain an HDR signal.

At timing T1301, a reset operation is performed for both PDa1001 and PDb1002 in sequence for each row, and accumulation begins. Next, at timing T1302, readout of PDa1001 begins, and each row is read out until all rows have been read out. Next, at timing T1303, readout of PDb1002 begins, and each row is read out until all rows have been read out. The subsequent operations from timing T1304 to T1306 are the same as those from timing T1301 to T1303. Although the accumulation start timing is the same for both PDa1001 and PDb1002, the readout timing is different, and therefore the accumulation time is different.

Figure 15 is a diagram showing the readout operation of the entire pixel region 900 when the gain switching method is used to obtain an HDR signal. At timing T1401, a reset operation is performed for both PDa1001 and PDb1002 in sequence for each row, and accumulation begins. Next, at timing T1402, readout begins for both PDa1001 and PDb1002, and each row is read out until all rows have been read out. The subsequent operations from timing T1403 to T1404 are the same as those from timing T1401 to T1402. Because the accumulation start timing and readout timing are the same for both PDa1001 and PDb1002, the accumulation time is the same.

In the gain switching method, if the brightness difference within the plane is large, gradation may not be obtained in dark areas where the amount of light is extremely low, even if gain is applied to the pixel signal. On the other hand, with the accumulation time switching method, a high exposure signal is obtained by extending the accumulation time without applying gain. Therefore, it is possible to obtain gradation in dark areas with a high exposure signal compared to the gain switching method.

However, when a moving subject is present, the photodiode with the longer accumulation time will cause subject blur. Therefore, in the case of a moving subject, it is desirable to use the gain switching method, and in the case of a non-moving subject, it is desirable to use the accumulation time switching method.

Figure 16 is a flowchart showing the operation of switching the method for obtaining an HDR signal depending on the estimated speed of the subject. Steps S1501 to S1503 are similar to steps S401 to S403 in Figure 5, so their explanation will be omitted.

In step S1504, the control unit 109 determines whether the estimated speed of the subject is equal to or greater than a predetermined threshold. If the estimated speed of the subject is equal to or greater than the threshold, the process proceeds to step S1505, where the image sensor 105A is caused to operate using the gain switching method. If the estimated speed of the subject is less than the threshold, the process proceeds to step S1506, where the image sensor 105A is caused to operate using the accumulation time switching method.

In step S1507, the control unit 109 determines whether or not an operation to end image capture has been performed. If an operation to end image capture has been performed, the captured image is recorded and the process ends in step S1508. If an operation to end image capture has not been performed in step S1507, the process returns to step S1501 and the series of operations is repeated.

As described above, in this embodiment, the method for obtaining an HDR signal is switched based on the estimated speed of the subject at the time of image capture after a predetermined time. This makes it possible to suppress subject blurring more effectively than when the method for obtaining an HDR signal is switched based on the shooting conditions, etc. Also, it makes it possible to suppress subject blurring more effectively than when the method for obtaining an HDR signal is mechanically switched based on the exposure amount and accumulation time thresholds.
Fourth Embodiment
In the first to third embodiments, the subject speed of the next frame or the frame after that is estimated from the image captured by live view shooting, and the imaging operation is controlled based on the estimation result. However, since the photographer presses the release button at any timing, the time at which the subject speed is estimated at regular intervals in live view shooting may differ from the shooting time.

Figure 17 is a diagram showing the timing of the operations of live view shooting and subject speed estimation in the first to third embodiments. The operation in Figure 17 is a so-called live view shooting operation in which still image shooting is performed while images output from the image sensor 105 at regular intervals are displayed on the display unit 114. Here, image readout is started at regular intervals of timing T1702, timing T1705, and timing T1708.

At timing T1701, a reset operation is performed for each row in sequence, and accumulation begins. Next, at timing T1702, readout begins for each row in sequence, and at timing T1703, readout of all rows is completed. After that, the estimation unit 108 estimates the subject speed at timing T1705 using the image acquired at timing T1703.

The control unit 109 controls the accumulation time by controlling the start timing of the reset operation at timing T1704 based on the estimated speed of the subject. Note that since there may be cases where the accumulation time is long and timing T1704 for performing the reset operation occurs before the completion of estimation, a limit may be placed on the accumulation time so that timing T1704 occurs after the completion of estimation of the subject speed.

After a reset operation is performed at timing T1704 and a read operation is performed at timing T1705, the estimation unit 108 uses the image acquired at timing T1706 to estimate the subject speed at timing T1708, when the next frame of the live view is read. At this time, timing T1708 is controlled by the control unit 109 based on the subject speed estimation at timing T1705.

Next, a reset operation is performed at timing T1707, and a readout operation is performed at timing T1708. Here, an example is shown in which the estimated speed of the subject at timing T1708 is fast. Therefore, the control unit 109 performs control to delay the start of the reset operation at timing T1707 so that the accumulation time is shortened based on the estimated speed of the subject.

Next, the operation when the release button is pressed will be described. When the release button is pressed at timing T1709, the control unit 109 instructs the reset operation and start of accumulation at timing T1709, and the start of readout at timing T1710, and a still image is acquired. In this case, since the estimated speed of the subject at timing T1709 has not been obtained, the control unit 109 controls the accumulation time based on the estimated speed of the subject at timing T1708, which is immediately before timing T1709 when the release button is pressed.

Therefore, if the subject speed differs between timing T1708 when the estimated subject speed is estimated and timing T1709 when the release button is pressed, the accumulation time control will not be appropriate.

Figure 18 shows an example of obtaining an estimated speed at any timing by interpolating the estimated speed between the subject speed estimations performed at regular intervals after a predetermined time in this embodiment.

Figure 18(a) is a diagram showing the timing of the operation of estimating the subject speed after two different predetermined times from the acquired images and interpolating the estimated subject speed between the two times. Regarding Figure 18(a), the differences from Figure 17 will be explained.

In FIG. 17, the estimation unit 108 uses the image acquired at timing T1703 to estimate the subject speed at read start timing T1705 of the next frame in live view. In contrast, in FIG. 18, the estimation unit 108 uses the image acquired at timing Ta1803 to estimate the subject speed at read start timing Ta1805 of the next frame in live view, as well as at read start timing Ta1808 of the frame after that. In other words, whereas in FIG. 17 the subject speed is estimated only for the next frame, in FIG. 18 it is estimated for the next frame and the frame after that.

Furthermore, the estimation unit 108 interpolates the estimated speed of the subject between timing Ta1805 and timing Ta1808 from the estimated speed of the subject at the obtained timing Ta1805 and timing Ta1808. This interpolation of the estimated speed can be realized by using a known method such as linear interpolation. Similarly, the estimation unit 108 also estimates the speed of the subject at timing Ta1808 when the next frame of the live view is read out and at timing Ta1811 when the frame after that is read out for the image obtained at timing Ta1806, and interpolates between them. This interpolation process makes it possible to obtain the estimated speed of the subject at any timing.

Therefore, even if the release button is pressed at timing Ta1810, the control unit 109 can control the accumulation time based on the estimated speed of the subject at the interpolated timing Ta1810.

If the time required for speed estimation is short, it may be possible to interpolate with the speed estimation result performed in the previous frame.

This will be explained in detail using FIG. 18(a). Taking the example of the interpolation between timing Ta1808 and timing Ta1811, at timing Ta1806, the speeds at timing Ta1808 and timing Ta1811 are estimated from the image obtained at timing Ta1806, and interpolation is performed between them.

Here, assume that the time required for speed estimation is short, and that speed estimation for the next frame, timing Ta1811, using the image obtained at timing Ta1809, is completed before Ta1810, when the release button is pressed. In this case, the newly calculated estimated speed is used for the estimated speed at timing Ta1811, and the estimated speed at timing Ta1810 is calculated by interpolating with the estimated speed at timing Ta1808 calculated from timing Ta1806.

This allows the speed estimate for timing Ta1811, which is used for interpolation between timing Ta1809 and timing Ta1811, to be obtained using the image of the closer timing Ta1809. This makes it possible to improve the accuracy of the interpolated estimated speed at timing Ta1810 when the release button is pressed.

Figure 18 (b) is a timing diagram of the operation of estimating the subject speed and acceleration after a predetermined time from the acquired image, and interpolating the time until the next estimated subject speed is obtained.

The subject acceleration here refers to the amount of change in the subject's speed over time, and can be achieved by using a learning model that uses well-known deep learning technology to associate the subject in an input image with the acceleration of the subject's next movement through machine learning.

In FIG. 18(b), the estimation unit 108 uses the image acquired at timing Tb1803 to estimate the subject speed and subject acceleration at timing Tb1805 when the next frame of the live view is read out.

Similarly, the estimation unit 108 uses the images acquired at timing Tb1806 and timing Tb1809 to estimate the subject speed and subject acceleration at timing Tb1808 and timing Tb1811, respectively, when the next frame of the live view is read out.

Furthermore, the estimation unit 108 uses the estimated speed and estimated acceleration of the subject at timing Tb1805 to interpolate the estimated speed of the subject up to timing Tb1808 when the next estimation result is obtained. Similarly, the estimation unit 108 interpolates the estimated speed of the subject up to timing Tb1811 from the estimation result at timing Tb1808. This interpolation process makes it possible to obtain the estimated speed of the subject at any timing.

Therefore, even if the release button is pressed at timing Tb1810, the control unit 109 can control the accumulation time based on the estimated speed of the subject at the interpolated timing Tb1810.

As explained above, the subject speed after a given time is estimated at regular intervals, and the estimated speed at any timing can be obtained by interpolating the estimated speed during that time. This makes it possible to control the imaging operation based on the estimated subject speed at the time the photographer presses the release button.

Fifth Embodiment
In the first to fourth embodiments, the imaging operation is controlled using the estimated speed of the subject at or near the timing when the release button is pressed. However, when the accumulation time is controlled according to the estimated speed of the subject, the speed of the subject may change even after the release button is pressed and accumulation is started. For example, when the subject is accelerating rapidly, the estimated speed of the subject is significantly different between the start and end of accumulation.

In this embodiment, the accumulation time is controlled using the estimated subject speed at the center of accumulation (the center time of accumulation) rather than the estimated subject speed at or near the timing when the release button is pressed. If the accumulation time is controlled according to the subject speed at the center of accumulation rather than the subject speed at the start of accumulation, subject blurring can be more appropriately suppressed.

Figure 19 shows the control of accumulation time according to the subject speed at the start of accumulation in the fourth embodiment. Figure 19(a) is a flowchart for determining the accumulation time, and Figure 19(b) shows the relationship between the estimated subject speed and accumulation timing when following the flowchart in Figure 19(a).

In step S1901, the subject speed at any timing is estimated. This estimation can be performed by using interpolation based on the estimated speed between two frames shown in FIG. 18(a) or by using the subject acceleration shown in FIG. 18(b). This allows the subject estimated speed at timing t shown by curve V in FIG. 19(b) to be obtained.

When the release button is pressed in step S1902, the timing to start accumulation is determined taking into account the release time lag. Next, accumulation begins in step S1903.

In FIG. 19B, the release occurs at timing T1901, accumulation starts at timing T1902, the accumulation time is determined based on the subject speed Vs at that time, and accumulation ends at timing T1903.

In this case, the subject speeds at timings T1902 and T1903 are Vs and Ve, respectively, resulting in a speed difference. Therefore, when the subject accelerates as shown in FIG. 19(b), subject blur may not be sufficiently suppressed under the shooting conditions (accumulation time) determined from the subject speed at the start of accumulation.

Figure 20 shows the control of accumulation time using the subject speed at the center of accumulation in this embodiment. Figure 20(a) is a flowchart for determining the accumulation time, and Figure 20(b) shows the relationship between the estimated subject speed and accumulation timing when following the flowchart in Figure 20(a).

In step S2001, the subject speed at any given timing is estimated. This estimation can be performed by using interpolation based on the estimated speed between two frames as shown in FIG. 18(a), or by using the subject acceleration as shown in FIG. 18(b).

In step S2002, the accumulation time is determined based on the estimated speed of the subject for the timing t. Furthermore, based on the correspondence between the subject speed and the accumulation time, the estimated speed of the subject for the timing t is converted into the accumulation time for the timing t.

Next, in step S2003, the accumulation start timing for time t is calculated from the accumulation time determined so that time t is the accumulation center.

The control of accumulation time using the subject speed at the center of accumulation according to the flowchart in Figure 20(a) will be explained in detail using Figure 20(b).

In step S2001, the subject speed at any given timing is estimated and interpolated. This results in an estimated subject speed for timing t, as shown by curve Y in FIG. 20(b).

In step S2002, the accumulation time at time t is obtained from the estimated subject speed for time t.

In step S2003, the accumulation start timing and accumulation end timing are determined so that the accumulation time for timing t is symmetrical with respect to timing t. This results in the accumulation start timing based on the subject's estimated speed at the accumulation center shown by curve X in FIG. 20(b) and the accumulation end timing based on the subject's estimated speed at the accumulation center shown by curve Z.

In step S2004, when the accumulation start timing is determined by the release, the accumulation time is determined by curve X and curve Z.

An example will be described in which a certain timing T is set as the accumulation center. An accumulation time A is determined based on the estimated speed of the subject at timing T (step S2002). Since this timing T is set as the accumulation center, the accumulation start time is T-(A/2) and the accumulation end time is T+(A/2) (step S2003).

As a result, in step S2004, if the release button is pressed at timing T2001 and accumulation starts at T-(A/2), and the accumulation time is A, control will be based on the subject speed at the center of accumulation.

In other words, whereas the accumulation time was previously determined from the estimated speed of the subject, the estimated speed of the subject at time t is converted to the accumulation start timing using the relationship between the estimated speed of the subject and the accumulation time so that the accumulation time is determined from the accumulation start timing.

As described above, according to this embodiment, the accumulation time can be controlled according to the subject speed at the accumulation center, making it possible to more appropriately suppress subject blur, etc. Note that the subject speed used for control may be switched between the speed at the start of accumulation, the speed at the accumulation center, and the speed at the end of accumulation according to the acceleration/deceleration of the subject.

Sixth Embodiment
The configuration of the image capture apparatus of the sixth embodiment is similar to the configuration of the digital camera 100 of the first embodiment shown in FIG. 1, and therefore a description thereof will be omitted.

Fig. 21 is a block diagram showing the configuration of the estimation unit 108 shown in Fig. 1 in a sixth embodiment. The estimation unit 108 includes a speed estimation unit 2101, an estimation correction unit 2102, a specific subject detection unit 2111, a speed calculation unit 2112, a learning correction unit 2113, a learning model adjustment unit 2114, and a learning model storage unit 2121.

The speed estimation unit 2101 references the learning model stored in the learning model storage unit 2121 for one image signal output from the memory unit 111, and estimates and outputs the speed of the subject in the image signal at the time of image capture after a predetermined time. The timing of image capture after the predetermined time is notified by the control unit 109.

This estimation of the subject's speed can be achieved by using a learning model that uses well-known deep learning technology, etc., to associate the subject in the input image with the speed of the subject's next movement and machine-learn it. The learning model learns to associate one image of a specific subject with the speed of the specific subject after a specified time.

Each specific subject image contains posture information of the specific subject, and for example, the learning model is trained using deep learning technology to associate the posture information of the specific subject with the speed of the specific subject after a specified time. Therefore, the speed of sudden movement can be estimated by using an input image that shows signs of sudden movement in advance. The specific subject detection unit 2111, speed calculation unit 2112, and learning model adjustment unit 2114 are used to train the learning model, which will be described later.

For example, when an image of a standing dog as shown in FIG. 3(a) is input to the estimation unit 108, the speed estimation unit 2101 outputs an estimated speed of 0 km/h. When an image of a running dog as shown in FIG. 3(b) is input to the speed estimation unit 2101, the speed estimation unit 2101 outputs an estimated speed of 30 km/h.

The estimation correction unit 2102 corrects the estimated speed output from the speed estimation unit 2101 according to the size of the subject on the image plane in the input image. Specifically, if the size of the subject on the image plane is small, the speed on the image plane is corrected to be slower, and conversely, if the size of the subject on the image plane is large, the speed on the image plane is corrected to be faster.

The estimation correction unit 2102 may also correct the estimated speed output from the speed estimation unit 2101 using the distance to the subject and the imaging magnification of the lens unit 101, rather than the size of the subject on the image plane. The estimation correction unit 2102 may also correct the estimated speed output from the speed estimation unit 2101 using distance measurement information obtained by a known distance measurement technique.

The estimation correction unit 2102 sends the corrected estimated speed of the subject to the control unit 109. The control unit 109 selects a corresponding program diagram according to the estimated speed of the subject, and determines imaging conditions such as shutter speed, ISO sensitivity, and aperture based on the EV value (Exposure value) obtained by the photometry unit 107. The control unit 109 determines the type of movement, whether non-moving, slow, medium, or fast, based on the estimated speed of the subject described above. Here, if the estimated speed of the subject is less than a threshold, it is treated as a non-moving subject.

The program diagrams applied to the non-moving, slow-moving, medium-moving, and fast-moving subjects are the same as the program diagram in Figure 4 shown in the first embodiment.

Specifically, FIG. 4(a) shows an example of a program diagram for a non-moving subject, FIG. 4(b) shows an example of a program diagram for a slow-moving subject, FIG. 4(c) shows an example of a program diagram for a medium-moving subject, and FIG. 4(d) shows an example of a program diagram for a fast-moving subject. The program diagram in FIG. 4 and the method of selecting it are the same as in the first embodiment, so a description thereof will be omitted.

In addition, in the P (program) mode in which the imaging device determines imaging conditions such as the shutter speed, ISO sensitivity, and aperture, so-called live view shooting, in which still images are shot while the image output from the imaging element 105 is displayed on the display unit 114, is the same as the operation in the first embodiment shown in FIG. 5, so a description thereof will be omitted. In other words, in this embodiment as well, the estimation unit 108 estimates the subject speed using an image captured beforehand during live view shooting, and the control unit 109 selects and switches a program diagram for determining imaging conditions according to the estimated speed from the program diagram shown in FIG. 4.

Next, we will explain the learning of the learning model in the estimation unit 108. Figure 22 is a diagram that provides an overview of the learning of the learning model.

In FIG. 22, information 9000 is information stored in memory unit 111, and includes shooting time information (9011, 9012, ...) and images (9001, 9002, ...). The shooting time of image 9001 matches shooting time information 1 (9011), the shooting time of image 9002 matches shooting time information 2 (9012), the shooting time of image 9003 matches shooting time information 3 (9013), and the shooting time of image 9004 matches shooting time information 4 (9014).

Information 9020 is information held in the specific subject detection unit 2111, and includes information extracted regarding the position and size of a specific subject on the image plane. Position and size information 9021 corresponds to image 9001, position and size information 9022 corresponds to image 9002, position and size information 9023 corresponds to image 9003, and position and size information 9024 corresponds to image 9004.

Information 9030 is part of the information held in the speed calculation unit 2112 and includes speed time information. Information 9040 is part of the information held in the speed calculation unit 2112 and includes speed information. Speed time information 9031 corresponds to speed information 9041, speed time information 9032 corresponds to speed information 9042, speed time information 9033 corresponds to speed information 9043, and speed time information 9034 corresponds to speed information 9044. The dashed lines in FIG. 22 are dashed lines that show generally which information is calculated based on which information.

Below, the learning method of the learning model is explained with reference to Figure 22.

The memory unit 111 stores at least two images (9001, 9002, ...) taken at different times along with their respective shooting time information (9011, 9012, ...).

The specific subject detection unit 2111 reads images (9011, 9012, ...) from the memory unit 111, detects a specific subject (e.g., a specific dog) that has been registered in advance from the image, and identifies the detected position and size information (9021, 9022, ...). The position and size information of the identified subject is retained for a predetermined number of frames.

The speed calculation unit 2112 calculates and stores the subject speed and the corresponding time (speed time information) from the position information of the specific subject read from the specific subject detection unit 2111 and the shooting time information read from the memory unit 111. Specifically, it reads two pieces of position information (9021, 9022) from the specific subject detection unit 2111 and calculates the amount of movement.

The image capture time information (9011, 9012) corresponding to the above position information (9021, 9022) is read from the memory unit 111 to calculate the time difference. The calculated amount of movement is divided by the calculated time difference to calculate and store the speed 9041 of the specific subject at the average time 9031 of the image capture times 9011 and 9012.

The speed calculated above is the speed in a plane perpendicular to the straight line connecting the camera and the subject (hereinafter referred to as the angle of view direction). Note that it is also possible to detect multiple feature points (e.g., the eyes of a specific dog) in a specific subject for each image, and calculate the speed in the direction of the straight line connecting the imaging device and the subject (hereinafter referred to as the normal direction) from the amount of change in each feature point between images (e.g., the amount of change in the length between the eyes of a specific dog), and then calculate the speed of the subject by vector calculation with the speed in the angle of view direction.

The learning correction unit 2113 corrects and holds the subject speed output from the speed calculation unit 2112 based on the size on the image plane of the specific subject read from the specific subject detection unit 2111, in order to make the subject speed a speed associated with the posture of the subject independent of the size on the image plane of the subject in the image.

Specifically, if the size of the specific subject on the image plane is small, the input from the speed calculation unit 2112 is increased and held, and if the subject size on the image plane is large, the input from the speed calculation unit 2112 is decreased and held. The learning correction unit 2113 may also correct the estimated speed read out from the speed calculation unit 2112 using the distance to the subject and the imaging magnification of the lens unit 101, rather than the size of the subject on the image plane. The estimated speed read out from the speed calculation unit 2112 may also be corrected using the subject distance calculated using known imaging plane distance measurement technology or the like.

By repeatedly performing the above speed calculations, it is possible to calculate the speed 9042 at the average time 9032 of the shooting times 9012 and 9013, the speed 9043 at the average time 9033 of the shooting times 9013 and 9014, etc.

The learning model adjustment unit 2114 reads the image from the memory unit 111, and reads the speed after a predetermined delay time from the image capture time from the learning correction unit 2113. Then, the learning model stored in the learning model storage unit 2121 is trained using the read image information as learning data and the read speed as teacher data. For example, the image 9001 is used as learning data, and the speed information 9042 is used as teacher data.

By repeating the above operations, learning data can be created and learned to predict the speed after a specified delay time from the posture of a specific subject that has been registered in advance and is contained in a single image.

In addition, while it has been described that the position and size information of the specific subject is stored in the specific subject detection unit 2111, the speed information and speed time information are stored in the speed calculation unit 2112, and the corrected speed information is stored in the learning correction unit 2113, they may also be stored collectively in the memory unit 111.

The learning model storage unit 2121 may also have multiple learning models and may learn multiple predetermined delay times. In this way, it is possible to predict the speed after multiple predetermined delay times from a single image.

In addition, the speed calculation unit 2112 may calculate acceleration based on the speed time information (9031, 9032, ...) and the corresponding speed information (9041, 9042, ...) and learn by associating the image with the acceleration.

Specifically, the learning model storage unit 2121 stores an image-acceleration learning model separately from the learning model. The speed calculation unit 2112 calculates and stores the subject acceleration by dividing the difference (difference information) between two pieces of speed information (e.g., speed information 9041, 9042) by the difference (difference information) between two corresponding pieces of speed time information (e.g., speed time information 9031, 9032).

The learning correction unit 2113 and learning model adjustment unit 2114 are the same as in the case of learning by associating an image with a speed. In this way, it is possible to predict the acceleration after a specified delay time from a single image.

In addition, a general-purpose learning model (for example, a model trained on dogs in general) may be stored as the learning model when shipped from the factory, and each user may create a learning model specialized for a specific subject by training on the specific subject. In this way, even if there is insufficient training on the specific subject, it is possible to predict the speed after a specified delay time from a single image.

Although the learning of the learning model may be repeated all the time the power switch is on, it is preferable to repeat the learning when the shutter button is half-pressed and instruction switch SW1 is on, or when the shutter button is fully pressed and instruction switch SW2 is on. This increases the probability that the specific subject the user wants to capture is within the angle of view, allowing for efficient learning. As another example of the timing for learning the learning model, it may be when the eye detection sensor 115 is on, or for a predetermined time after a touch operation is performed on the touch panel.

In addition, when the camera is moving significantly due to camera shake or the like, it may not be possible to accurately determine the speed of the subject, and so if a signal from when the camera is moving significantly is used for learning, the learning model may become inaccurate. For this reason, when a signal indicating that the camera is moving significantly is output from the acceleration sensor in the motion sensor 110, the learning of the learning model may be stopped.

As explained above, by estimating the speed of the subject when capturing an image after a specified time and selecting a corresponding program diagram, it is possible to set imaging conditions that are less likely to cause subject blurring regardless of changes in the subject's state.

In this embodiment, a program diagram for panning is used when performing panning shots, but the operation when performing panning shots is similar to the operation shown in FIG. 7 of the first embodiment, so a description thereof will be omitted.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

105: Image sensor, 106: Signal processing circuit, 108: Estimation unit, 109: Control unit, 110: Motion sensor, 201: Speed estimation unit, 202: Correction unit

Claims (28)

An imaging means for imaging a subject that can move in a real space at a predetermined speed ;
an estimation means for estimating, from an image of the subject captured by the imaging means, a velocity of the subject at a time point when a subsequent image is captured, using a trained model created by machine learning that associates posture information of the specific subject with the velocity of the specific subject after a predetermined time;
a control means for performing control to determine an aperture value, a shutter speed, and an ISO sensitivity as imaging conditions for capturing the subsequent images in the imaging means, based on the estimated speed of the subject estimated by the estimation means;
An imaging device comprising: The imaging device according to claim 1, characterized in that the estimation means estimates the speed of the subject at the time of capturing the subsequent image from the image of the subject captured by the imaging means during live view operation. 3. The imaging device according to claim 1, wherein the control means controls the imaging conditions by switching a program diagram for determining an aperture value, a shutter speed, and an ISO sensitivity for capturing the subsequent images. The imaging device according to any one of claims 1 to 3, further comprising a correction means for correcting the estimated speed according to the size of the subject on the image plane. The imaging device according to any one of claims 1 to 4, further comprising an angular velocity sensor that detects the angular velocity of the imaging device and an acceleration sensor that detects acceleration, and the control means controls the imaging operation for capturing the subsequent images based further on the outputs of the angular velocity sensor and the acceleration sensor. The imaging device according to claim 5, characterized in that the control means calculates the relative velocity between the subject and the imaging device based on the estimated velocity and the velocity of the imaging device calculated based on the output of the angular velocity sensor and the output of the acceleration sensor. The imaging device according to claim 6, characterized in that the control means determines that panning is being performed when the relative speed is less than a first threshold and the speed of the imaging device is equal to or greater than a second threshold. The imaging device according to claim 7, characterized in that, when the control means determines that panning is being performed, the aperture value, shutter speed, and ISO sensitivity for capturing the subsequent images are determined using a program diagram for panning. The imaging device according to claim 1, characterized in that the control means controls the frame rate of the live view based on the estimated speed. The imaging device according to claim 9, characterized in that the control means controls the live view frame rate to be higher when the estimated speed is faster than when the estimated speed is slow. 2. The imaging apparatus according to claim 1, wherein the control means further switches a method for obtaining a high dynamic range signal based on the estimated speed. The imaging device according to claim 11, characterized in that the control means uses a gain switching method as a method for obtaining a high dynamic range signal when the estimated speed is equal to or greater than a third threshold, and uses an accumulation time switching method as a method for obtaining a high dynamic range signal when the estimated speed is less than the third threshold. The imaging device according to any one of claims 1 to 12, characterized in that the estimation means further estimates the acceleration of the subject. The imaging device according to any one of claims 1 to 13, characterized in that the estimation means estimates the speed of the subject at the center time of the accumulation time of the imaging means, and the control means controls the imaging operation based on the estimated speed at the center time of the accumulation time. The imaging device according to any one of claims 1 to 14, further comprising an extraction means for extracting a specific subject from at least two images captured by the imaging means at different times, and a calculation means for calculating the speed of the specific subject extracted by the extraction means, and the trained model further performs learning for estimating speed using the image of the specific subject extracted by the extraction means and the speed of the specific subject calculated by the calculation means. The imaging device according to claim 15, characterized in that the calculation means calculates the velocity of the specific subject in the angle of view direction from difference information of the position of the specific subject extracted by the extraction means from the at least two images, calculates the velocity of the specific subject in the normal direction from the movement amount of a plurality of feature points of the specific subject, and calculates the velocity of the specific subject using the velocity in the angle of view direction and the velocity in the normal direction. The imaging device according to claim 15 or 16, further comprising a second correction means for correcting the speed calculated by the calculation means. The imaging device according to claim 17, characterized in that the second correction means corrects the speed according to the size of the subject on the image plane. The imaging device according to claim 17, characterized in that the second correction means corrects the speed based on the imaging magnification of the lens used. The imaging device according to claim 17, characterized in that the second correction means corrects the speed based on distance measurement information by the imaging means. The imaging device according to any one of claims 15 to 20, further comprising a shutter button, and the trained model further performs training for estimating speed while the shutter button is half-pressed or fully pressed. The imaging device according to any one of claims 15 to 20, further comprising an electronic viewfinder and a proximity sensor, and the trained model further performs training for estimating speed while the proximity sensor detects the proximity of an object to the electronic viewfinder. The imaging device according to any one of claims 15 to 20, further comprising a touch panel, wherein the trained model further performs training for estimating speed for a certain period of time after a touch action on the touch panel is detected. The imaging device according to any one of claims 15 to 23, characterized in that the trained model does not perform training for estimating speed when the imaging device moves significantly. The imaging device according to any one of claims 1 to 24, characterized in that the estimation means further has a second trained model that estimates the acceleration of the subject after a predetermined time. A method for controlling an imaging device including an imaging unit that captures an image of a subject that can move in a real space at a predetermined speed, comprising the steps of:
an estimation step of estimating, from the image of the subject captured by the imaging means , the velocity of the subject at the time of capturing a subsequent image, using a trained model created by machine learning that associates posture information of the specific subject with the velocity of the specific subject after a predetermined time;
a control step of performing control to determine an aperture value, a shutter speed, and an ISO sensitivity as imaging conditions for capturing the subsequent images by the imaging means, based on the estimated speed of the subject estimated in the estimation step;
13. A method for controlling an imaging apparatus comprising: A program for causing a computer to execute each step of the control method described in claim 26. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method described in claim 26.

Images