JP7580935B2 - Image processing device, control method for image processing device, and program - Google Patents

Description

The present invention relates to an imaging device that has a shooting assistance function for recapturing a subject when the subject moves out of the shooting angle of view.

Conventionally, imaging devices are known that have a shooting assist function that makes it easy to recapture a subject that has moved out of the narrow shooting angle of view, such as in telephoto shooting. For example, when a photographer starts shooting assist via an operating member in an attempt to recapture a subject that has moved out of the shooting angle of view, the camera automatically zooms out to the wide-angle side to capture the subject, and when the shooting assist is then released, the camera automatically zooms in to the telephoto side (framing assist).

Patent document 1 discloses a technology that recognizes the subject that the photographer intends to capture based on information about the gaze position of the photographer looking through the viewfinder. Patent document 2 discloses a technology that performs focus control based on the variation in gaze position and distance information.

JP 2004-8323 A Patent No. 5828070

However, when the subject moves out of the shooting angle and needs to be recaptured, the subject temporarily disappears, making it difficult to apply the techniques disclosed in Patent Documents 1 and 2.

The present invention aims to provide an image processing device, an imaging device, a control method for an image processing device, and a program that can easily recapture a subject that has fallen out of the shooting angle of view, thereby reducing lost shooting opportunities.

An image processing device as one aspect of the present invention has a display means for displaying an image captured via an optical system, a detection means for detecting the gaze position of a user looking at the display means, and a control means for changing the zoom position of the image displayed on the display means, wherein when an instruction to start assist control is received, the control means zooms out the image displayed on the display means from a first zoom position to a second zoom position which is wider than the first zoom position, and when an instruction to stop the assist control is received, the control means zooms in the zoomed-out image from the second zoom position to a third zoom position which is more telephoto than the second zoom position based on the gaze position detected by the detection means , and changes the speed of zooming in from the second zoom position to the third zoom position based on the gaze time at the gaze position .
Other objects and features of the present invention will be described in the following embodiments.

The present invention provides an image processing device, an imaging device, a control method for an image processing device, and a program that can easily recapture a subject that has moved out of the shooting angle of view, thereby reducing lost shooting opportunities.

FIG. 1 is a block diagram of an imaging device according to first to third embodiments. FIG. 2 is a cross-sectional view of an imaging device according to the first to third embodiments. 3 is an explanatory diagram of the principle of a gaze detection method in each embodiment. 5A to 5C are diagrams illustrating an eyeball image projected onto an eyeball imaging element and an output intensity of the eyeball imaging element in each embodiment. 1 is a flowchart of a gaze detection method in each embodiment. 4 is a flowchart of a control method according to the first embodiment. FIG. 2 is a schematic diagram of a display screen of a display unit according to the first embodiment. 10 is a flowchart of a control method according to a second embodiment. 13 is a flowchart of a control method according to a third embodiment. 13A and 13B are schematic diagrams of a display screen of a display unit in the second and third embodiments. FIG. 13 is a block diagram of an imaging apparatus according to a fourth embodiment. 13 is a flowchart of a control method according to the fourth embodiment. FIG. 13 is a schematic diagram of a display screen of a display unit according to a fourth embodiment. FIG. 13 is a schematic diagram illustrating the appearance of an imaging device according to a fifth embodiment. FIG. 13 is a block diagram showing the electrical configuration of an imaging apparatus according to a fifth embodiment. FIG. 13 is an explanatory diagram of a field of view within a finder in the fifth embodiment.

13A to 13C are diagrams illustrating changes in the display angle of view in the fifth embodiment. 13 is a flowchart of a shooting assistance process according to the fifth embodiment. 13 is a flowchart of a control method according to the fifth embodiment. FIG. 23 is a diagram showing a change in the display angle of view in the sixth embodiment. 13 is a flowchart of a control method according to a sixth embodiment. FIG. 23 is a diagram showing a change in the display angle of view in the seventh embodiment. 23 is a flowchart of a control method according to the seventh embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings.

(First embodiment)
First, an imaging device (image processing device) according to a first embodiment of the present invention will be described with reference to Figures 1 and 2. Figure 1 is a block diagram of a digital camera (imaging device) 100. Figure 2 is a cross-sectional view of the digital camera 100. Note that in Figures 1 and 2, corresponding parts are denoted by the same numbers.

In FIG. 1, the imaging optical unit 101 has an optical system (imaging optical system) including a plurality of lens groups including a focus lens and an anti-vibration lens, and an aperture. The imaging optical unit 101 also has an aperture control circuit 211 and a focus adjustment circuit 212. When taking a photograph, the imaging optical unit 101 performs focus adjustment by the focus adjustment circuit 212, exposure adjustment by the aperture control circuit 211, and blur correction, etc., and forms an optical image on the image sensor 102. The image sensor 102 has a photoelectric conversion function that converts an optical image into an electrical signal (analog image signal), and is composed of a CCD sensor, a CMOS sensor, etc. The A/D conversion unit 103 converts the analog image signal from the image sensor 102 into a digital image signal. The converted image data is input to the image processing unit 104 in the subsequent stage. The bus 116 is a system bus mainly for transmitting control signals for each block from the CPU 114, etc. The bus 117 is mainly a data bus for transferring image data.

The CPU 114 is composed of a microcomputer that controls the entire image processing device, and issues operation instructions to each functional block to execute various control processes. The CPU 114 also performs calculations required for various control processes. The CPU 114 controls the image sensor 102, image processing unit 104, data transfer unit 105, memory control unit 106, non-volatile memory control unit 108, recording media control unit 110, display control unit 112, and operation unit (operation means) 115 via the bus 116. The CPU 114 also controls the line of sight detection circuit 120, photometry circuit 202, autofocus detection circuit 203, and light source drive circuit 205 via the bus 116. The microcomputer that constitutes the CPU 114 realizes each process of this embodiment by executing a program recorded in the ROM 109. The CPU 114 also controls the lens and aperture of the imaging optical unit 101 to acquire information such as focal length. The CPU 114 also has an estimation means 114a and a control means 114b. The estimation means 114a estimates the photographer's gaze position on the display unit 113 using the gaze detection circuit 120 and the eye image sensor 121. The control means 114b changes the focal length of the imaging optical unit 101.

The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that transfer data. The DRAM (memory) 107 is a storage unit that stores data, and has a storage capacity sufficient to store a predetermined number of still images, a predetermined amount of moving images, audio data, constants for the operation of the CPU 114, programs, etc. The memory control unit 106 writes data to and reads data from the DRAM 107 in response to instructions from the CPU 114 or the data transfer unit 105. The non-volatile memory control unit 108 writes data to and reads data from the ROM (non-volatile memory) 109 in response to instructions from the CPU 114. The ROM 109 is an electrically erasable and recordable memory, and an EEPROM or the like is used. The ROM 109 stores constants, programs, etc. for the operation of the CPU 114.

The image processing unit 104 is composed of various image processing units and a buffer memory, and performs appropriate magnification chromatic aberration correction, development processing, noise reduction processing, geometric transformation, and enlargement/reduction processing (resizing processing) on the image data. The image processing unit 104 also includes an imaging correction unit that performs appropriate pixel correction, black level correction, shading correction, and scratch correction on the image data converted by the A/D converter 103.

The recording medium 111 is a recording medium such as an SD card, and is controlled by the recording media control unit 110 to record image data and read out the recorded data. The display unit (display means) 113 is made up of an LCD display and an electronic viewfinder, and is controlled by the display control unit 112 to display various image data (such as images captured via the imaging optical unit 101) and menu screens transferred from the image processing unit 104. In addition, before still image shooting and during video shooting, image data input from the A/D conversion unit 103 is processed in real time and displayed.

The operation unit 115, which is an operating means, includes switches, buttons, a touch panel, etc. that are operated by the photographer, and is used for operations such as turning the power on/off and the shutter on/off. It is also provided with a dedicated button for implementing the shooting assist function, and pressing this button starts the shooting assist function.

The light source drive circuit 205 drives the light source (illumination light source) 210 (210a, 210b) for illuminating the photographer's eyeball. The gaze detection circuit 120 detects the gaze direction based on the positional relationship between the eyeball imaged on the eyeball image sensor 121 and the image of the light source 210 due to the corneal reflection.

In FIG. 2, 1A is a photographing lens (lens device) in an interchangeable lens camera, and corresponds to the imaging optical unit 101 in FIG. 1. In this embodiment, for convenience, the photographing lens 1A is shown to have two lenses 2101 and 2102 inside, but in reality it is composed of many lenses. 1B is a camera housing unit (camera body). The image sensor 102 is disposed on the intended imaging plane of the photographing lens 1A of the digital camera 100. The eyepiece lens 12 is disposed to observe the subject image displayed on the display unit 113.

210a, 210b are light sources for illuminating the photographer's eyeball 14 to detect the gaze direction from the relationship between the pupil and the reflection image of the light source due to the corneal reflection. The light sources 210a, 210b are made of infrared light emitting diodes and are arranged around the eyepiece lens 12. The illuminated eyeball image and the image due to the corneal reflection of the light sources 210a, 210b pass through the eyepiece lens 12, are reflected by the light splitter 15, and are imaged by the light receiving lens 16 on the eyeball image sensor 121, which is a two-dimensional array of photoelectric elements such as CCDs. The light receiving lens 16 positions the pupil of the photographer's eyeball 14 and the eyeball image sensor 121 in a conjugate image-forming relationship. The CPU 114 controls the gaze detection circuit 120, and detects the gaze direction from the positional relationship between the eyeball imaged on the eyeball image sensor 121 and the image due to the corneal reflection of the light sources 210a, 210b using a predetermined algorithm described later.

2111 is an aperture provided in the photographic lens 1, 211 is an aperture control circuit, 213 is a lens drive motor, 214 is a lens drive member consisting of a drive gear, etc. 215 is a photocoupler that detects the rotation of a pulse plate 216 linked to the lens drive member 214 and transmits the rotation (rotation information) to the focus adjustment circuit 212. The focus adjustment circuit 212 drives the lens drive motor 213 by a predetermined amount based on the rotation information and the lens drive amount information from the camera side, and moves the photographic lens 1A to the in-focus position. 217 is a mount contact that serves as an interface between the photographic lens 1A and the camera body 1B. The operation unit 115 includes operation members such as a touch panel compatible liquid crystal, a photographing assist button, and a button-type cross key. The operation unit 115 is used for control by a photographing assist operation described later. In this embodiment, the digital camera (imaging device) 100 is a photographing lens (interchangeable lens) 1A that can be attached to and detached from a camera housing (camera body) 1B, but this embodiment can also be applied to an imaging device in which the camera body and the photographing lens are integrated.

Next, the gaze detection method in this embodiment will be described with reference to Figs. 3 to 5. Fig. 3 is an explanatory diagram of the principle of the gaze detection method, and an explanatory diagram of an optical system for performing gaze detection. In Fig. 2, light sources 210a and 210b are light sources such as light-emitting diodes that emit infrared light that is insensitive to the observer. The light sources 210a and 210b are arranged approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the observer's eyeball 14. A portion of the illumination light reflected by the eyeball 14 is focused by the light receiving lens 16 onto the eyeball image sensor 121.

Figure 4(a) is a schematic diagram of an eyeball image projected onto the eyeball image sensor 121, and Figure 4(b) is a diagram of the output intensity of the CCD in the eyeball image sensor 121. Figure 5 is a flowchart of a gaze detection routine (gaze detection method). The gaze detection routine of Figure 5 is executed by a gaze point position estimation means (estimation means) including the CPU 114 and the gaze detection circuit 120.

In FIG. 5, when the gaze detection routine starts, first in step S501, the CPU 114 controls the light source drive circuit 205 to drive the light sources 210a and 210b. The light sources 210a and 210b emit infrared light toward the observer's eyeball 14 according to the control of the CPU 114. The image of the observer's eyeball illuminated by the infrared light from the light sources 210a and 210b is formed on the eyeball image sensor 121 through the light receiving lens 16 and is photoelectrically converted by the eyeball image sensor 121, so that the eyeball image can be processed as an electrical signal (eyeball image signal). Next, in step S502, the eyeball image signal obtained from the eyeball image sensor 121 is sent to the CPU 114.

Next, in step S503, the CPU 114 obtains the coordinates of the corneal reflection images Pd, Pe of the light sources 210a, 210b and the point corresponding to the pupil center c shown in FIG. 3 based on the information of the eyeball image signal obtained in step S502. The infrared light emitted from the light sources 210a, 210b illuminates the cornea 142 of the observer's eyeball 14. At this time, the corneal reflection images Pd, Pe formed by a portion of the infrared light reflected by the surface of the cornea 142 are collected by the light receiving lens 16 and imaged on the eyeball image sensor 121 (points Pd', Pe' in the figure). Similarly, the light beams from the ends a, b of the pupil 141 are also imaged on the eyeball image sensor 121.

Figure 4(a) shows an example of a reflected image obtained from the eye image sensor 121. Figure 4(b) shows an example of luminance information obtained from the eye image sensor 121 in the region α of the image example in Figure 4(a). As shown in Figure 4(a), the horizontal direction is the X axis, and the vertical direction is the Y axis. At this time, the coordinates in the X-axis direction (horizontal direction) of the images Pd', Pe' formed by the corneal reflection images of the light sources 210a, 210b are Xd, Xe. Also, the coordinates in the X-axis direction of the images a', b' formed by the light beams from the ends (pupil ends a, b) of the pupil 14b are Xa, Xb, respectively. In the example of luminance information in Figure 4(b), a strong level of luminance is obtained at positions Xd, Xe corresponding to the images Pd', Pe' formed by the corneal reflection images of the light sources 210a, 210b. A low level of luminance is obtained in the area between coordinates Xa and Xb, except for positions Xd and Xe, which corresponds to the area of the pupil 141. In contrast, in areas with X coordinate values lower than Xa and areas with X coordinate values higher than Xb, which correspond to the area of the iris 143 outside the pupil 141, intermediate values of the two luminance levels mentioned above are obtained. Based on the information on the variation in luminance level with respect to the X coordinate position, it is possible to obtain the X coordinates Xd and Xe of the images Pd' and Pe' formed by the corneal reflection images of the light sources 210a and 210b, and the X coordinates Xa and Xb of the images a' and b' of the pupil edge.

In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the coordinate Xc of the point (assumed to be c') corresponding to the pupil center c imaged on the eyeball image sensor 121 can be expressed as Xc ≒ (Xa + Xb) / 2. This makes it possible to estimate the X coordinate of c' corresponding to the pupil center imaged on the eyeball image sensor 121, and the corneal reflection images Pd', Pe' of the light sources 210a, 210b.

Next, in step S504 in FIG. 5, the CPU 114 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be calculated essentially as a function of the distance (Xd-Xe) between the corneal reflection images Pd', Pe'.

Next, in step S505, the two-axis eyeball rotation angle (rotation angles θx, θy) is set so that the X-coordinate of the midpoint of the corneal reflection images Pd, Pe and the X-coordinate of the center of curvature O of the cornea 142 approximately coincide. Therefore, if the standard distance from the center of curvature O of the cornea 142 to the center c of the pupil 141 is Oc, the rotation angle θx of the optical axis of the eyeball 14 in the Z-X plane is expressed by the following equation (1).

θx=β*Oc*SINθx≒{(Xd+Xe)/2}-Xc… (1)
3 and 4 show an example of calculating the rotation angle θx when the observer's eyeball rotates in a plane perpendicular to the Y axis. The method for calculating the rotation angle θy when rotating within a plane is similar.

When the rotation angles θx, θy of the optical axis of the observer's eyeball 14 are calculated in step S505, the process proceeds to step S506. In step S506, the CPU 114 reads the correction coefficient data stored in the DRAM (memory) 107. Next, in step S507, the CPU 114 uses the rotation angles θx, θy to determine the position of the observer's gaze (position of the point of gaze (point of gaze)) on the display unit 113. If the position of the point of gaze is defined as coordinates (Hx, Hy) corresponding to the center c of the pupil 141 on the display unit 113, the coordinates Hx, Hy are expressed by the following equations (2) and (3), respectively.

Hx=m×(Ax×θx+Bx)…(2)
Hy=m×(Ay×θy+By)… (3)
In this case, the coefficient m is a constant determined by the configuration of the viewfinder optical system of the digital camera 100, that is, a conversion coefficient for converting the rotation angles θx and θy into position coordinates corresponding to the center c of the pupil 141 on the display unit 113. , are determined in advance and stored in the DRAM 107. Ax, Bx, Ay, and By are line-of-sight correction coefficients for correcting individual differences in the line of sight of the observer, and are obtained by performing a calibration operation described later. , are stored in the DRAM 107 before the gaze detection routine starts.

After calculating the coordinates (Hx, Hy) of the center c of the pupil 141 on the display unit 113 as described above, in step S508, the CPU 114 stores the calculated coordinates in the DRAM 107 and ends the gaze detection routine of FIG. 5. The CPU 114 also measures how long the gaze position remained in a certain area, and stores this in the DRAM 107 as gaze time.

In this embodiment, a method for acquiring the gaze point coordinates on the display element using the corneal reflection images of the light sources 210a and 210b is shown, but this is not limited to the above method, and a method for acquiring the eyeball rotation angle from a captured eyeball image may also be used.

Next, the control method in this embodiment will be described with reference to Figs. 6 and 7. Fig. 6 is a flowchart of the control method in this embodiment. Each step in Fig. 6 is mainly executed by the CPU 114. Fig. 7 is a schematic diagram of the display screen of the display unit 113 in this embodiment.

When the flow in FIG. 6 is started, the subject 1403 in FIG. 7(a) is displayed on the display unit 113. First, in step S601, the CPU 114 determines whether or not the shooting assist button on the operation unit 115 has been pressed and a shooting assist operation (shooting assist control) has started. If the shooting assist operation has not started, step S601 is repeated. On the other hand, if the shooting assist operation has started, the process proceeds to step S602.

In step S602, the CPU 114 controls the lens drive motor 213 to perform a zoom-out operation to a predetermined wide-angle angle of view as shown in FIG. 6B. That is, the control means 114b zooms out the image displayed on the display unit 113 from the first focal length to a second focal length that is wider than the first focal length. The focal length when performing the zoom-out operation can be set to a desired value by the photographer. 1412 in FIG. 6B indicates the field of view frame (photographing field of view) at the focal length at the start of the shooting assist control. The field of view frame 1412 of the focal length at the start of the shooting assist control becomes smaller in size as it is displayed on the display unit 113 according to the zoom-out operation. 1411 in FIG. 6B is a gaze position display frame that includes the gaze point position (gaze position) calculated by the gaze detection routine in FIG. 5. The gaze position display frame 1411 is displayed at the same size as the field of view frame 1412 of the focal length at the start of the shooting assist control. In FIG. 6(b), the photographer's gaze point is located around the subject 1402.

Next, in step S603, the CPU 114 determines whether the shooting assist button has been released and the shooting assist operation has been released (stopped). If the shooting assist operation has not been released, step S603 is repeated. On the other hand, if the shooting assist button has been released, the process proceeds to step S604. In step S604, the CPU 114 controls the lens drive motor 213 to perform a zoom-in operation to the maximum focal length at which the gaze position display frame 1411 is included within the angle of view, as shown in FIG. 6(c). That is, the control unit 114b zooms in the image zoomed out in step S602 (zoomed-out image) from the second focal length to a third focal length that is more telephoto than the second focal length, based on the gaze position estimated by the estimation unit 114a.

As described above, in this embodiment, the control means 114b zooms out the image displayed on the display unit 113 from the first focal length to the second focal length that is wider than the first focal length. Then, the control means 114b zooms in the zoomed-out image from the second focal length to the third focal length that is more telephoto than the second focal length, based on the gaze position estimated by the estimation means 114a.

Preferably, the control unit 114b zooms in the zoomed-out image to the third focal length within the range in which the gaze position is displayed on the display unit 113. Also preferably, the display unit 113 displays a frame (gaze position display frame 1411) that includes the gaze position on the display unit 113. More preferably, this frame is the same size as the field of view frame of the first focal length. Also preferably, when the control unit 114b starts the shooting assist control, it zooms out the image displayed on the display unit 113 from the first focal length to the second focal length. And when the control unit 114b stops the shooting assist operation, it zooms in the zoomed-out image from the second focal length to the third focal length. More preferably, when the control unit 114b receives a command to release the shooting assist control based on the operation of the operation unit 115 by the photographer, it stops the shooting assist control.

According to this embodiment, when the shooting assist operation is cancelled, a zoom-in operation is performed to a frame including the photographer's gaze point position. This prevents the subject to be photographed from moving out of the angle of view due to the zoom-in operation even when the centre point position moves due to an unexpected movement of the subject, or when the gaze point position moves to a different subject, etc., reducing the loss of shooting opportunities.

Second Embodiment
Next, a second embodiment of the present invention will be described with reference to Fig. 8 and Fig. 10. Fig. 8 is a flowchart of a control method in this embodiment. Fig. 10 is a schematic diagram of a display screen of a display unit 113 in this embodiment. Note that the configuration of the digital camera in this embodiment is similar to that of the digital camera 100 described in the first embodiment, and therefore description thereof will be omitted.

First, in step S801, the CPU 114 determines whether or not the shooting assist button on the operation unit 115 has been pressed and a shooting assist operation (shooting assist control) has started. If the shooting assist operation has not started, step S801 is repeated. On the other hand, if the shooting assist operation has started, the process proceeds to step S802. The CPU 114 also measures the gaze time at the gaze point position.

In step S802, the CPU 114 controls the lens drive motor 213 to perform a zoom-out operation to a predetermined wide-angle angle of view, as in the first embodiment. FIG. 8(a) shows an example of a screen displayed on the display unit 113 after the zoom-out operation. In FIG. 8(a), 1000 is the field of view frame of the focal length at the start of the shooting assist control, 1001 is the optical center position of the lens, and 1002 is the gaze position display frame that includes the gaze point position calculated by the gaze detection routine in FIG. 5.

Next, in step S803, the CPU 114 determines whether the field of view frame 1000 of the focal length at the start of the shooting assist control and the gaze position display frame 1002 overlap. In this embodiment, when the area of the overlapping region of the field of view frame 1000 and the gaze position display frame 1002 is 70% or more of the area of each frame, it is determined that the frames overlap, but this is not limited to this. FIG. 10B shows a state in which the field of view frame 1000 of the focal length at the start of the shooting assist control and the gaze position display frame 1002 overlap by the photographer panning the digital camera 100 to the right without changing the gaze position (the position of the gaze position display frame 1002). If the field of view frame 1000 and the gaze position display frame 1002 do not overlap in step S803, step S803 is repeated. On the other hand, if it is determined that the field of view frame 1000 and the gaze position display frame 1002 overlap, the process proceeds to step S804.

In step S804, the CPU 114 cancels (stops) the shooting assist operation. That is, the control unit 114b stops the shooting assist operation when it determines that the field of view frame 1000 of the first focal length and the frame 1002 containing the gaze position overlap. In addition to the condition for canceling the shooting assist operation, that is, that the field of view frame 1000 of the focal length at the start of shooting assist control and the gaze position display frame 1002 overlap, a determination condition may be added that the shooting assist button on the operation unit 115 has been released from being pressed. By adding this determination condition, the timing for canceling the shooting assist operation can be determined to be more in line with the photographer's intention.

Next, in step S805, the CPU 114 controls the lens drive motor 213 to perform a zoom-in operation to the focal length at the start of the shooting assist control as shown in FIG. 10(c). The CPU 114 may also control the driving speed of the lens drive motor 213 to increase the zoom-in speed after the shooting assist operation is released by increasing the driving speed of the lens drive motor 213 according to the length of the measured gaze time. That is, the control unit 114b may change the speed at which the zoom-in is performed from the second focal length to the third focal length based on the gaze time at the gaze position. By controlling the zoom-in speed, if the photographer clearly identifies the subject to be photographed and is concentrating, the gaze time becomes longer, making it possible to zoom in on the subject to be photographed quickly. On the other hand, if the subject to be photographed is not fixed and the gaze moves to surrounding subjects, etc., and the gaze time becomes shorter, the zoom-in to the focal length at the start of the shooting assist control becomes slower, allowing the photographer to observe the surrounding subjects for a longer period of time and to quickly respond to sudden changes in the subject.

According to this embodiment, it is possible to quickly determine the timing to cancel the shooting assist when the viewing angle frame 1000 of the focal length at the start of shooting assist control overlaps with the gaze position display frame 1002, thereby reducing the loss of shooting opportunities.

Third Embodiment
Next, a third embodiment of the present invention will be described with reference to Figs. 9 and 10. Fig. 9 is a flowchart of a control method in this embodiment. Fig. 10 is a schematic diagram of a display screen of a display unit 113 in this embodiment. Note that the configuration of the digital camera in this embodiment is similar to that of the digital camera 100 described in the first embodiment, and therefore a description thereof will be omitted. Also, the flowchart in Fig. 9 includes the same processes as those in the flowchart in Fig. 8 described in the second embodiment, and therefore a duplicate description of the same processes will be omitted.

Step S901 is the same as step S801 in FIG. 8. After determining in step S901 whether to start the capture assist operation, the process proceeds to step S902. In step S902, the CPU 114 starts measuring the time until the field of view frame 1000 of the focal length at the start of capture assist control overlaps with the gaze position display frame 1002.

The following steps S903 and S904 are similar to steps S802 and S803 in FIG. 8, respectively. If it is determined in step S904 that the focal length field of view frame 1000 at the start of the shooting assist control overlaps with the gaze position display frame 1002, the process proceeds to step S905. In step S905, the CPU 114 completes the time measurement started in step S902. The following step S906 is similar to step S804 in FIG. 8.

Next, in step S907, the CPU 114 controls the lens drive motor 213 to perform a zoom-in operation to the focal length at the start of the shooting assist control, as shown in FIG. 10(c). The CPU 114 also changes the drive speed of the lens drive motor 213 depending on the length of the measurement time until the viewing position display frame 1002 overlaps with the field of view frame 1000 of the focal length at the start of the shooting assist control (first focal length), and changes the zoom-in speed after the shooting assist operation is released. That is, if the measurement time is short, the zoom-in speed is increased, and if the measurement time is long, the zoom-in speed is decreased.

As described above, in this embodiment, the control unit 114b stops the shooting assist control when the gaze time at the gaze position exceeds a predetermined time. Preferably, the control unit 114b changes the speed of zooming in from the second focal length to the third focal length based on the time it takes for the distance between the field of view frame 1000 of the first focal length and the gaze position display frame 1002 to become smaller than a predetermined distance.

According to this embodiment, if the photographer clearly identifies the subject to be photographed and quickly points the digital camera 100 at it, it becomes possible to quickly zoom in on the subject to be photographed. On the other hand, if the subject to be photographed is not fixed and the digital camera 100 is pointed slowly, the zoom in to the focal length at the start of the photographing assist control becomes slower, and the surrounding subjects can be observed for a longer period of time. As a result, it becomes possible to quickly respond to a sudden change in subject, and the loss of photographic opportunities can be reduced.

(Fourth embodiment)
Next, an imaging device (image processing device) according to a fourth embodiment of the present invention will be described. The imaging device according to this embodiment has a sub-imaging unit that captures wide-angle images to assist framing, in addition to a main imaging unit for recording. Note that the configuration of the imaging device according to this embodiment and a part of the retained shooting area display process are the same as those of the first embodiment, and only the different parts will be described, and the description of the same parts will be omitted.

Figure 11 is a block diagram of a digital camera (imaging device) 500 in this embodiment. The same elements as those in the first embodiment are given the same symbols, and detailed description here is omitted. In Figure 11, the main camera imaging optical unit 501, which mainly performs telephoto shooting, is equipped with a plurality of lens groups including a focus lens and an anti-vibration lens, and an aperture. When shooting, the main camera imaging optical unit 501 performs focus adjustment by the focus adjustment circuit 5212, exposure adjustment by the aperture control circuit 5211, and blur correction, and forms an optical image on the main camera imaging element 502. The main camera imaging element 502 has a photoelectric conversion function that converts an optical image into an electrical signal (analog image signal), and is composed of a CCD sensor, a CMOS sensor, or the like. The main camera A/D conversion unit 503 converts the analog image signal from the main camera imaging element 502 into a digital image signal. The converted image data is input to the image processing unit 104 in the subsequent stage.

The digital camera 500 also includes a sub-camera imaging optical unit 504, a sub-camera image sensor 505, and a sub-camera A/D conversion unit 506, which have similar functions for the purpose of taking wide-angle photographs. The converted image data is input to the downstream image processing unit 104. The sub-camera imaging optical unit 504 includes a focus adjustment circuit 5214 and an aperture control circuit 5213, just like the main camera imaging optical unit 501. Note that in this embodiment, the elements following the image processing unit 104 are the same as those in the first embodiment described with reference to FIG. 1.

Next, the control method in this embodiment will be described with reference to Figures 12 and 13. Figure 12 is a flowchart of the control method in this embodiment. Figure 13 is a schematic diagram of the display screen of the display unit 113 in this embodiment.

First, in step S1201, the CPU 114 determines whether the shooting assist button on the operation unit 115 has been pressed and a shooting assist operation (shooting assist control) has started. If the shooting assist operation has not started, step S1201 is repeated. On the other hand, if the shooting assist operation has started, the process proceeds to step S1202.

In step S1202, the CPU 114 switches the display screen of the display unit 113 to an image (wide-angle image) captured by a sub-camera capable of capturing wider-angle images than the main camera, as shown in FIG. 13(a). In FIG. 13(a), 1303 indicates the capturing angle of view of the main camera, and 1302 indicates a gaze position display frame that includes the gaze point position calculated by the gaze detection routine in FIG. 5.

Next, in step S1203, the CPU 114 determines whether the main camera's shooting angle of view 1303 and the gaze position display frame 1302 overlap. In this embodiment, when the area of the overlapping area between the main camera's shooting angle of view 1303 and the gaze position display frame 1302 is 70% or more of the area of each frame, it is determined that the main camera's shooting angle of view 1303 and the gaze position display frame 1302 overlap, but this is not limited to this. FIG. 13B shows a state in which the main camera's shooting angle of view 1303 and the gaze position display frame 1302 overlap by the photographer panning the digital camera 100 to the right without changing the gaze position (the position of the gaze position display frame 1302). If the main camera's shooting angle of view 1303 and the gaze position display frame 1302 do not overlap in step S1203, step S1203 is repeated. On the other hand, if the main camera's shooting angle of view 1303 overlaps with the gaze position display frame 1302 in step S1203, the process proceeds to step S1204.

In step S1204, the CPU 114 cancels (stops) the shooting assist operation. In addition to the conditions for canceling the shooting assist operation, which are that the shooting angle of view 1303 of the main camera overlaps with the gaze position display frame 1002, a determination condition may be added that the shooting assist button on the operation unit 115 has been released from being pressed. By adding this determination condition, the timing for canceling the shooting assist operation can be determined more precisely according to the photographer's intention. Next, in step S1205, the CPU 114 switches the display screen of the display unit 113 to an image captured by the main camera (telephoto angle of view) which is more telephoto than the sub camera, as shown in FIG. 13(c).

As described above, in this embodiment, the imaging device 500 has a first imaging element (main camera imaging element 502) and a second imaging element (sub camera imaging element 505). The first imaging element photoelectrically converts an optical image formed through the first optical system (main camera imaging optical unit 501) to obtain a first image. The second imaging element photoelectrically converts an optical image formed through the second optical system (sub camera imaging optical unit 504) to obtain a second image with a wider angle than the first image. The control means 114b displays the second image on the display unit 113. Then, the control means 114b determines whether the field of view frame (main camera shooting field of view 1303) of the first image displayed superimposed on the second image overlaps with a frame (gazing position display frame 1302) containing the gaze position in the second image. When the field of view frame of the first image overlaps with the frame that contains the gaze position, the control means 114b switches the second image displayed on the display unit 113 to the first image.

According to this embodiment, it is possible to quickly determine the timing to cancel the shooting assist function when the shooting angle of view 1303 of the main camera overlaps with the gaze position display frame 1002, thereby reducing the loss of shooting opportunities.

Fifth Embodiment
Next, a fifth embodiment of the present invention will be described with reference to Figs. 14 to 19. Fig. 14 is a schematic external view of a digital camera (imaging device) 100 in this embodiment. Fig. 14(a) shows a front perspective view of the digital camera 100, and Fig. 14(b) shows a rear perspective view. In this embodiment, the digital camera 100 is configured to include a photographing lens (lens device) 1A and a camera housing section (camera body) 1B, as shown in Fig. 14(a). The camera housing section 1B is also provided with a release button 5 that constitutes a part of an operation section 115 that receives an image capturing operation from a photographer. As shown in Fig. 14(b), an eyepiece 12 is provided on the rear surface of the digital camera 100, through which the photographer peers into the display section 113. The digital camera 100 is also provided with an operation member α (touch panel compatible liquid crystal 41), an operation member β (lever type operation member 42), and an operation member γ (button type cross key 43) that are used for camera operation, which will be described later, as part of the operation section 115. The operation members α, β, and γ are used, for example, when manually controlling the movement of the estimated gaze frame position, which will be described later. Fig. 1 corresponds to a cross-sectional view of the digital camera 100 cut along the YZ plane defined by the Y axis and the Z axis in Fig. 14(a). Note that in Fig. 1 and Fig. 14, corresponding parts are denoted by the same numbers.

Figure 15 is a block diagram showing the electrical configuration of the digital camera 100. In Figure 15, the same components as those in Figure 1 are given the same numbers. The CPU 114 is connected to the line of sight detection circuit 120, the photometry circuit 202, the autofocus detection circuit 203, the signal input circuit 204, the display control unit 112, the light source drive circuit 205, the tracking circuit 207, the recognition circuit 208, and the image processing circuit 209. The CPU 114 also transmits signals to the focus adjustment circuit 212 and the aperture control circuit 211 arranged in the photographic lens 1A via the mount contacts 217. The memory unit 4 associated with the CPU 114 has a function of storing image signals from the image sensor 102 and the eye image sensor 121, and a function of storing line of sight correction data that corrects individual differences in the line of sight, which will be described later.

The gaze detection circuit 120 A/D converts the output resulting from the formation of an eyeball image from the eyeball image sensor 121 (CCD-EYE) and sends this image information to the CPU 114. The CPU 114 extracts each feature point of the eyeball image required for gaze detection according to a predetermined algorithm described below, and further calculates the photographer's gaze from the position of each feature point. Based on the signal obtained from the image sensor 102, which also functions as a photometry sensor, the photometry circuit 202 amplifies the luminance signal output corresponding to the brightness of the subject field, then performs logarithmic compression and A/D conversion, and sends it to the CPU 114 as subject field luminance information.

The auto focus detection circuit 203 A/D converts signal voltages from multiple pixels used for phase difference detection, which are included in the CCD in the image sensor 102, and sends the converted signal to the CPU 114. The CPU 114 calculates the distance to the subject corresponding to each focus detection point from the signals from the multiple pixels. This is a well-known technique known as image plane phase difference AF. In this embodiment, as an example, it is assumed that there are 180 focus detection points at positions on the image plane corresponding to the locations shown in the viewfinder field image in Figure 16.

The tracking circuit 207 is a circuit that inputs an image and tracks a subject under the control of the CPU 114, and transmits information on the tracking frame of the image information to the CPU 114. The tracking process is performed by, for example, determining the similarity between two images using SAD (Sum Of Absolute Difference). The tracking circuit 207 may also use a tracking process other than SAD. The recognition circuit 208 is a circuit that recognizes a subject from an input image, and performs, for example, face detection of a person or animal detection. The image processing circuit 209 is composed of various image processing units and a buffer memory, etc., and appropriately performs processes such as magnification chromatic aberration correction, development processing, noise reduction processing, geometric transformation, and resizing such as enlargement and reduction on the image data. In addition, the image processing circuit 209 is equipped with a correction unit, etc. that appropriately performs pixel correction, black level correction, shading correction, and scratch correction on the image data.

Switch SW1, which is turned on by the first stroke of the release button 5 and starts the camera's photometry, distance measurement, gaze detection, etc., and switch SW2, which is turned on by the second stroke of the release button and starts the release operation, are connected to the signal input circuit 204. The above signals are input to the signal input circuit 204 and sent to the CPU 114. Operation member α (touch panel compatible liquid crystal 41), operation member β (lever type operation member 42), and operation member γ (button type cross key 43) are configured to transmit their operation signals to the CPU 114, and in response, movement operation control of the estimated gaze frame position, which will be described later, is performed.

Figure 16 is an explanatory diagram of the viewfinder field of view, showing the state when the display unit 113 is operating. In Figure 16, 300 is a field of view mask, 400 is a focus detection area, and 4001 to 4180 indicate 180 ranging point targets superimposed on the through image shown on the display unit 113 at positions corresponding to multiple focus detection points on the imaging surface. Among these targets, the one corresponding to the current estimated gaze point position is displayed with a frame appearing, as in estimated gaze point A in the figure. The gaze detection method of this embodiment is as described in the first embodiment with reference to Figures 3 to 5.

The calibration process will be described with reference to FIG. 16. As described above, in the gaze detection routine, the rotation angles θx and θy of the eyeball are obtained from the eyeball image, and the gaze point position is estimated by performing a calculation to convert the pupil center position into a corresponding position on the display unit 113.

However, due to factors such as individual differences in the shape of the human eyeball, unless the photographer adjusts the values of the gaze correction coefficients Ax, Ay, Bx, and By to appropriate values, a discrepancy will occur between position B where the photographer is actually gazing and the calculated position of the estimated gaze point C, as shown in FIG. 16(b). In the above example, even though the photographer wants to gaze at the person at position B, the camera erroneously estimates that the person is gazing at the background, making it impossible to perform appropriate focus detection and adjustment. Therefore, before capturing an image with the camera, a calibration task must be performed to obtain the appropriate correction coefficient values for the user and store them in the camera.

Conventionally, calibration work is performed by highlighting multiple indices at different positions in the viewfinder field of view before imaging, as shown in Figure 16 (c), and having the observer look at the indices. A known technique is to perform a gaze point detection flow when gazing at each target, and to find the appropriate coefficient value from the calculated multiple estimated gaze point coordinates and the position of each target coordinate.

Next, the shooting assistance process will be described with reference to Figures 17 and 18.

Figure 17 is a diagram showing changes to the display angle of view in this embodiment, showing the case where the display angle of view is automatically zoomed out and in when shooting (when shooting assistance is performed). Figure 18 is a flowchart of the shooting assistance process. Each step in Figure 18 is mainly executed by the CPU 114.

800 in FIG. 17(a) is an image captured by the image sensor 102 in the case of electronic zoom, and in the case of optical zoom, it is a schematic representation of a shooting scene outside the angle of view captured by the image sensor 102. 801 is a display image displayed on the display unit 113. 802 represents a subject that the photographer wants to shoot. The dotted frame of 803 in FIG. 17(b) indicates a warning display, and indicates that the frame of the image displayed on the display unit 113 is displayed, for example, in a blinking red color. FIG. 17(a) is an example of shooting a sports day game, and shows a state in which the shooting subject 802 has fallen off the display unit 113 due to camera shake. FIG. 17(b) shows a state in which the shooting assist processing flow described later detects that the shooting subject 802 has been lost, and the outermost corner of the display is displayed in a blinking manner to inform the photographer in advance that the display angle of view should be changed to the wide-angle side. FIG. 17(c) shows a state in which the display angle of view is widened after the warning display in FIG. 17(b). In FIG. 17(c), the subject 802 has been found, but the angle of view is gradually widened until the subject is found. FIG. 17(d) shows the state in which the photographer subsequently returns the subject 802 to the center position of the display unit 113, thereby returning the angle of view to that of FIG. 17(a).

Next, a specific shooting assist process will be described with reference to FIG. 18. Note that in this embodiment, the zoom operation will be described as optical zoom, but this is not limited to this. When the zoom operation is performed as electronic zoom, the same operation can be achieved by cropping the image in the image processing circuit 209 and changing the angle of view. This is also true for the first to fourth embodiments described above.

When the shooting assistance process is started, the CPU 114 performs a process of determining whether or not the subject has been lost in step S100. The process of determining whether or not the subject has been lost in step S100 will be described later with reference to FIG. 19, but by performing this process, it is determined whether or not the subject has been lost. Next, in step S101, the CPU 114 determines whether or not the photographer has lost sight of the subject based on the result of the process of determining whether or not the photographer has lost sight of the subject. If it is determined that the photographer has lost sight of the subject, the process proceeds to step S102. On the other hand, if it is determined that the photographer has not lost sight of the subject (the photographer has captured the subject), the process ends without taking any action.

Next, in step S102, the CPU 114 determines whether a warning display mode is set, which is a mode for displaying or not displaying a warning when the angle of view is automatically changed by the shooting assist function, and is preset in the camera menu. If the warning display mode is set, the process proceeds to step S103. On the other hand, if the warning display mode is not set, the process proceeds to step S105.

In step S103, the CPU 114 displays a warning on the display unit 113 for a threshold (c) seconds, as shown in FIG. 17(b). Next, in step S104, the CPU 114 performs the process of determining whether the subject has been lost sight of again, as in step S100. Next, in step S105, the CPU 114 performs the same determination as in step S101, or determines whether the maximum angle of view has been reached. If it is determined that the subject has been lost, or that the maximum angle of view is reached, the process proceeds to step S106. On the other hand, if it is determined that the subject has not been lost (the photographer has captured the subject) and that the maximum angle of view is not reached, the process ends without doing anything.

In step S106, the CPU 114 controls the lens drive motor 213 to zoom out to display a predetermined angle of view, as shown in FIG. 17(c). Next, in step S107, the CPU 114 performs the process of determining whether the subject has been lost sight of again, as in step S100. Next, in step S108, the CPU 114 performs the same determination as in step S101. If it is determined that the subject has been lost sight of, the process returns to step S102. In other words, if the subject has been lost sight of, steps S102 to S106 are repeated, and the angle of view is gradually widened until the subject is found. This is to avoid the need to search for many subjects when the angle of view is widened to the maximum in one go. On the other hand, if it is determined that the subject has not been lost sight of, the process proceeds to step S109.

In step S109, the CPU 114 controls the lens drive motor 213 to zoom in to an angle of view that includes the gaze area. Next, in step S110, the CPU 114 determines whether the angle of view is the same as before zooming out. If it is determined that the angle of view has returned to the angle of view before zooming out, the process ends. The display form when the process ends is as shown in FIG. 17(d). On the other hand, if the angle of view is not the same as before zooming out, the process proceeds to step S111. In step S111, the CPU 114 determines whether the photographer panned the camera to capture the subject 802 in the center of the screen. If it is determined that the camera has been panned, the process returns to step S109. On the other hand, if it is not determined that the camera has been panned, the process proceeds to step S112. The determination in step S111 may be made based on whether the subject 802 has been positioned toward the center of the screen.

In step S112, the CPU 114 determines whether the angle of view has remained the same for at least the threshold (d) time, or whether the release button 5 has been pressed and a photographing operation has been performed. If the angle of view has remained the same for at least the threshold (d) time, or a photographing operation has been performed, the process ends. On the other hand, if the angle of view has not remained the same for at least the threshold (d) time, and a photographing operation has not been performed, the process returns to step S111. The display form when the process ends is as shown in FIG. 17(d).

Next, the process of determining whether or not a subject has been lost in the course of photography in this embodiment will be described with reference to FIG. 19. FIG. 19 is a flowchart of the control method (process of determining whether or not a subject has been lost in the course of photography) in this embodiment.

First, in step S200, the CPU 114 resets and then starts a timer for measuring the number of times the gaze point moves within a threshold (a) time. Next, in step S201, the CPU 114 determines whether or not a gaze point has been detected in the gaze detection routine. If a gaze point is detected, the process proceeds to step S202. On the other hand, if a gaze point is not detected, the process proceeds to step S203.

In step S202, the CPU 114 determines whether the gaze point determined in step S201 has moved within the threshold (e) seconds. If the gaze point has moved, the process proceeds to step S203. On the other hand, if the gaze point has not moved, the process proceeds to step S206.

In step S203, the CPU 114 determines whether or not it has been within the threshold (a) seconds since the first gaze point was detected. If it has been within the threshold (a) seconds since the first gaze point was detected, the process proceeds to step S204. On the other hand, if it has not been within the threshold (a) seconds since the first gaze point was detected, the process returns to step S200.

In step S204, the CPU 114 determines whether the gaze point has moved a threshold number (b) of times or more. If the gaze point has moved a threshold number (b) of times or more, the process proceeds to step S205. On the other hand, if the gaze point has not moved a threshold number (b) of times or more, the process returns to step S201. In step S205, the CPU 114 determines that the subject has been lost and ends the process. In step S206, the CPU 114 determines that a subject is being photographed and ends the process.

As described above, in the process of determining whether or not a subject has been lost during shooting in this embodiment, loss of sight is determined based on whether or not the gaze point has moved within a certain area within a certain specified time. According to this embodiment, it is possible to detect when a subject has been lost and reduce the possibility of missing the timing to shoot.

Sixth Embodiment
Next, a sixth embodiment of the present invention will be described with reference to Fig. 20 and Fig. 21. This embodiment differs from the fifth embodiment in that a gaze point and a result of recognition of a subject by the recognition circuit 208 are used.

20 is a diagram showing a change in the display angle of view in this embodiment, showing the same scene as in FIG. 17. The face frame 1000 in FIG. 20(a) shows a face frame for the face of the subject recognized by the recognition circuit 208. Also, FIG. 20(a) shows that a face frame is displayed on the subject of the display image 801, although no number is attached. Note that in this embodiment, a face frame is described, but a subject frame detected by the recognition circuit 208 may be displayed. The face frame 1002 in FIG. 20(b) shows the face frame that the photographer has already gazed upon with a dotted line, and the face frame 1001 is a face frame that the photographer is not gazed upon. Note that the gazed upon face frame 1002 is shown with a dotted line, but as long as the photographer can recognize the determination of the gazed upon frame and the ungaze upon frame, any display form may be used, for example, the gazed upon face frame may be erased. FIG. 20(c) is a diagram when the display angle of view is widened, similar to FIG. 17(c).

Next, the process of determining whether or not a subject has been lost in the course of shooting in this embodiment will be described with reference to FIG. 21. FIG. 21 is a flowchart of the control method (process of determining whether or not a subject has been lost in the course of shooting) in this embodiment.

First, in step S300, the CPU 114 displays a face or subject frame. Note that if a face or subject frame is already displayed before the start of the shooting assist process, step S300 does not need to be performed. Next, in step S301, the CPU 114 determines in step S302 the face or subject frame that is in the vicinity of the gaze point detected in the gaze detection routine. Next, in step S302, the CPU 114 determines whether or not the face or subject frame is being gazed at. Note that a face frame that is within a threshold (f) pixels from the center of the gazed frame is likely to be seen at the same time, so it is also possible to consider it as being gazed at. If the face or subject frame is being gazed at, the process proceeds to step S303. On the other hand, if the face or subject frame is not being gazed at, the process proceeds to step S304.

In step S303, the CPU 114 determines whether the gaze point has moved within the threshold (e) seconds. The threshold (e) may be changed depending on the size of the face or subject frame. If the gaze point has moved within the threshold (e) seconds, the process proceeds to step S304. On the other hand, if the gaze point has not moved within the threshold (e) seconds, the process proceeds to step S307.

In step S304, the CPU 114 changes the display form of the face or subject frame before the gaze point is moved, that is, the frame targeted in step S301, to the face frame 1002 in FIG. 20(b). Next, in step S305, the CPU 114 determines whether or not all of the faces or subject frames on the image displayed on the display unit 113 have been confirmed. The number of times to gaze at one face or subject frame may be changed depending on the size of the face or subject frame. This is to avoid the possibility that the photographer may have seen the subject once if it is small, but may have missed it even though it was actually a subject that was originally photographed. If all of the faces or subject frames on the display screen have been confirmed, the process proceeds to step S306. On the other hand, if there are faces or subject frames that have not been confirmed, the process returns to step S301. In step S306, the CPU 114 determines that the subject has been lost, and ends the process. In step S307, the CPU 114 determines that there is a subject to be photographed, and ends the process.

As described above, in the photographing subject loss judgment process of this embodiment, the subject loss is judged depending on whether or not the photographer is gazing at the subject frame. According to this embodiment, it is possible to detect the loss of sight of the judged subject and reduce the possibility of missing the timing to take a photograph. In addition, by displaying a face frame and making it easy to judge whether or not the photographer is gazing at the face frame, it is possible to prevent the photographer from having to check the same subject multiple times.

Seventh Embodiment
Next, a seventh embodiment of the present invention will be described with reference to Figures 22 and 23. This embodiment differs from the sixth embodiment in that the subject is a moving object.

Figure 22 is a diagram showing a change in the display angle of view in this embodiment. Figure 22 (a) shows the display screen of the display unit 113 when a certain time is captured when a racing car (subject) is moving from the upper right to the lower left. Frames are displayed at the four corners of each subject as the subject frame 1300 recognized by the recognition circuit 208. Figure 22 (b) shows a state in which the photographed subject 802 is determined not to exist in Figure 22 (a) and the angle of view of the image displayed on the display unit 113 is widened, and the photographed subject 802 has entered the angle of view as a result of the widening of the angle of view. In addition, the subject frame 1302 shows the face frame that the photographer has already gazed upon with a dotted line. Note that although the gazed upon subject frame 1302 is shown with a dotted line, any display form may be used as long as the photographer can recognize the determination of the gazed upon frame and the ungaze upon frame, and for example, the gazed upon frame may be erased. Like FIG. 17(d), FIG. 22(c) shows the angle of view in which the subject 802 fits completely or the angle of view before zooming out, in this case zoomed in to the angle of view of FIG. 22(a).

Next, the process of determining whether or not a subject has been lost in the course of shooting in this embodiment will be described with reference to FIG. 23. FIG. 23 is a flowchart of the control method (process of determining whether or not a subject has been lost in the course of shooting) in this embodiment.

First, in step S400, the CPU 114 displays a subject frame. Note that if the subject frame has been displayed before the start of the shooting assist process, step S400 does not need to be performed. Next, in step S401, the CPU 114 determines the subject near the gaze point detected in the gaze detection routine as the determination subject in step S402. Next, in step S402, the CPU 114 calculates the subject's motion vector (motion information) from the previous frame and the current frame using a block matching method or the like. Note that a different method may be used for the motion vector calculation. Next, in step S403, the CPU 114 calculates the subject's motion vector from the previous frame and the current frame of the image used in the gaze detection using the same method as step S402. Note that a different method may be used for the motion vector calculation.

Next, in step S404, the CPU 114 determines whether the motion vector of the subject calculated in step S402 is the same as the motion vector of the gaze point calculated in step S403. If the motion vector of the subject and the motion vector of the gaze point are the same, the process proceeds to step S405. On the other hand, if the motion vector of the subject and the motion vector of the gaze point are different, the process proceeds to step S406.

In step S405, the CPU 114 determines that the subject is gazing at a moving subject, and proceeds to step S407. In step S406, the CPU 114 determines that the subject is not gazing at a moving subject, and proceeds to step S408. In step S407, the CPU 114 determines whether the gaze point has moved within a threshold (e) seconds. Note that the threshold (e) may be changed depending on the size of the subject frame. If the gaze point has moved within the threshold (e) seconds, proceeds to step S408. On the other hand, if the gaze point has not moved within the threshold (e) seconds, proceeds to step S411.

In step S408, the CPU 114 changes the display form of the subject frame before the gaze point is moved, that is, the subject frame targeted in step S401, to the subject frame 1302 in FIG. 22(b). Next, in step S409, the CPU 114 determines whether or not all of the subject frames on the image displayed on the display unit 113 have been confirmed. The number of times to gaze at one subject frame may be changed depending on the size of the subject frame. This is to prevent the possibility that the photographer may see the subject once if it is small, but may miss it even though it is the subject that was originally photographed. If all of the subject frames on the display screen have been confirmed, proceed to step S410. On the other hand, if there is a subject frame that has not yet been confirmed, return to step S401. In step S410, the CPU 114 determines that the subject has been lost, and ends the process. In step S411, the CPU 114 determines that the photographed subject exists, and ends the process.

As described above, the process of determining whether or not the subject is being lost uses the motion vector of the subject frame and the motion vector of the gaze point to determine whether or not the subject is being watched (moving object), and determines whether or not the subject has been lost in response to movement of the gaze point. According to this embodiment, it is possible to detect whether or not the subject has been lost, thereby reducing the possibility of missing the timing to capture a photo.

As described above, in the fifth to seventh embodiments, the control means (CPU 114) changes the angle of view when it is determined that the gaze position has changed. Preferably, the control means changes the angle of view when it is determined that the gaze position has changed a threshold number of times or more within a threshold time. Also preferably, the control means changes the angle of view by cropping the image or by driving the optical system to change the imaging position.

In addition, in the fifth to seventh embodiments, when the control means determines that the gaze position has changed, the control means displays a warning on the display means (display unit 113). Preferably, when the control means determines that the gaze position has changed a threshold number of times or more within a threshold time, the control means displays a warning.

Preferably, the image processing device has a recognition means (recognition circuit 208) that recognizes an object from an image. The control means changes the angle of view when it is determined that the gaze position is the position of the object recognized by the recognition means and that the gaze position has been changed a threshold number of times or more within the threshold time. Also preferably, the control means displays a warning when it is determined that the gaze position is the position of the object recognized by the recognition means and that the gaze position has been changed a threshold number of times or more within the threshold time. Also preferably, the control means displays object recognition information on the display means for each of the first object and the second object recognized by the recognition means. Then, the control means changes the display method of the object recognition information when the gaze position is each of the positions of the first object and the second object recognized by the recognition means. More preferably, the control means changes the display method of the object recognition information for the first object when it is determined that the gaze time of the first object is equal to or less than the threshold and that the gaze position has moved from the first object to the second object. Also preferably, the control means determines whether or not the gaze position is the position of the subject recognized by the recognition means based on the movement information of the subject recognized by the recognition means and the movement information of the gaze position. That is, the control means determines that the gaze position is the position of the subject recognized by the recognition means when the degree of agreement between the movement information of the subject and the movement information of the gaze position is equal to or greater than a threshold value.

Other Embodiments
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The image processing device of each embodiment uses information on the gaze position as a condition for determining whether to deactivate the shooting assist function, and can perform zoom-in control to a focal length where the gaze position is included within the angle of view after the shooting assist function is deactivated. Therefore, according to each embodiment, it is possible to provide an image processing device, imaging device, control method, and program that can easily recapture a subject that has moved out of the shooting angle of view, thereby reducing the loss of shooting opportunities.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

100 Imaging device (image processing device)
113 Display section (display means)
114a Estimation means 114b Control means

Claims (8)

A display means for displaying an image captured via an optical system;
A detection means for detecting a gaze position of a user looking at the display means;
a control means for changing a zoom position of an image displayed on the display means,
The control means
When a command to start assist control is received, the image displayed on the display means is zoomed out from a first zoom position to a second zoom position that is wider than the first zoom position;
When a command to stop the assist control is received, the zoom-out image is zoomed in from the second zoom position to a third zoom position that is more telephoto than the second zoom position , based on the gaze position detected by the detection means;
2. An image processing device comprising: an image processing apparatus which changes a speed of zooming in from the second zoom position to the third zoom position based on a gaze time at the gaze position . The image processing device according to claim 1, characterized in that the control means zooms in the zoomed-out image to the third zoom position within a range in which the gaze position is displayed on the display means. The control means displays a frame including the gaze position on the display means,
3. The image processing device according to claim 1, wherein the frame has a size corresponding to an angle of view corresponding to the first zoom position. The image processing device according to any one of claims 1 to 3, characterized in that the control means determines that an instruction to stop the assist control has been issued when it is determined that the field of view frame of the first zoom position overlaps with a frame that includes the gaze position. The image processing device according to any one of claims 1 to 4, characterized in that the control means determines that an instruction to stop the assist control has been issued based on an operation of an operating means by a user. 6. The image processing device according to claim 1, wherein the control means changes a speed of zooming in from the second zoom position to the third zoom position based on a time required for a distance between a field of view frame corresponding to the first zoom position and a frame containing the gaze position to become smaller than a predetermined distance. A display means for displaying an image captured via an optical system;
A control method for an image processing device having a detection means for detecting a gaze position of a user looking at the display means, and a control means for changing a zoom position of an image displayed on the display means, comprising:
receiving an instruction to start assist control, zooming out the image displayed on the display means from a first zoom position to a second zoom position that is wider than the first zoom position;
receiving an instruction to stop the assist control, zooming in a zoomed-out image from the second zoom position to a third zoom position that is more telephoto than the second zoom position, based on the gaze position detected by the detection means;
and changing a speed of zooming in from the second zoom position to the third zoom position based on a gaze time at the gaze position . A program for causing a computer to execute the control method according to claim 7 .