JP7845352B2 - Information processing device, information processing method, information processing program - Google Patents

Description

This technology relates to information processing equipment, information processing methods, and information processing programs.

Some cameras are capable of autofocus control using the photographer's gaze. Furthermore, a technology has been proposed that switches the autofocus mode between continuous AF (Auto Focus), which continuously focuses on the subject, and one-shot AF, which fixes the focus position, depending on whether the pattern of the photographer's gaze trajectory matches the trajectory of the subject's movement (Patent Document 1).

Japanese Patent Publication No. 2017-034569

Traditional autofocus systems that rely on eye movement may fail to focus on the intended subject, or it may take a long time to focus on the intended subject.

For example, when shooting still images using continuous autofocus with a wide setting and relying on eye-tracking information to focus, the focus may shift to a subject unintended by the photographer, or if the photographer looks at a different location and then returns their gaze to the original location, the focus may not be locked, and it may take some time to lock on.

This technology was developed in consideration of these points, and aims to provide an information processing device, an information processing method, and an information processing program that can perform shooting in accordance with the photographer's intentions while giving the impression that the imaging device is understanding the photographer's intentions in response to the photographer's gaze movements.

To solve the above-mentioned problems, the first technology is an information processing device that determines the control target position based on reference line of sight information including the trajectory on the display unit of the imaging device and the latest line of sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting . The reference line of sight information includes the trajectory of the photographer's gaze toward the display unit during shooting prior to the time of shooting and the gaze position which is a position on the line of sight where the photographer's gaze was aligned for a predetermined time or longer. The control target position determination unit determines the gaze position as the control target position if part or all of the reference line of sight information and the latest line of sight information match, and furthermore, if the photographer's gaze in the trajectory of the latest line of sight information is directed toward the gaze position included in the reference line of sight information .

The second technology is an information processing method for determining the control target position based on reference line-of-sight information including the trajectory on the display unit of the imaging device and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting. The reference line-of-sight information includes the trajectory of the photographer's gaze toward the display unit during shooting prior to the time of shooting, and the gaze position, which is a position on the line-of-sight trajectory where the photographer's gaze was aligned for a predetermined period of time or longer. If the reference line-of-sight information and the latest line-of-sight information are partially or completely the same, and the gaze in the photographer's line-of-sight trajectory in the latest line-of-sight information is directed toward the gaze position included in the reference line-of-sight information, the gaze position is determined as the control target position .

Furthermore, the third technology is an information processing method that determines the control target position based on reference line of sight information including the trajectory on the display unit of the imaging device and the latest line of sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting. The reference line of sight information includes the trajectory of the photographer's gaze toward the display unit during shooting prior to the time of shooting and the gaze position, which is a position on the line of sight where the photographer's gaze was aligned for a predetermined period of time or longer. The information processing program causes a computer to execute an information processing method that determines the gaze position as the control target position if part or all of the reference line of sight information and the latest line of sight information coincide, and furthermore, the gaze in the photographer's line of sight trajectory in the latest line of sight information is directed toward the gaze position included in the reference line of sight information.

This is a block diagram showing the configuration of the imaging device 100. This is a block diagram showing the configuration of the information processing device 200. This flowchart shows the process for registering reference line-of-sight information. This is an explanatory diagram of the registration process for reference line-of-sight information. This is a flowchart showing the process for determining the position of the controlled object. This flowchart shows the process for determining the position of the controlled object in eye-line focus-fixed priority control. This flowchart shows the process for determining the position of the controlled object in eye-focus movement priority control. This flowchart shows the controls related to shooting. This figure shows a first specific example of gaze focus lock priority control. This figure shows a second specific example of gaze focus lock priority control. This figure shows a first specific example where gaze focus lock priority control is not performed. This figure shows a second specific example where gaze focus lock priority control is not performed. This figure shows a third specific example where gaze focus lock priority control is not performed. This figure shows a specific example of eye-focus movement priority control.

The embodiments of this technology will be described below with reference to the drawings. The description will be presented in the following order.
<1. Embodiments>
[1-1. Configuration of the imaging device 100]
[1-2. Configuration of the Information Processing Device 200]
[1-3. Processing and control by the information processing device 200]
[1-3-1. Registration process for eye-tracking information]
[1-3-2. Determination of the controlled object's position and image capture-related control]
[1-3-3. Specific Examples of Control]
<2. Variations>

<1. Embodiments>
[1-1. Configuration of the imaging device 100]
The configuration of the imaging device 100 will be described with reference to Figure 1. The imaging device 100 is composed of a control unit 101, an optical imaging system 102, a lens drive driver 103, an image sensor 104, an image signal processing unit 105, a camera processing unit 106, a storage unit 107, an interface 108, an input unit 109, a display unit 110, a microphone 111, a subject recognition unit 112, an AF processing unit 113, a gaze detection unit 114, and an information processing device 200.

The control unit 101 consists of a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory). The CPU controls the entire imaging device 100 and its individual parts by executing various processes and issuing commands according to the program stored in the ROM.

The optical imaging system 102 consists of an imaging lens for focusing light from the subject onto the image sensor 104, a drive mechanism for moving the imaging lens to focus and zoom, a shutter mechanism, an iris mechanism, and the like. These are driven based on control signals from the control unit 101 and the lens drive driver 103. The optical image of the subject obtained through the optical imaging system 102 is formed on the image sensor 104.

The lens drive driver 103, for example, is configured by a microcontroller and performs autofocus to focus on the target subject by moving the imaging lens by a predetermined amount along the optical axis direction based on focus control information supplied from the AF processing unit 113 or the information processing unit 200. It also controls the operation of the drive mechanism, shutter mechanism, iris mechanism, etc., of the optical imaging system 102 according to the control unit 101. This allows for adjustment of exposure, aperture value (F-number), and other settings.

The image sensor 104 converts the incident light from the subject obtained through the imaging lens into an electric charge and outputs an imaging signal. The image sensor 104 then outputs the pixel signal to the image signal processing unit 105. The image sensor 104 can be a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor), or the like.

The image sensor 104 has normal pixels such as R (Red), G (Green), and B (Blue) pixels, as well as phase-difference detection pixels for phase-difference detection. Phase-difference AF (Auto Focus) can be performed using the phase-difference information output from these phase-difference detection pixels. Note that the phase-difference detection pixels may function only as phase-difference detection pixels and not as normal pixels, or a single pixel may be composed of two independent photodiodes, allowing it to function for both imaging and phase-difference detection. Furthermore, phase-difference detection may be performed using a dedicated AF sensor.

The image signal processing unit 105 processes the imaging signal output from the image sensor 104, performing sample-and-hold (CDS) processing to maintain a good signal-to-noise ratio (S/N), AGC (Auto Gain Control) processing, A/D (Analog/Digital) conversion, and other operations to create an image signal. Furthermore, the image signal processing unit 105 applies recording processing to image signals intended for recording and display processing to image signals intended for display.

The camera processing unit 106 applies signal processing to the image signal, such as white balance adjustment, color correction, gamma correction, Y/C conversion, and AE (Auto Exposure) processing.

The storage unit 107 is, for example, a large-capacity storage medium such as a hard disk or flash memory. Video data and image data processed by the image signal processing unit 105 and the camera processing unit 106 are stored in a compressed or uncompressed state according to a predetermined standard. Furthermore, EXIF (Exchangeable Image File Format), which includes additional information such as information about the stored data, imaging location information indicating the imaging location, and imaging time information indicating the date and time of imaging, is also stored associated with the data.

Interface 108 is an interface between the system and other devices or the internet. Interface 108 may include wired or wireless communication interfaces. More specifically, wired or wireless communication interfaces may include cellular communication such as 3TTE, Wi-Fi, Bluetooth®, NFC (Near Field Communication), Ethernet®, HDMI® (High-Definition Multimedia Interface), USB (Universal Serial Bus), etc. Furthermore, if the imaging device 100 and the information processing device 200 are connected in hardware, interface 108 may include connection terminals between the devices and buses within the devices (hereinafter, these will also be referred to as internal device interfaces). Furthermore, if the imaging device 100 and the information processing device 200 are implemented in a distributed manner across multiple devices, interface 108 may include different types of interfaces for each device. For example, interface 108 may include both communication interfaces and internal device interfaces.

The input unit 109 is for the user to give various instructions to the imaging device 100. When the user provides input to the input unit 109, a control signal corresponding to that input is generated and supplied to the control unit 101. The control unit 101 then performs various processing corresponding to that control signal. The input unit 109 can include a shutter button for shutter input, physical buttons for various operations, a touch panel, and a touchscreen integrated with the display unit 110.

The display unit 110 is a display device such as an EVF (Electronic View Finder) that displays a through image, which is a display image signal that has been processed for display, an image/video that has been processed for recording and stored in the storage unit 107, a GUI (Graphical User Interface), etc.

The microphone 111 is a sound-collecting device used to record audio during video recording.

The subject recognition unit 112 recognizes a specific subject (such as a person's face or an object) from image data generated by the capture, using known subject recognition processing. Known subject recognition techniques may include template matching, matching based on the subject's brightness distribution information, methods based on skin-colored areas or human facial features in the image, and methods using artificial intelligence. Furthermore, these methods may be combined to improve recognition accuracy.

The AF processing unit 113 performs AF detection using phase difference information supplied from the image sensor 104 to determine the focus position and calculate the amount of defocus. Based on the amount of defocus, it generates focus control information indicating where within the field of view to focus on the target subject, and how much the lens drive driver 103 of the imaging device 100 should move the lens. The autofocus processing is then performed by supplying the focus control information to the lens drive driver 103.

Furthermore, the AF processing unit 113 acquires the photographer's gaze information from the gaze detection unit 114 and performs autofocus control (also referred to as gaze input AF, etc.) to automatically focus on the position where the photographer's gaze is directed on the display unit 110 detected by the gaze detection unit 114. Note that if the control related to shooting performed by the information processing device 200 is AF control, the shooting-related control unit 203 may perform AF control by controlling the AF processing unit 113, or the shooting-related control unit 203 itself may operate as the AF processing unit.

The gaze detection unit 114 consists of an infrared LED, an infrared-compatible image sensor, etc. It detects the photographer's gaze on the display unit 110 based on an image taken by illuminating the photographer's face with the infrared LED and capturing the photographer's eyeballs with the infrared-compatible image sensor. Based on the position of the pupil relative to the position of the corneal reflection of the eyeball, it can detect the direction of the photographer's gaze and determine which part of the display unit 110 the photographer is fixated on. The gaze detection unit 114 can detect the movement of the photographer's gaze by continuously detecting the photographer's gaze. The gaze detection unit 114 supplies the detected photographer's gaze to the information processing device 200 as coordinate data in a two-dimensional XY plane with a predetermined position on the display surface of the display unit 110 as the origin.

The imaging device 100 is configured as described above. The imaging device 100 may be a digital camera, SLR camera, camcorder, professional camera, professional-grade photographic equipment, or other device specializing in camera functions, as well as a smartphone, tablet, wearable device, or portable game console equipped with camera functions.

In the above description, the imaging device 100 is equipped with an AF function using image plane phase-detection AF, but other AF methods such as phase-detection AF and contrast AF may also be used.

[1-2. Configuration of the Information Processing Device 200]
Next, the configuration of the information processing device 200 will be described with reference to Figure 2. The information processing device 200 is composed of a gaze information management unit 201, a control target position determination unit 202, a shooting-related control unit 203, and a display control unit 204.

The gaze information management unit 201 analyzes coordinate data indicating the movement of the photographer's gaze toward the display unit 110, detected by the gaze detection unit 114, to generate gaze information. In predetermined cases, it registers, stores, and manages this gaze information as reference gaze information. The reference gaze information includes the trajectory of the photographer's gaze toward the display unit 110 in photographs taken prior to the most recent photograph. Furthermore, the reference gaze information includes the gaze position, which is the position on the photographer's gaze trajectory included in the reference gaze information where the photographer's gaze was aligned for a predetermined period of time or longer, and the gaze duration, which is the time the photographer's gaze was aligned with that gaze position. The gaze information includes the gaze trajectory, gaze position, and gaze duration. In addition, the gaze information management unit 201 also stores and manages coordinate data, the latest gaze information, etc. Note that the gaze information management unit 201 itself may be equipped with a storage medium to store the reference gaze information, coordinate data, and the latest gaze information, or it may process the data to be stored in the storage unit 107.

A trajectory is formed by connecting at least two coordinates, indicated by coordinate data, with line segments based on the order in which the coordinate data was detected, and represents the path of the line of sight's movement.

The gaze position is the position on the trajectory of the photographer's line of sight to the display unit 110 of the imaging device 100 where the gaze is fixed (stared at) for a predetermined period of time or longer.

The gaze duration is the length of time that the photographer focuses on a specific point along the trajectory of their gaze.

The gaze information management unit 201 may have its own storage medium and store the reference gaze information itself, or it may utilize the storage unit 107 provided by the imaging device 100 and process the data to store the reference gaze information in the storage unit 107.

The control target position determination unit 202 determines the control target position, which is the position within the field of view displayed on the display unit 110 that is subject to control by the shooting-related control unit 203.

The shooting-related control unit 203 performs various controls related to shooting, utilizing the control target position determined by the control target position determination unit 202. For example, if the control related to shooting is AF control, the shooting-related control unit 203 may perform AF control by controlling the AF processing unit 113 of the imaging device 100, or the shooting-related control unit 203 itself may operate as the AF processing unit.

In this embodiment, a first control and a second control will be described as controls that utilize the position of the target object. The first control is a gaze-focus-fixed priority control that keeps the focus fixed on the target object even when the photographer's gaze moves. The second control is a gaze-focus-movement priority control that, when the photographer's gaze moves, adjusts the focus to the target object located at the destination of the gaze.

In this embodiment, the shooting control that utilizes the gaze by operating the information processing device 200 in the imaging device 100 is referred to as the gaze-based mode. When taking a picture, the photographer must turn on the gaze-based mode to operate the information processing device 200, and furthermore, must pre-select and set whether to have the information processing device 200 execute the gaze-focus-fixed priority mode, which prioritizes gaze-focus-fixed control, or the gaze-focus-movement priority mode, which prioritizes gaze-focus-movement control. Note that the information processing device 200 can also automatically set whether to execute the gaze-focus-fixed priority mode or the gaze-focus-movement priority mode.

The display control unit 204 displays a user interface on the display unit 110 that corresponds to the processing by the shooting-related control unit 203, and presents information regarding the controlled position to the photographer within that user interface. This information regarding the controlled position includes an AF frame indicating the position where autofocus is correct. A specific example of the user interface will be described later.

The information processing device 200 is configured as described above. The information processing device 200 may also be configured as a dedicated device using hardware having that function. If the information processing device 200 is configured as hardware, the information processing device 200 may include a gaze detection unit 114.

Furthermore, the information processing device 200 may not only be configured as a dedicated device, but may also be configured by a program, and the imaging device 100 may function as the information processing device 200 by executing that program. The program may be pre-installed on the imaging device 100, or it may be distributed via download or storage media, allowing the user to install it on the imaging device 100 themselves.

[1-3. Processing and control by the information processing device 200]
[1-3-1. Registration process for eye-tracking information]
Next, referring to Figures 3 and 4, we will explain the process of registering gaze information, which shows the movement of the photographer's gaze, as reference gaze information. Figure 4 shows the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110.

As shown in Figure 3, first, in step S101, if the gaze-based mode in which the information processing device 200 operates is turned on in the imaging device 100, the process proceeds to step S102 (Yes in step S101). Next, in step S102, the gaze information management unit 201 queries the gaze detection unit 114 to obtain coordinate data indicating the movement of the photographer's gaze detected by the gaze detection unit 114. The gaze detection unit 114 detects coordinate data by continuously detecting the photographer's gaze. The gaze detection unit 114 may acquire coordinate data at a predetermined period and actively supply the coordinate data to the gaze information management unit 201 regardless of whether an query is made.

Next, in step S103, the gaze information management unit 201 analyzes the coordinate data. By analyzing the coordinate data, the trajectory of the gaze, the gaze position, and the gaze duration can be obtained. While the gaze detection unit 114 and the information processing device 200 are operating, coordinate data indicating the movement of the photographer's gaze is constantly detected, and the gaze information management unit 201 acquires gaze information by analyzing the coordinate data.

The system detects a gaze position as a position on the trajectory of the gaze information where the gaze is directed for a predetermined time, which is a first threshold, and further detects the length of time the gaze is directed at that gaze position as the gaze time. Since the gaze position is determined by the gaze time relative to a position on the trajectory, there may be one gaze position or multiple gaze positions on a single trajectory. If there are multiple gaze positions, the system may obtain a comparison result of the gaze times of the multiple gaze positions. The first threshold is a predetermined length of time for detecting a gaze position. The first threshold may be set by default in the information processing device 200, set by the photographer, pre-set by default, or automatically set according to the shooting settings, etc.

Next, in step S104, the gaze information management unit 201 stores the gaze information and coordinate data. Alternatively, the gaze information management unit 201 may itself have a storage medium and store the gaze information and coordinate data, or it may perform processing to cause the storage unit 107 to store the gaze information and coordinate data.

Next, in step S105, the gaze information management unit 201 checks whether the gaze information stored in step S104, namely the trajectory, gaze position, and gaze time, has been repeated with a predetermined frequency. The predetermined frequency is, for example, twice, but is not limited to a specific frequency. If it has been repeated with the predetermined frequency, the process proceeds to step S106 (Yes in step S105). Note that the predetermined frequency of repetition may be continuous or discontinuous, and the trajectory may be a reciprocating motion or a non-reciprocating motion.

Then, in step S106, the gaze information management unit 201 registers and stores gaze information, which consists of the trajectory of the gaze, the gaze position, and the gaze duration repeated at a predetermined frequency, as reference gaze information. However, it is not necessary to store coordinate data together with the reference gaze information. In the following description, the process of registering gaze information as reference gaze information includes the process of storing that reference gaze information.

Reference gaze information may be stored in the storage unit 107 of the imaging device 100 through storage control by the gaze information management unit 201, or the information processing device 200 may be equipped with a storage unit for such storage. Any storage medium or storage method is acceptable as long as it can be used in processing after being stored in either the imaging device 100 or the information processing device 200.

If the trajectory is a non-reciprocating motion, for example, after the photographer moves their gaze toward the display unit 110 of the imaging device 100, they move their gaze away from the field of view displayed on the display unit 110, and then when the photographer returns their gaze to the field of view and the same movement occurs, that gaze information is registered as reference gaze information.

On the other hand, if the gaze information in step S105— namely the trajectory, gaze position, and gaze duration—is not repeated at a predetermined frequency, the process returns to step S101 (No. in step S105).

Then, steps S101 to S106 are repeated until it is confirmed in step S101 that the gaze-gazing mode is off, and the reference gaze information is stored.

The gaze information management unit 201 may, after registering reference gaze information, register the different gaze information as new reference gaze information if the gaze usage mode remains on and the photographer repeats gaze movements indicated by different gaze information at a predetermined frequency, while retaining the already registered reference gaze information. In this case, the number of reference gaze information will increase. Furthermore, even if the gaze information is different, it may be registered as new reference gaze information only if certain conditions are met, such as when the controlled target position is the same or when part of the gaze information is the same. Alternatively, the reference gaze information may be deleted and the different gaze information may be registered as new reference gaze information.

This is how the eye-tracking information registration process is performed. Note that the photographer may register eye-tracking information as reference eye-tracking information in advance before shooting, or they may have the eye-tracking information at the time of shooting registered as reference eye-tracking information.

The gaze information management unit 201 always performs the process shown in Figure 3 when acquiring reference gaze information from coordinate data indicating the movement of the photographer's gaze. Therefore, even after registering reference gaze information, gaze information is acquired from new coordinate data obtained from the gaze detection unit 114, and when the trajectory, gaze position, and gaze time of that gaze information are repeated at a predetermined frequency, it is registered as reference gaze information. In other words, even after the registration of reference gaze information, new reference gaze information will be registered. When registering new reference gaze information, the previous reference gaze information may be retained as history or deleted.

Figure 4 shows a specific example of reference gaze information registered by the gaze information management unit 201 described in Figure 3. Figure 4 shows the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110.

First, let's assume that the photographer's line of sight, as shown in Figure 4A, is focused on a person's face at position A above a first threshold, then moves to position B as shown in Figure 4B, then to position C where the animal is located as shown in Figure 4C, then moves again to position B as shown in Figure 4B, and finally returns to position A as shown in Figure 4A, in a back-and-forth motion.

The rectangular frames in the through-image shown in Figure 4 represent AF frames. Solid AF frames indicate that the focus is set to that position by eye-tracking AF, while dashed AF frames indicate that the AF frame is moving and the focus is not set to that position.

The focus is initially set to position A, where the gaze is directed, and then to position C, where the gaze is directed. Therefore, the AF frame displayed on the display unit 110 is displayed over position A where the person's face is located, then passes through position B, and is displayed over position C where the animal is located. Finally, the AF frame passes through position B and is displayed again over position A where the person's face is located.

If this eye movement is repeated at a predetermined frequency, the eye-tracking information management unit 201 registers the eye-tracking information obtained from the coordinate data indicating that eye movement as reference eye-tracking information. Here, it is assumed that the eye gaze is focused on position A by a first threshold or more, and that position A is registered as the gaze position in the reference eye-tracking information.

In the example in Figure 4, position A, which is the point of fixation, is the endpoint of the line of sight in the trajectory of the reference line of sight information. However, this is merely an example, and the point of fixation is not necessarily limited to the endpoints of the trajectory. Furthermore, the trajectory is not necessarily straight; it can be curved or meandering. Moreover, the trajectory is not limited to returning to the starting point of movement as shown in Figure 4.

[1-3-2. Determination of the controlled object's position and image capture-related control]
Next, with reference to Figures 5 to 8, the determination of the control target position by the control target position determination unit 202 and the shooting-related control by the shooting-related control unit 203 will be explained. These are performed during shooting after the registration process of the reference line-of-sight information shown in Figures 3 and 4.

First, the determination of the controlled object position will be explained with reference to Figures 5, 6, and 7. First, in step S201 of the flowchart in Figure 5, if the line-of-sight utilization mode, in which the information processing device 200 operates, is turned on in the imaging device 100, the process proceeds to step S202 (Yes in step S201).

Next, in step S202, the control target position determination unit 202 obtains the latest gaze information (referred to as the latest gaze information) by querying the gaze information management unit 201, which has analyzed the coordinate data showing the latest movement of the photographer's gaze at the time of shooting after the registration of the reference gaze information. The gaze information management unit 201 supplies the latest gaze information in response to the inquiry from the control target position determination unit 202 and also stores it as history. Note that the latest gaze information is not stored as reference gaze information.

Thus, even when shooting after registering reference gaze information, the gaze information management unit 201 analyzes the coordinate data to acquire gaze information and stores it as history, and the control target position determination unit 202 acquires the latest gaze information from the gaze information management unit 201.

The latest gaze information is used in processing by the control target position determination unit 202, as well as in processing by the gaze information management unit 201. If it is repeated at a predetermined frequency, it is registered as reference gaze information. This reference gaze information is then used in the control target position determination process for subsequent shooting.

Next, in step S203, the control target position determination unit 202 reads and acquires the reference gaze information stored by the gaze information management unit 201 in the process shown in Figure 3.

Next, in step S204, if the operating mode of the information processing device 200 is the gaze focus fixed priority mode, the process proceeds to step S205 (Yes in step S204).

Next, in step S205, the control target position determination unit 202 compares and determines whether at least a portion of the reference gaze information and the latest gaze information match. In this comparison, in addition to comparing whether the trajectories of the reference gaze information and the latest gaze information match, it also compares whether the gaze positions match and whether the gaze times match. Furthermore, if there are multiple gaze positions, it may also compare whether the relative lengths of the gaze times for each gaze position match.

It should be noted that while it is stated that at least a portion of the reference gaze information and the latest gaze information must match, this does not exclude cases where all of them match. Cases where all of them match may be processed in the same way as cases where only a portion matches.

Furthermore, when comparing the trajectory of the reference gaze information with the trajectory of the latest gaze information, agreement may be included not only when the trajectories completely overlap and match, but also when the difference in angle, length, etc., of the trajectories is within a predetermined tolerance. In addition, when comparing the gaze position of the reference gaze information with the gaze position of the latest gaze information, agreement may be included when the difference in gaze position is within a predetermined tolerance. Moreover, when comparing the gaze time of the reference gaze information with the gaze time of the latest gaze information, agreement may be included when the difference in gaze time is within a predetermined tolerance.

If a portion of the reference gaze information and the latest gaze information match, the process proceeds to step S206 (Yes in step S205). Here, "partially matching" means, for example, that the trajectory of the reference gaze information and the trajectory of the latest gaze information match to a predetermined extent, the gaze position of the latest gaze information is included within a predetermined tolerance range that includes the gaze position of the reference gaze information, and the gaze time of the latest gaze information is included within a tolerance range set by upper and lower threshold limits for the gaze time of the reference gaze information.

Then, in step S206, the control target position determination unit 202 performs the control target position determination process in the gaze focus fixed priority mode. The control target position determination process in the gaze focus fixed priority mode will be described later with reference to Figure 6.

On the other hand, if the reference gaze information and the latest gaze information do not match, the process proceeds to step S207 (No. of step S205), and the control target position determination unit 202 determines the control target position based on the latest gaze information. Note that in the first cycle of the process, if the latest gaze information is fixated on only one coordinate position (such as the starting point of the trajectory) and that coordinate position is fixed on below the first threshold, then the latest gaze information and the reference gaze information do not partially match, so the process in Figure 5 proceeds from step S205 to step S207 (No. of step S205). Then, in step S207, the coordinate position indicated by the latest gaze information is determined as the control target position. Therefore, when the process in Figure 5 is started, the control target position is always determined in step S207 first.

The explanation returns to step S204. If the operating mode of the information processing device 200 in step S204 is not the gaze focus fixed priority mode, the process proceeds to step S208 (No. in step S204). If the operating mode is not the gaze focus fixed priority mode, it means that the operating mode is the gaze focus movement priority mode. Note that step S204 may also be a process to confirm whether the operating mode is the gaze focus movement priority mode. In that case, if Yes, the process proceeds to step S208, and if No, the process proceeds to step S205.

Next, in step S208, a comparison is performed to determine whether a portion of the reference line of sight information and the latest line of sight information match. The comparison method is the same as the method described in step S205.

If a portion of the reference gaze information matches the latest gaze information, the process proceeds to step S209 (Yes in step S208). In step S209, the control target position determination unit 202 performs the control target position determination process in gaze focus movement priority mode. The control target position determination process in gaze focus movement priority mode will be described later with reference to Figure 7.

If the reference gaze information and the latest gaze information do not partially match, the process proceeds to step S207 (No. of step S208), and the control target position determination unit 202 determines the control target position based on the latest gaze information.

The process shown in Figure 5 is performed in the manner described above.

Next, with reference to the flowchart in Figure 6, the process of determining the position of the target object in the gaze focus-fixed priority mode by the control target position determination unit 202 will be explained. This process corresponds to step S206 in the flowchart in Figure 5. The gaze focus-fixed priority mode keeps the focus fixed at the gaze position on the trajectory of the reference gaze information as long as a portion of the reference gaze information and the latest gaze information match.

In the explanation below, "the coordinate position indicated by the latest gaze information" refers to the latest position where the photographer's gaze is aligned along the trajectory of the gaze included in the latest gaze information.

First, in step S301, it is determined whether the coordinate position indicated by the latest gaze information has been gazed upon for a time exceeding the second threshold. If it has not been gazed upon for a time exceeding the second threshold, the process proceeds to step S302 (No. of step S301).

Then, in step S302, it is determined whether the current controlled target position is the gaze position of the reference gaze information, and whether the gaze time for that gaze position is greater than or equal to the second threshold. If it is greater than or equal to the second threshold, the process proceeds to step S303 (Yes in step S302). Then, in S303, the controlled target position is determined to be the same as the current position (the controlled target position is not changed).

On the other hand, if in step S301 the coordinate position indicated by the latest gaze information has been gazed upon for a time greater than the second threshold, or if in S302 the current controlled target position is the gaze position of the reference gaze information and the gaze time for that gaze position is less than the second threshold, the process proceeds to step S304 (Yes in step S301 or No in step S302). Then, in step S304, the controlled target position determination unit 202 determines the coordinate position indicated by the latest gaze information on the trajectory of the reference gaze information as the controlled target position. The information of the determined controlled target position is stored either by the information processing device 200 itself or by storing it in the storage unit 107 of the imaging device 100.

Next, referring to the flowchart in Figure 7, the process of determining the control target position in the eye-tracking focus movement priority mode by the control target position determination unit 202 will be explained. This process corresponds to step S209 in the flowchart in Figure 5. The eye-tracking focus movement priority mode anticipates the movement of the photographer's gaze during shooting and adjusts the focus to the control target position.

First, in step S401, if the trajectory of the reference gaze information and the trajectory of the latest gaze information match by a predetermined percentage or more, the process proceeds to step S402 (Yes in step S401).

Next, in step S402, if the gaze in the trajectory of the latest gaze information is directed towards the gaze position on the trajectory of the reference gaze information, the process proceeds to step S403 (Yes in step S402).

Next, in step S403, if there are multiple gaze positions on the trajectory of the reference line of sight information, the process proceeds to step S404 (Yes in step S403).

Next, in step S404, if the gaze time at the gaze position on the trajectory of the reference gaze information, which was determined in step S402 to be where the photographer's gaze was directed, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404). This gaze time is the gaze time included in the reference gaze information.

Then, in step S405, the control target position determination unit 202 determines the gaze position to be the control target position in the trajectory of the latest gaze information. The information of the determined control target position is stored either by the information processing device 200 itself or by storing it in the storage unit 107 of the imaging device 100.

If, in step S401, the trajectory of the reference gaze information and the trajectory of the latest gaze information do not match by a predetermined percentage or more (No. in step S401), the process proceeds to step S406. In step S406, the control target position determination unit 202 determines the control target position based on the latest gaze information. The same applies if, in step S402, the movement of the gaze in the latest gaze information is not directed toward the gaze position (No. in step S402), or if, in step S404, the gaze time toward the gaze position on the trajectory of the reference gaze information, which the photographer's gaze is said to be directed toward, is not equal to or greater than the second threshold (No. in step S403).

Next, the processing performed by the image capture control unit 203 will be explained with reference to the flowchart in Figure 8. The processing performed by the image capture control unit 203 is carried out after the control target position has been determined by the control target position determination unit 202.

First, in step S501, it is checked whether the control function is enabled. If the control function of the information processing device 200 is enabled, the process proceeds to step S502 (Yes in step S501).

Next, in step S502, if the gaze-based mode in which the information processing device 200 operates is turned on in the imaging device 100, the process proceeds to step S503 (Yes in step S502). Next, in step S503, the shooting-related control unit 203 reads out the control target position determined by the control target position determination unit 202.

Next, in step S504, the shooting-related control unit 203 performs control using the control target position. On the other hand, if the gaze-based mode is not turned on in step S502, the process proceeds to step S504 (No. of step S501), and control is performed without using the control target position.

In this embodiment, the first and second controls, which utilize the control target position controlled by the shooting-related control unit 203, will be described. The first control is a gaze-focus-fixed priority control that keeps the focus fixed and aligned to the control target position even if the photographer's gaze moves. The second control is a gaze-focus-movement priority control that, when the photographer's gaze moves, adjusts the focus to the control target position located at the destination of the gaze.

[1-3-3. Specific Examples of Control]
Next, with reference to Figures 9 to 14, the determination of the control target position by the control target position determination unit 202 and the control by the shooting-related control unit 203 will be explained based on specific examples. Figures 9 to 14 show the through image displayed on the display unit 110 of the imaging device 100 and the photographer's line of sight to the display unit 110 during shooting. As a prerequisite, it is assumed that the line of sight of the photographer to the display unit 110 is stored in advance as reference line of sight information by the line of sight information management unit 201.

First, a first example of the first control will be explained with reference to Figure 9. The first control is a control in a gaze focus-fixed priority mode that keeps the focus fixed to the control target position. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, and positions A and C are assumed to be the gaze positions.

Then, assume that the photographer's gaze moves from position A shown in Figure 9A to position B shown in Figure 9B during the shooting after the registration of the reference gaze information. This movement of the photographer's gaze during the shooting after the registration of the reference gaze information is the latest gaze information. If the trajectory of the gaze in the latest gaze information matches a part of the trajectory of the gaze in the reference gaze information, the process in Figure 5 proceeds from step S205 to step S206, and the control target position determination process in Figure 6 is performed. Assume that position A is the gaze position in the latest gaze information.

In the latest gaze information, at the point when the gaze moves from position A to position B, the coordinate position indicated by the latest gaze information is position B. Even if the gaze moves from position A to position B, if position B, which is the coordinate position indicated by the latest gaze information, is not gazed at for a time equal to or greater than the first threshold, the process in Figure 6 proceeds from step S301 to step S302 (No. in step S301). Then, in step S302, it is determined whether position A, which is the current controlled target position, is the gaze position of the reference gaze information, and whether the gaze time for that gaze position is equal to or greater than the second threshold. If the gaze time in the reference gaze information for position A is equal to or greater than the second threshold, the process proceeds to step S303 (Yes in step S302), and the controlled target position determination unit 202 determines the controlled target position to be the same as the current one (the controlled target position is not changed). Therefore, if the controlled target position was already determined to be position A before the photographer's gaze moved to position B, the controlled target position remains at position A. Once position A, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus fixing priority control so that it continues to focus on position A, which is the control target position, even if the photographer's gaze moves from position A to position B.

If, in the latest gaze information, the gaze has moved from position A to position B, and the trajectory of the gaze in the latest gaze information does not match a part of the trajectory of the gaze in the reference gaze information, or if the gaze is fixed on position B for a period exceeding the second threshold, or if the gaze time in the reference gaze information for position A is less than the second threshold, the coordinate position indicated by the latest gaze information is determined to be the control target position (step S207 or step S304).

Next, a second example of the first control will be explained with reference to Figure 10. As shown in Figure 10A, assume that the photographer's gaze in the image taken after registering the reference gaze information moved from position A to position B shown in Figure 10B, then to position C shown in Figure 10C, and finally back to position B shown in Figure 10B. This movement of the photographer's gaze in the image taken after registering the reference gaze information is the latest gaze information.

First, referring to Figure 10, we will explain an example of maintaining focus on position A, where the gaze time for position A is greater than or equal to the second threshold in the reference gaze information, and the gaze time for position C is less than the second threshold.

First, let's explain the case where the gaze moves from position A to position B in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, for less than the first threshold, the process proceeds from step S205 to step S206 (Yes in step S205). Since the gaze has not been fixed on position B for a time greater than the second threshold, the process proceeds further from step S301 to step S302 (No in step S301). Next, if position A, which is the current target of control, is the fixation position in the reference gaze information, and the fixation time for position A in the reference gaze information is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the target of control position determination unit 202 determines position A as the target of control, and the shooting-related control unit 203 controls the system to keep the focus fixed on position A.

Next, we will explain the case where the gaze moves from position A to position B, and then from position B to position C in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position C, which is the coordinate position indicated by the latest gaze information, for less than the second threshold, the process proceeds from step S205 to step S206 (Yes in step S205), and then from step S301 to step S302 (No in step S301). Next, if position A, which is the current control target position, is the fixation position in the reference gaze information, and the fixation time for position A, which is the fixation position in the reference gaze information, is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the control target position determination unit 202 determines position A as the control target position, and the shooting-related control unit 203 controls the system to continue fixing the focus on position A.

Next, we will explain the case where the gaze moves from position A to position B, from position B to position C, and then from position C back to position B in the latest gaze information. In this case, if the reference gaze information and the latest gaze information partially match, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, for less than the first threshold, the process proceeds from step S205 to step S206 (Yes in step S205). Since the gaze has not been fixed for a time greater than the second threshold, the process proceeds further from step S301 to step S302 (No in step S301). Next, if position A, which is the current target of control, is the fixation position in the reference gaze information, and the fixation time for position A in the reference gaze information is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the target of control position determination unit 202 determines position A as the target of control, and the shooting-related control unit 203 controls the system to keep the focus fixed on position A.

Next, we will explain the case where the reference gaze information and the latest gaze information do not partially match. A case where the reference gaze information and the latest gaze information do not partially match is when the gaze trajectory in the latest gaze information does not match the gaze trajectory in the reference gaze information, or when the gaze trajectory matches, but the latest gaze information shows that position B was gazed at for more than the first threshold, or position C was gazed at for more than the second threshold. In this case, the process proceeds from step S205 to step S207 (No. of step S205), and in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information (position B, position C, or any other coordinate position) as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position A and controls the system to focus on the new coordinate position which is the control target position.

Next, as an example of a change in the fixed focus position from position A to position C, we will explain the case where the gaze time for positions A and C in the reference gaze information is greater than or equal to the second threshold.

If, in the latest gaze information, the gaze moves from position A to position B, and the reference gaze information and the latest gaze information partially coincide, and the gaze is fixed on position B, which is the coordinate position indicated by the latest gaze information, below the first threshold, then this is the same as the case where the gaze moves from position A to position B in the example above where the focus is kept fixed on position A.

Furthermore, if the gaze moves from position A to position B, and then from position B to position C, and the reference gaze information and the latest gaze information partially coincide, and the gaze is fixed on position C, which is the coordinate position indicated by the latest gaze information, below the second threshold, then it is the same as in the example above where the gaze moves from position A to position B, and then from position B to position C, while maintaining focus on position A.

Furthermore, if the gaze moves from position A to position B in the latest gaze information, or from position A to position B, and then from position B to position C, and in any case the reference gaze information and the latest gaze information do not partially match, the process proceeds from step S205 to step S207 (No. of step S205). Next, in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position A and controls the system to focus on the new coordinate position which is the control target position.

Next, referring to Figure 10, we will explain the case where the focus fixed position changes from position A to position C. First, if the latest gaze information and the reference gaze information partially match, and in the latest gaze information the gaze moves from position A to position B, and then from position B to position C, and fixes on position C for a second threshold or more, the process proceeds from step S301 to step S304 (Yes in step S301). Then, in step S304, the control target position determination unit 202 determines position C as the control target position, and the shooting-related control unit 203 controls the system to release the fixation of position A and focus on position C.

Next, we will explain the processing after the gaze moves from position A to position B, and from position B to position C in the latest gaze information, and the focus fixed position changes from position A to position C. In the latest gaze information, the gaze moves from position C to position B, and the latest gaze information and the reference gaze information partially match (partial match means that position B is being gazed at for less than the first threshold), and if position B, which is the coordinate indicated by the latest gaze information, is being gazed at for less than the first threshold, then position B is not being gazed at for more than the second threshold, so the process proceeds from step S205 to step S206 (Yes in step S205), and the process proceeds from step S301 to step S302 (No in step S301). Next, if position C, which is the current controlled target position, is the gaze position in the reference gaze information, and the gaze time for position C, which is the gaze position in the reference gaze information, is greater than or equal to the second threshold, the process proceeds from step S302 to step S303 (Yes in step S302). Then, in step S303, the control target position determination unit 202 determines position C as the control target position, and the shooting-related control unit 203 controls the system to keep the focus fixed on position C.

Next, we will explain the case where the reference gaze information and the latest gaze information do not partially match. A case where the reference gaze information and the latest gaze information do not partially match includes cases where the gaze trajectory in the latest gaze information does not match the gaze trajectory in the reference gaze information, and cases where the gaze trajectories match, but the latest gaze information shows the gaze fixed on position B above the first threshold. In this case, the process proceeds from step S205 to step S207 (No. of step S205), and in step S207, the control target position determination unit 202 determines the coordinate position indicated by the latest gaze information as the control target position. Then, the shooting-related control unit 203 releases the focus lock at position C and controls the system to focus on the new coordinate position which is the control target position.

This prevents the focus from shifting to a different position against the photographer's intention. Furthermore, it gives the photographer the impression that the imaging device 100 is understanding their intentions. This eye-line focus-fixing priority control is particularly effective in video recording, where the focus often needs to remain fixed at the same position. However, this control can also be used in still image shooting.

Furthermore, if the trajectory of the latest gaze information after registration of the reference gaze information matches only in a certain point, without including the starting point of the reference gaze information trajectory, gaze focus lock priority control will not be performed. For example, if the trajectory of the reference gaze information is position A, position B, position C, position B, position A, and the trajectory of the latest gaze information does not start from position A, but matches only from position B to position C.

Next, with reference to Figures 11 to 13, we will explain an example in which the gaze focus lock priority control is not performed because the trajectory of the latest gaze information deviates from the trajectory of the reference gaze information. Figures 11 to 13 show the through image displayed on the display unit 110 of the imaging device 100 and the photographer's gaze toward the display unit 110 during shooting. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, with positions A and C being gaze positions, and the gaze time at position A being greater than or equal to the second threshold, and the gaze time at position C being less than the second threshold.

As shown in Figure 11A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 11B, the gaze moves to position B, then to position C as shown in Figure 11C, and then to position D as shown in Figure 11D. In this case, because the gaze has moved to position D, the reference gaze information in Figure 4 and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (No. of step S205), and the control target position determination unit 202 determines position D, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the system to focus on position D, which is a new control target position that is not on the trajectory of the reference gaze information. Since position D is not on the trajectory of the reference gaze information, the system exits the gaze focus fixed priority control.

In the example shown in Figure 11, when the photographer's gaze moves to position D in the latest gaze information after the registration of the reference gaze information, gaze focus lock priority control is not performed, and control is performed to focus on position D. However, until the gaze moves to position D, gaze focus lock priority control is performed to keep the focus on position A.

Furthermore, as shown in Figure 12A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 12B, the gaze moves to position E. In this case, because the gaze is moving towards position E, the reference gaze information in Figure 4 and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (No. of step S205), and the control target position determination unit 202 determines position E, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the system to focus on position E, which is a new control target position that is not on the trajectory of the reference gaze information. Since the gaze is not on the trajectory of the reference gaze information when it moves towards position E, the focus lock operation is exited.

Furthermore, as shown in Figure 13A, after registering the reference gaze information, the latest gaze information shows that the photographer's gaze movement aligns with position A and the focus is set to position A. Then, as shown in Figure 13B, suppose the photographer's gaze moves to position G to change the subject from the person on the right to the person on the left. In this case, because the photographer has moved towards position G, the reference gaze information and the latest gaze information no longer match. Therefore, in the process shown in Figure 5, the process proceeds from step S205 to step S207 (No. of step S205), and the control target position determination unit 202 determines position G, which is the position of the latest gaze, as the control target position. The shooting-related control unit 203 then controls the camera to focus on position G, which is the new control target position and is not a position on the trajectory of the reference gaze information. Since the camera is no longer at the trajectory of the reference gaze information when it moves towards position G, the camera exits the focus lock operation.

Next, the second control will be explained with reference to Figure 14. The second control is a control in the gaze focus movement priority mode, which anticipates the movement of the photographer's gaze and moves the focus to the target position. Here, as described above in Figure 4, the trajectory of the reference gaze information is the trajectory in which the gaze moves in the order of position A, position B, position C, position B, position A, with positions A and C being the gaze positions.

As shown in Figure 14, in the latest gaze information after the registration of the reference gaze information, the movement of the photographer's gaze is assumed to be from position A to position B shown in Figure 14B, then to position C shown in Figure 14C, and then back to position B shown in Figure 14B.

First, if the latest gaze information shows that the movement of the gaze is from position A through position B towards position C, and the reference gaze information and the latest gaze information match by a predetermined ratio, the process in Figure 7 proceeds from step S401 to step S402 (Yes in step S401). Next, since the movement of the gaze in the latest gaze information is toward position C, which is the gaze position of the reference gaze information, the process proceeds from step S402 to step S403 (Yes in step S402). Next, since there are multiple gaze positions at positions A and C in the reference gaze information, the process proceeds from step S403 to step S404 (Yes in step S403). Then, if the gaze time at position C, which is the gaze position toward which the gaze is directed in the latest gaze information, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404), and position C toward which the gaze is directed in the latest gaze information is determined to be the control target position. When position C, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus movement priority control to move the focus to position C, which is the control target position, before the photographer's gaze reaches position C, and focuses on position C.

Subsequently, if the latest gaze information shows that the movement of the gaze is from position C through position B towards position A, and the reference gaze information and the latest gaze information match by a predetermined ratio, the process in Figure 7 proceeds from step S401 to step S402 (Yes in step S401). Since the movement of the gaze in the latest gaze information is toward position A, which is the gaze position of the reference gaze information, the process proceeds from step S402 to step S403 (Yes in step S402). Since there are multiple gaze positions at positions A and C in the reference gaze information, the process proceeds from step S403 to step S404 (Yes in step S403). Then, if the gaze time at position A, which is the gaze position toward in the latest gaze information, is greater than or equal to the second threshold, the process proceeds to step S405 (Yes in step S404), and position A, toward which the gaze is toward in the latest gaze information, is determined to be the control target position. When position A, which is the gaze position, is determined to be the control target position, the shooting-related control unit 203 performs gaze focus movement priority control to move the focus to position A, which is the control target position, before the photographer's gaze reaches position A, and focuses on position A.

Furthermore, if the reference gaze information and the latest gaze information do not match by a predetermined ratio when the gaze movement is from position A through position B to position C in the latest gaze information (No. in step S401), or if the gaze time at position C, which is the gaze position to which the gaze is directed in the latest gaze information, is not equal to or greater than the second threshold (No. in step S404), the process determines the controlled target position based on the latest gaze information in step S406.

Furthermore, if the latest gaze information does not match the reference gaze information by a predetermined ratio when the gaze movement is from position C to position A via position B, or if the gaze time at position C, which is the gaze position to which the gaze is directed, is not equal to or greater than the second threshold (No. in step S404), the process determines the controlled target position based on the latest gaze information in step S406.

In this way, the system can anticipate the photographer's intentions and focus on the optimal position. Because the system anticipates the photographer's intentions and focuses on position A before the photographer's gaze aligns with position A, the photographer can immediately take a picture. Furthermore, the photographer can be given the impression that the imaging device 100 understands their intentions. Eye-tracking focus shift priority control is particularly effective in still image shooting, where the focus point often changes. However, this control may also be used in video recording.

The photographer may choose whether the shooting-related control unit 203 prioritizes eye-tracking focus locking or eye-tracking focus movement, or the information processing device 200 may automatically set which control to perform based on the operating mode of the imaging device 100. For example, it may be automatically set to prioritize eye-tracking focus locking in video shooting mode and prioritize eye-tracking focus movement in still image shooting mode.

Alternatively, the system may automatically set which control to perform based on the autofocus mode of the imaging device 100. For example, it may be set to prioritize eye-tracking focus movement in single AF mode and to prioritize eye-tracking focus locking in continuous AF mode.

Single AF is an autofocus mode where the autofocus engages when the shutter button is half-pressed, and once focus is achieved, it remains locked in that position. This mode is suitable for shooting stationary subjects such as landscapes and snapshots. Continuous AF, on the other hand, is a mode where the autofocus works as long as the shutter button is half-pressed, continuously maintaining focus. This mode is suitable for shooting moving subjects such as sports and railway photography.

Alternatively, the system may automatically set which control to perform based on the autofocus subject tracking sensitivity. Generally, an imaging device 100 with AF functionality has an AF tracking sensitivity setting function that allows the user to select from multiple levels (for example, five levels) of AF tracking sensitivity when the subject moves out of the focus area during still image shooting.

For example, if the tracking sensitivity has five levels from 1 to 5, the highest sensitivity, "5," is "sensitive" and is effective for shooting scenes where you want to successively focus on subjects at different distances. The lowest sensitivity, "1," is "sticky," and even if another subject crosses in front of the subject you want to photograph, it prevents the focus from shifting to the other subject, allowing for stable focus tracking. "3" is the sensitivity in between.

The system is set to prioritize eye-tracking focus lock when the tracking sensitivity is low (sticky), and to prioritize eye-tracking focus movement when the subject tracking sensitivity is high (sensitive).

Alternatively, the system can automatically determine which control method to use based on the latest gaze information. If the latest gaze information indicates that the photographer's gaze frequently returns to the target position, the system can be set to prioritize gaze focus locking. Conversely, if the latest gaze information indicates that the photographer's gaze rarely returns to the target position, the system can be set to prioritize gaze focus movement.

Furthermore, if the latest gaze information indicates that the gaze remains fixed (stares) on the target location for a long period, the system will be set to prioritize gaze focus locking. Conversely, if the latest gaze information indicates that the gaze remains fixed (stares) on the target location for a short period, the system will be set to prioritize gaze focus movement.

Furthermore, if the latest gaze information shows a large relative ratio of the time the gaze remains (is fixed on) at the controlled target position compared to other positions, the system will be set to prioritize gaze focus locking. On the other hand, if the latest gaze information shows a small relative ratio of the time the gaze remains (is fixed on) at the controlled target position compared to other positions, the system will be set to prioritize gaze focus movement.

Furthermore, the system may be configured to automatically set whether to perform the first control or the second control based on the subject recognized by the subject recognition unit 112. If the subject at the position where the photographer's gaze is directed (i.e., the subject is focused on) is larger than a predetermined size, the system may be configured to prioritize fixed gaze focus control. On the other hand, if the subject at the position where the photographer's gaze is directed (i.e., the subject is focused on) is smaller than a predetermined size, the system may be configured to prioritize shifting gaze focus control.

Next, we will explain other controls.

In the third control example, the shooting-related control unit 203 adjusts the exposure based on the controlled position. The shooting-related control unit 203 adjusts the exposure by transmitting an exposure adjustment control signal to the lens drive driver 103. By adjusting the exposure, the amount of light captured can be adjusted according to the subject at the controlled position, thereby obtaining videos and still images with optimal brightness. Alternatively, the shooting-related control unit 203 may control the lens drive driver 103 to perform exposure adjustment, or the shooting-related control unit 203 itself may be configured to function as the lens drive driver 103.

In the fourth control example, the shooting-related control unit 203 adjusts the white balance based on the controlled position. The shooting-related control unit 203 adjusts the white balance by having the camera processing unit 106 transmit a control signal for white balance adjustment. By adjusting the white balance to match the subject at the controlled position, it is possible to obtain videos and still images with a color tone more desirable to the user. Alternatively, the shooting-related control unit 203 may control the camera processing unit 106 to perform white balance adjustment, or the shooting-related control unit 203 itself may be configured to function as the camera processing unit 106 that performs white balance adjustment.

In the fifth control example, the imaging-related control unit 203 generates metadata related to the controlled position, which is attached to the captured image. This metadata may include, for example, the type, size, and color of the subject recognized at the controlled position. The metadata may be used internally within the imaging device 100 or by an external device outside the imaging device 100. The imaging device 100 may be controlled so that the imaging-related control unit 203 generates the metadata, or the imaging-related control unit 203 may be configured to generate the metadata itself.

In the sixth control example, the shooting-related control unit 203 inputs a shutter button when the target position is within the field of view, and controls the imaging device 100 to obtain a video or still image with the target position within the field of view. Furthermore, the shooting-related control unit 203 may also perform recording control to save the image data (still images or video images) obtained from the shooting to the storage unit 107.

In the seventh control example, the shooting-related control unit 203 determines the object to be recognized by the subject recognition processing performed by the subject recognition unit 112. For example, the object to be recognized is determined to be a subject located at the control target position. The accuracy of autofocus can be improved by performing autofocus to focus on the subject recognized based on the subject recognition information. The shooting-related control unit 203 may control the subject recognition unit 112, or the shooting-related control unit 203 itself may be configured to function as a subject recognition unit.

When the subject recognition unit 112 recognizes a subject present at the control target position, the control target position is set to the position of the recognized subject on the display unit 110. Therefore, when the subject moves, the control target position can follow the subject's movement. As a result, even if the subject moves, the control target position will move in accordance with the subject's position. Thus, even if the control target position moves, the shooting-related control unit 203 can continue to perform control, such as maintaining focus on the control target position.

The processing in this technology is carried out as described above. According to this technology, the imaging device 100 can perform shooting in accordance with the photographer's intentions, giving the impression that it is understanding the photographer's intentions in response to the photographer's eye movements.

<2. Variations>
Although embodiments of this technology have been described in detail above, this technology is not limited to the embodiments described above, and various modifications are possible based on the technical concept of this technology.

In the control target position determination process in this embodiment, the comparison between the reference gaze information and the latest gaze information includes a comparison of the gaze position and gaze time. However, control may be performed by comparing only the trajectory and gaze position without including the gaze time. Alternatively, control may be performed by comparing only the trajectory without including the gaze position and gaze time.

Furthermore, if the imaging device 100 has a function that allows the AF frame to be fixed at an arbitrary position based on input from the photographer (such as half-pressing the shutter), the control target position determination unit 202 may determine the position specified by the user as the control target position.

Furthermore, the half-press function of the shutter of the imaging device 100 may be used for processing by the information processing device 200. The first specific example is that the gaze position that the photographer was looking at when the shutter was half-pressed while their gaze was aligned with the display unit 110 is treated as a gaze position of equivalent importance to one that was looked at for a time exceeding the second threshold.

The second specific example involves determining whether or not the shutter was half-pressed, instead of, or in addition to, determining whether or not the process in step S105 of the process in Figure 3 is repeated at a predetermined frequency.

Furthermore, a third specific example involves replacing step S302 of the process in Figure 6 with a determination of whether the current controlled target position lies on the trajectory of the reference line of sight information and coincides with the gaze position where the shutter was half-pressed.

In this embodiment, the information processing device 200 was described as operating within the imaging device 100. However, this technology may also be implemented by connecting the information processing device 200, configured on a server, and the imaging device 100 via a network.

In the embodiment, it was explained that the shooting-related control unit 203 performs predetermined control by comparing reference gaze information with the latest gaze information. However, instead of reference gaze information, the gaze information management unit 201 may store the time the gaze is aligned to a specific position and/or the frequency of repeated gaze movements, and if the gaze movement during shooting after storage matches the stored content, the shooting-related control unit 203 may perform predetermined control.

In this embodiment, the reference gaze information was described as the movement of the photographer's gaze relative to the display unit 110, but the reference gaze information may be something other than that. For example, it may be preset data showing the movement of the gaze that has been prepared in advance before shooting. This preset data consists of position data, trajectory data, gaze position data, etc., that show the movement of the gaze. This preset data may be prepared in advance as a preset by the manufacturer of the imaging device 100 and stored in the information processing device 200, or it may be created by the photographer or another user and stored in the information processing device 200. The preset data can also be created by referring to the photographer's gaze movement habits obtained by analyzing the photographer's gaze movement, or to the gaze movements common to multiple photographers obtained by analyzing the gaze movements of multiple photographers.

In this embodiment, a vector may be used instead of the direction of movement in step S402 of Figure 7.

In this embodiment, control was performed based on the photographer's eye movement information as input. However, control may also be performed based on other input methods as long as they reflect the photographer's intentions.

This technology can also be configured as follows:
(1)
An information processing device comprising a control target position determination unit that determines the control target position based on reference line of sight information including the trajectory on the display unit of an imaging device, and the latest line of sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting.
(2)
The information processing device according to (1), wherein the reference line of sight information includes the trajectory of the photographer's line of sight to the display unit during a photograph taken prior to the time of the photograph.
(3)
The information processing device according to (2), wherein the reference gaze information includes a gaze position which is a position on the trajectory where the photographer's gaze was aligned for a predetermined period of time or longer.
(4)
The information processing device according to (3), wherein the reference gaze information includes a gaze time, which is the time the photographer's gaze was aligned with the gaze position.
(5)
The information processing device according to (3), wherein the control target position determination unit determines the gaze position included in the reference gaze information as the control target position when part or all of the reference gaze information matches the latest gaze information.
(6)
The information processing apparatus according to (5), wherein the control target position determination unit determines the control target position based on a comparison result between the gaze time, which is the time the photographer's line of sight was aligned with the gaze position, and a predetermined threshold.
(7)
The information processing device according to (5), wherein the control target position determination unit determines the gaze position as the control target position when the gaze is directed toward the gaze position included in the reference gaze information in the trajectory of the latest gaze information.
(8)
An information processing apparatus according to any one of (1) to (7), comprising an imaging-related control unit that performs imaging-related control by the imaging device based on the control target position.
(9)
The information processing device according to (8), wherein the shooting-related control unit performs autofocus control to maintain focus on the target position or to move the focus position to the target position.
(10)
The information processing device according to (8), wherein the shooting-related control unit performs autofocus control to maintain focus on the target position when the imaging device is shooting a video, and performs autofocus control to move the focus position to the target position when the imaging device is shooting a still image.
(11)
The information processing device described in (8) is an information processing device that performs control to adjust the exposure based on the control target position, wherein the shooting-related control unit performs control to adjust the exposure based on the control target position.
(12)
The aforementioned shooting-related control unit performs control to adjust the white balance with reference to the control target position, as described in (8).
(13)
The information processing device according to (8), wherein the imaging-related control unit performs both or either control to output information about the controlled target position and control to generate information about the controlled target position as metadata.
(14)
The information processing device according to (8), wherein the shooting-related control unit performs shooting control so that the controlled target position is within the field of view.
(15)
The aforementioned shooting-related control unit performs recording control of the image data generated by the shooting, as described in (8).
(16)
An information processing apparatus according to any one of (1) to (15), comprising a display control unit that displays information relating to the controlled position on the display unit.
(17)
The reference line-of-sight information is prepared in advance as data before the time of the photograph. The information processing device according to any one of (1) to (16).
(18)
The information processing device according to any one of (1) to (17), wherein the control target position determination unit determines the position specified by the input to the imaging device by the photographer as the control target position.
(19)
An information processing method for determining the position of a target to be controlled based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting.
(20)
An information processing program that causes a computer to execute an information processing method for determining the position of a control target based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting.

100...Imaging device 110...Display unit 200...Information processing device 201...Eyes-line information management unit 202...Control target position determination unit 203...Shooting-related control unit

Claims (16)

The system includes a control target position determination unit that determines the control target position based on reference line-of-sight information including the trajectory on the display unit of the imaging device and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit during shooting.
The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer.
The control target position determination unit determines the gaze position as the control target position if part or all of the reference gaze information and the latest gaze information match, and furthermore, the gaze in the trajectory of the photographer's gaze in the latest gaze information is directed toward the gaze position included in the reference gaze information. The information processing device according to claim 1, wherein the reference gaze information includes a gaze time, which is the time during which the photographer's gaze was aligned with the gaze position. The information processing apparatus according to claim 2, wherein the control target position determination unit determines the control target position based on a comparison result between the gaze time, which is the time the photographer's line of sight was aligned with the gaze position, and a predetermined threshold. The information processing apparatus according to claim 1, further comprising a shooting-related control unit that performs control related to shooting by the imaging device based on the control target position. The information processing apparatus according to claim 4 , wherein the shooting-related control unit performs autofocus control to maintain focus on the target position or to move the focus position to the target position. The information processing apparatus according to claim 4, wherein the shooting-related control unit performs autofocus control to maintain focus on the target position when the imaging device is shooting a video, and performs autofocus control to move the focus position to the target position when the imaging device is shooting a still image . The information processing apparatus according to claim 4 , wherein the shooting-related control unit performs control to adjust the exposure based on the controlled target position. The information processing apparatus according to claim 4 , wherein the shooting-related control unit performs control to adjust the white balance with reference to the control target position. The information processing apparatus according to claim 4 , wherein the imaging-related control unit performs both or either of the following: a control that outputs information regarding the controlled target position and a control that generates information regarding the controlled target position as metadata. The information processing apparatus according to claim 4 , wherein the shooting-related control unit performs shooting control so that the controlled target position falls within the field of view. The information processing apparatus according to claim 4 , wherein the shooting-related control unit controls the recording of image data generated by the shooting. The information processing apparatus according to claim 1, further comprising a display control unit that displays information relating to the controlled position on the display unit. The information processing device according to claim 1, wherein the reference line of sight information is prepared in advance as data before the time of shooting. The information processing apparatus according to claim 1, wherein the control target position determination unit determines the position specified by the input to the imaging device by the photographer as the control target position. This is an information processing method for determining the position of a target to be controlled based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting.
The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer.
An information processing method for determining the gaze position as the control target position when the reference gaze information and the latest gaze information match in part or in whole, and further, the gaze in the trajectory of the photographer's gaze in the latest gaze information is directed toward the gaze position included in the reference gaze information. This is an information processing method for determining the position of a target to be controlled based on reference line-of-sight information including the trajectory on the display unit of an imaging device, and the latest line-of-sight information including the trajectory of the photographer's gaze toward the display unit at the time of shooting.
The aforementioned reference gaze information includes the trajectory of the photographer's gaze toward the display unit during a photograph taken prior to the time of the photograph, and the gaze position which is a position on the trajectory of the gaze where the photographer's gaze was aligned for a predetermined period of time or longer.
An information processing program that causes a computer to execute an information processing method that determines the gaze position as the control target position when part or all of the reference gaze information and the latest gaze information match, and furthermore, when the gaze in the trajectory of the photographer's gaze in the latest gaze information is directed toward the gaze position included in the reference gaze information.