JP6281426B2 - Image display control device, image display control method, and image display control program - Google Patents

Description

  The present invention relates to an image display control device, an image display control method, and an image display control program.

  In recent years, devices equipped with photographing functions such as digital cameras, mobile phones, smartphones, tablet terminals, and notebook PCs (personal computers) have become widespread. In addition, for example, there are increasing opportunities for users to send information on blogs, social networks, and the like, and accordingly, opportunities for users to take pictures have also increased.

  On the other hand, the user's interest in shooting that takes advantage of blurring effect performed by a single-lens reflex camera or the like is increasing. For example, a plurality of images are shot by changing a subject to be focused at a certain angle of view. Things have been done.

  In this regard, there has been known a technique related to an imaging control device for a camera that performs automatic focus and exposure control using information on a gazing point of a photographer's gaze detected by a gaze detection device. In addition, a technique for providing an omnidirectional high-definition photographing system applicable indoors is known. A technique for suppressing generation and recording of an in-focus image contrary to the user's intention is known. Techniques for reducing the burden on the user's eyes are known. (For example, Patent Documents 1 to 4) Further, Patent Documents 5 to 8 also describe related techniques.

Japanese Patent Laid-Open No. 3-107932 JP 2004-194075 A JP 2012-44563 A JP 2013-242671 A JP 2010-114611 A JP 2001-159730 A Japanese Unexamined Patent Publication No. 2-032312 JP 2007-004471 A

  However, for example, when acquiring a plurality of images by focusing on each of a plurality of subjects at a certain angle of view, it is appropriate to reflect the movement of the line of sight of the photographer when shooting the acquired images. There is no known technique for displaying on. An object of one aspect of the present invention is to appropriately display a plurality of images acquired by focusing on each of a plurality of subjects at a certain angle of view.

  One aspect of the present invention provides an image display control device including an output unit. The output unit displays a plurality of images obtained by focusing on each of a plurality of subjects detected based on a photographer's line of sight according to a signal instructing shooting based on a degree of gaze. Display information for output. The gaze degree is obtained from the line of sight of the photographer toward each of the plurality of subjects.

  According to one aspect, it is possible to appropriately display a plurality of images acquired by focusing on each of a plurality of subjects at a certain angle of view.

It is a figure which illustrates the some to-be-photographed object in a different distance in a certain angle of view of an image display control apparatus. It is a figure which illustrates about imaging | photography of the some to-be-photographed object within the angle of view. It is a figure which illustrates the image display control apparatus which concerns on embodiment. It is a figure which illustrates the functional block structure of the image display control apparatus which concerns on embodiment. It is a figure which illustrates the operation | movement flow of the imaging | photography process which concerns on 1st Embodiment. It is a figure which illustrates the operation | movement flow of the display information output process of the image based on the score which concerns on 1st Embodiment. It is a figure which illustrates the display of the image by the display information which concerns on 1st Embodiment. It is a figure which illustrates the some subject in a certain angle of view of an image display control apparatus. It is a figure which shows the example which integrates and image | photographs the some to-be-photographed object within the angle of view. It is a figure which illustrates the operation | movement flow of the imaging | photography process which concerns on 2nd Embodiment. It is a figure which illustrates the operation | movement flow of the imaging efficiency improvement process which concerns on 2nd Embodiment. It is a figure which illustrates the hardware constitutions of the computer for implement | achieving the image display control apparatus which concerns on embodiment.

  As described above, in recent years, the user's interest in shooting that takes advantage of the blurring effect performed by a single-lens reflex camera or the like has increased. For example, a plurality of images can be obtained by changing a subject to be focused at a certain angle of view. It is often done such as shooting. Therefore, for example, when a plurality of subjects are within a certain angle of view at the time of shooting, when the user performs one shooting operation, a plurality of images are focused on each of the plurality of subjects. It is conceivable to take a picture of. In this way, for example, it is possible to save the user from having to retake the image a plurality of times while changing the focus at a certain angle of view, or to specify the position of the focus after shooting. However, for example, when a plurality of images are obtained by focusing on each of a plurality of subjects at a certain angle of view, the movement of the line of sight of the photographer at the time of capturing the plurality of images is reflected. And the technology to display it properly is not known. Then, it is difficult to preferentially display an image focused on a subject that is likely to be viewed by the user from among the plurality of obtained images.

  For example, when the user captures an image at a certain angle of view, a plurality of subjects having different distances from the image display control device 3 may be contained within the angle of view. FIG. 1 is a diagram illustrating a plurality of subjects at different distances at a certain angle of view of the image display control device 3. In the example shown in FIG. 1A, the subject a: a bird, the subject b: a house, and the subject c: a background are within the angle of view. As shown in FIG. 1B, the distance from the image display control device 3 to the subject a: the bird is about 0.3 m, and the subject b from the image display control device 3 to the house is about a distance. The distance from the image display control device 3 to the subject c: background is 100 m or more.

  In this case, it is conceivable to shoot a plurality of images while focusing on each of a plurality of subjects having different distances from the image display control device 3. FIG. 2 is a diagram illustrating the photographing of a plurality of subjects that are within the angle of view. In FIG. 2A, the subject a: the bird is focused, and the other subject b: the house and the subject c: the background at different distances are blurred. In the image shown in FIG. 2B, the subject b is focused on the house, and the other subject a: the bird and the subject c: the background are blurred. Similarly, the image shown in FIG. 2C is focused on the subject c: background, and other subjects are blurred. For example, in this way, a plurality of images are acquired by focusing on each of a plurality of subjects within a certain angle of view. In this case, it is difficult to preferentially display an image that is likely to be desired by the user among a plurality of captured images. For this reason, for example, when confirming an image taken by a user, it takes time and effort to sequentially look at the obtained image until an image focused on an object to be taken is displayed.

  Accordingly, in some embodiments, a gaze degree score representing the degree to which the photographer gazes at the subject at the time of shooting is calculated, and an object with a high gaze degree is focused based on the obtained gaze degree score. Display the selected image with priority. For example, when the photographer takes an image, a subject with a long time (gazing time) that the photographer is gazing in the finder of the image display control device 3 is an object that the photographer wants to photograph. It is estimated that the possibility is high. Similarly, it is presumed that a subject with a high ratio (gaze rate) being watched at a predetermined time interval at the time of shooting is likely to be a target that the photographer wants to take. In addition, it is estimated that an object with a fast turn (gaze order) that is watched by the photographer at the time of photographing is likely to be a target that the photographer wants to photograph. Therefore, in some embodiments, for example, a gaze degree score is calculated from values based on the gaze of the photographer, such as the gaze time, gaze rate, and gaze order, and gaze is performed based on the gaze degree score. An image focused on a high-level subject is displayed with priority. Thereby, for example, it is possible to preferentially display an image that is likely to be desired by the user. Several embodiments will be described below with reference to the drawings.

<First Embodiment>
FIG. 3 is a diagram illustrating the image display control device 3 according to the embodiment. FIG. 3A is a rear view of the image display control device 3, and FIG. 3B illustrates a front view of the image display control device 3. The image display control device 3 may be a device such as a digital camera, a mobile phone, a smartphone, a tablet terminal, a bangle terminal, or a notebook PC, for example. The image display control device 3 may have a photographing function. For example, the image display control device 3 may include an image sensor 31 such as a camera using a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or the like. Further, as shown in FIG. 3B, the image display control device 3 may include, for example, a line-of-sight sensor 32 for detecting a photographer's line of sight. The line-of-sight sensor 32 may be, for example, an image sensor such as a camera, or may be a line-of-sight detection device including an infrared light source and a light receiving element. Further, the image display control device 3 may include a display device 33 such as a display, for example, and the display device 33 may function as a finder at the time of shooting.

  FIG. 4 is a diagram illustrating a functional block configuration of the image display control device 3 according to the embodiment. The image display control device 3 includes, for example, a control unit 400 and a storage unit 420. The control unit 400 includes functional units such as an acquisition unit 411, a calculation unit 412, and an output unit 413. The storage unit 420 stores information such as the program 430, for example. The control unit 400 of the image display control device 3 may function as functional units such as the acquisition unit 411, the calculation unit 412, and the output unit 413 by reading and executing the program 430. Details of these functional units and details of information stored in the storage unit 420 will be described later.

  FIG. 5 is a diagram illustrating an operation flow of imaging processing according to the first embodiment. In one embodiment, the imaging process starts when an execution instruction for starting the imaging function in the image display control device 3 is input to the control unit 400.

  In step 501 (hereinafter, step is described as “S”, for example, expressed as S501), the control unit 400 of the image display control device 3 activates the line-of-sight sensor 32 to start line-of-sight detection, and the control is performed in step S502. The unit 400 detects the photographer's line of sight. In the line-of-sight detection, the control unit 400 detects the direction of the line of sight of the photographer, and identifies where the photographer is gazing in the image displayed on the viewfinder. The line-of-sight detection may be executed by various methods, for example, using the method described in Patent Document 4.

  In S502, when the control unit 400 detects the photographer's line of sight, for example, the control unit 400 continues to detect the movement of the photographer's line of sight over a predetermined period. In step S <b> 503, for example, when there is a part of the finder's line-of-sight movement detected over a predetermined period of time that has been gazed on the viewfinder for a long time, the control unit 400 gazes at the subject at that part. Extracted as a subject. That is, for example, the control unit 400 may extract a subject that has been watched over a predetermined time in the movement of the photographer's line of sight detected over a predetermined period.

  In step S504, the control unit 400 determines whether there are a plurality of subjects extracted in step S503. If there is one extracted subject (NO in S504), the flow proceeds to S505. In S505, the control unit 400 selects one extracted subject, and focuses the selected subject in S506. Note that the control unit 400 may display, for example, an image focused on the selected subject on the finder of the image display control device 3. In step S507, the control unit 400 locks the focus while the selected subject is focused. In S508, for example, when the control unit 400 receives an input for instructing photographing from the photographer, for example, when the photographer depresses the shutter button, the flow proceeds to S509. In step S <b> 509, the control unit 400 of the image display control device 3 performs shooting while focusing on one extracted subject, and acquires one image. In step S510, the control unit 400 of the image display control device 3 stores the obtained image data, for example, in the storage unit 420, and the flow returns to step S501.

  On the other hand, if there are a plurality of subjects extracted in S504 (YES in S504), the flow proceeds to S511. In S511, for example, the control unit 400 calculates a gaze rate for each of the plurality of subjects extracted in S503. The gaze rate may be calculated as, for example, a ratio of time when each of the plurality of subjects extracted in S503 is gazed after a predetermined time elapses after the photographer's line of sight is detected in S502. That is, for example, the control unit 400 uses the period from when the line of sight of the photographer is detected in S502 to the extraction of the subject in S503 as a denominator, and the time during which the individual subjects extracted in this period are being watched. The ratio obtained as a molecule may be calculated as the gaze rate.

  Subsequently, in S512, the control unit 400 compares the gaze rate with respect to each subject calculated in S511, and determines whether or not the gaze rates with respect to each of the plurality of subjects extracted in S503 are approximately the same. For example, the control unit 400 may determine that the gaze rates are the same if the calculated differences in the gaze rates for each subject are all equal to or less than a predetermined value.

  If the gaze rate is different in S512 (NO in S512), the flow proceeds to S513. In S513, the control unit 400 of the image display control device 3 calculates a gaze degree score representing the degree of gaze for each of the plurality of subjects extracted in S503. For example, the control unit 400 may calculate a gaze degree score based on the gaze rate. In this case, the control unit 400 sets the score of the subject with the lowest gaze rate as one point, and gives each subject a score that is increased one point at a time from the lowest gaze rate to the higher. A gaze degree score for the subject may be calculated.

  Alternatively, the control unit 400 may calculate a gaze degree score based on, for example, the gaze time for each subject. For example, in this case, the control unit 400 sets the score of the subject with the shortest gazing time as one point during the period from when the photographer's line of sight is detected in S502 until the subject is extracted in S503. Then, the control unit 400 calculates a gaze degree score for each subject by giving the subject a score that is increased by one point in order from the subject with the shortest gaze time to the longer gaze time. It's okay.

  In yet another example, the control unit 400 may calculate a gaze degree score based on, for example, the gaze order of each subject. For example, in this case, the control unit 400 sets the score of the slowest subject in which the subject is watched as one point in the period from when the photographer's line of sight is detected in S502 to when the subject is extracted in S503. To do. Then, the control unit 400 may calculate a gaze degree score for each subject by giving the subject a score that is increased one point at a time from the slowest to the fastest.

  In still another embodiment, the score of the degree of gaze of the subject may be calculated by combining the gaze rate, the gaze time, and the score obtained based on the gaze order. That is, for example, the score of the gaze degree may be calculated by adding the scores obtained based on the gaze rate, the gaze time, and the gaze order. In other words, the controller 400 may calculate a gaze degree score using at least one of a gaze rate, a gaze time, and a gaze order, for example. In the following description, a case where a higher score is given to a subject having a higher degree of gazing at the time of shooting will be described as an example. However, the embodiment is not limited to this, and in another embodiment, a lower score may be given to a subject having a higher degree of gaze by the photographer at the time of shooting.

  Subsequently, in S514, the control unit 400 selects a representative subject representing the plurality of subjects extracted in S503 based on the calculated score of the gaze degree. The representative subject may be, for example, a subject that is most closely watched by the photographer. For example, as described above, when a score is calculated so that a high score is given to a subject that is being watched, the representative subject may be a subject having the highest score. In step S515, the control unit 400 focuses on the selected representative subject. Further, the control unit 400 may display an image focused on the selected representative subject on the finder of the image display control device 3.

  In S516, the control unit 400 locks the combined focus. In S517, for example, when the control unit 400 receives an input for instructing photographing from the photographer, for example, when the photographer depresses the shutter button, the flow proceeds to S518. In S518, the control unit 400 of the image display control device 3 acquires a plurality of images by focusing on each of the plurality of subjects extracted in S503. In one embodiment, for example, when the image display control device 3 is a monocular camera, the control unit 400 may acquire a plurality of images by continuously shooting. In addition, when the image display control device 3 is a compound eye camera such as a light field camera, the control unit 400 performs a plurality of images by performing focusing after shooting using a technique described in Patent Document 8, for example. May be obtained.

  In S519, the control unit 400 saves the obtained image data in, for example, the storage unit 420, and the flow returns to S501. In one embodiment, the score calculated in S513 for the subject focused on in the image may be stored in the header area of the image data saved in S519. In another embodiment, information that associates a plurality of captured images with a score calculated for a subject that is focused on each of the plurality of images may be stored in the storage unit 420. .

  If the gaze rates of the plurality of subjects are approximately the same in S512 (YES in S512), the flow proceeds to S520. In S520, the control unit 400 calculates the distance from the image display control device 3 to each of the plurality of subjects extracted in S503. In step S521, the control unit 400 calculates the score of each subject based on the calculated distance. For example, it is presumed that a subject closer to the image display control device 3 is more likely to be a target that the photographer wants to photograph. Therefore, the control part 400 may calculate a score so that a score may become high in order from the far side to the near side. For example, the control unit 400 calculates the score for a subject by setting the score of the subject with the longest distance as one point, and assigning a score to the subject that increases one point at a time in order from the subject. Good. When the score is calculated in S521, the flow proceeds to S522.

  In step S522, the control unit 400 selects a representative subject that represents the plurality of subjects extracted in step S503. The representative subject may be, for example, the subject having the shortest distance to the subject. As illustrated above, when the score is calculated so that a high score is given to a subject that is close to the subject, the representative subject is S521. It may be the subject with the highest score calculated in. Subsequently, in S523, the control unit 400 focuses on the selected representative subject. Further, the control unit 400 may display an image focused on the selected representative subject on the finder of the image display control device 3. In S524, the control unit 400 locks the combined focus. In S525, for example, when the photographer receives an input for instructing photographing from the photographer by pressing the shutter button down, the flow proceeds to S526. In step S526, the control unit 400 of the image display control device 3 acquires a plurality of images focused on each of the extracted plurality of subjects.

  In S527, the control unit 400 stores the obtained plurality of image data, for example, in the storage unit 420, and the flow returns to S501. In one embodiment, the score calculated in S521 for the subject focused on the image may be stored in the header area of the image data saved in S527. In another embodiment, information that associates a plurality of captured images with a score calculated for a subject that is focused on each of the plurality of images may be stored in the storage unit 420. .

  In the operation flow of FIG. 5, the control unit 400 of the image display control device 3 functions as, for example, the acquisition unit 411 in the processes of S501 to S510, S514 to S519, and S522 to S527. Further, in the processing of S511 to S513 and S520 to S521, the control unit 400 of the image display control device 3 functions as, for example, the calculation unit 412.

  With the imaging processing according to the first embodiment of FIG. 5 described above, the control unit 400 detects the subject from within the angle of view based on the photographer's line of sight, and captures an image by focusing on the detected subject. . When there are a plurality of subjects detected from within the angle of view, for example, the control unit 400 determines whether or not the gaze rates of the detected plurality of subjects are approximately the same. If the gaze rates of the plurality of detected subjects are not substantially the same, the control unit 400 gazes each of the detected plurality of subjects based on, for example, the gaze rate, the gaze time, and the gaze order. Calculate the degree score. On the other hand, when the gaze rates of the plurality of detected subjects are substantially the same, the control unit 400 calculates a score based on the distances from the image display control device 3 of the detected plurality of subjects. For example, the control unit 400 obtains a plurality of images by focusing on each of the detected plurality of subjects. The calculated score is stored in, for example, header areas of data of a plurality of images acquired by focusing on each of the detected plurality of subjects. As will be described later, the calculated score is used, for example, to display an image focused on a subject that is likely to be desired by the user when displaying a captured image. Can do. Further, as described in the description of S515 and S523, the calculated score may be used, for example, to determine a subject to be focused on in an image displayed on the finder at the time of shooting. By determining the subject to be focused on in the image displayed on the finder at the time of shooting based on the score, for example, the trouble of the photographer selecting the subject to be focused on the finder can be saved.

  Furthermore, the image display control device 3 may be designed so that when the image is captured, the captured image is reduced and displayed on the viewfinder. In this case, in S518, for example, the control unit 400 may display, on the finder, an image photographed with the representative subject selected in S514 in focus among a plurality of photographed images. . Similarly, in S526, the control unit 400 may display, on the finder, for example, an image photographed with a focus on the representative subject selected in S522 from among a plurality of photographed images. .

  Next, display of an image based on the score calculated in the shooting process according to the first embodiment will be described. FIG. 6 is a diagram illustrating an operation flow of image display information output processing based on the score according to the first embodiment. In one embodiment, when an image browsing instruction is input to the control unit 400 of the image display control device 3, the display information output process of FIG. 6 starts.

  In S <b> 601, the control unit 400 of the image display control device 3 sorts the image files stored in the storage unit 420, for example, in the shooting order. In step S602, the control unit 400 determines whether there is an image with a score. For example, in S519 and S527 of the operation flow in FIG. 5 described above, the control unit 400 may write the score in the header of the image file. In this case, the control unit 400 may determine whether there is an image file with a score depending on whether a score is written in the header of the image file in S602. If there is no image file with a score in S602 (NO in S602), the flow proceeds to S603, and the control unit 400 displays display information for displaying the image file sorted in the shooting order, for example, an image display control device. 3 is output to the display device 33 provided in the display 3. When the display information is input, the display device 33 provided in the image display control device 3 may display, for example, image files sorted in the shooting order on the display screen. FIG. 7 is a diagram illustrating display of an image by display information according to the first embodiment. FIG. 7A illustrates a display screen of image files sorted in the shooting order displayed when there is no image file with a score, and all images are sorted and displayed in the shooting order. .

  On the other hand, when there is an image file to which a score is attached in S602 (S602 is YES), the flow proceeds to S604. In step S <b> 604, the control unit 400 sorts the image files in the order of score for each set of a plurality of images that have been scored for a single shooting operation input. In step S603, the control unit 400 outputs display information for displaying the image file rearranged in step S604, for example, to the display device 33 provided in the image display control device 3. Then, when the display information is input, the display device 33 provided in the image display control device 3 may display the image files rearranged in S604 on the display screen, for example.

  FIG. 7B illustrates a display screen displayed when there is an image file with a score. In FIG. 7B, the image A, the image B, and the image C included in the image set 70 are a plurality of images stored in, for example, S519 or S527 with respect to an input of one photographing operation. Therefore, for these multiple images, for example, a score is calculated in S513 or S521. Further, FIG. 7C illustrates the relationship between the ranking of the images A, B, and C included in the set 70 in FIG. In the example of FIG. 7C, the subject focused on the image A is a bird. The subject focused on the image B is a house. The subject in focus in image C is the background. In the example shown in FIG. 7C, the order of photographing is performed in the order of image A, image B, and image C, but the order based on the score is the order of image B, image A, and image C. Note that the order of the scores in FIG. 7C is, for example, that the photographer takes a house that is focused on the image B longer than the bird that is focused on the image A, or It may indicate that he was gazing in early order. The order of the scores in FIG. 7 (c) is, for example, that the photographer focuses the bird focused on the image A longer than the background focused on the image C, or It may indicate that he was gazing in early order. Therefore, the images A to C included in the set 70 are rearranged and displayed in the set 70 as shown in FIG. 7B according to the score ranking shown in FIG. On the other hand, other image files without a score stored in the image display control device 3 are displayed in accordance with the order of shooting.

  FIG. 7D is another example of a display screen for displaying thumbnails of image files displayed when there is an image file with a score. In the example of the display screen shown in FIG. 7D, a plurality of image sets 70 stored in, for example, S519 or S527 are collected for one input of the shooting operation, and the image A included in the set 70 is displayed. -Image B, which is the highest in the ranking of scores on behalf of image C, is displayed. FIG. 7E illustrates a display screen displayed when, for example, the user taps the image B on the display screen in FIG. 7D, and a plurality of images included in the set 70 are displayed. They are displayed according to the score ranking.

  As described above, when there is an image file with a score, display information in which a plurality of images included in the set 70 are rearranged according to the order of scores may be output. Further, in this case, the display device 33 to which the display information is input is exemplified in FIG. 7B, FIG. 7D, and FIG. 7E according to the display information rearranged according to the score order. A display screen may be displayed. That is, the control unit 400 may preferentially display an image having a higher rank based on the score.

  When the control unit 400 outputs display information for displaying the image file to the display device 33 provided in the image display control device 3 in S603, for example, the operation flow ends.

  In the operation flow of FIG. 6, in the processing of S601 to S604, the control unit 400 of the image display control device 3 functions as the output unit 413, for example.

  As described above, according to the first embodiment, a plurality of images are acquired by inputting a single shooting operation while focusing on each of a plurality of subjects detected based on the photographer's line of sight. The Also, the score of the subject is calculated based on the degree of gaze such as the gaze time, the gaze rate, and the gaze order of each subject at the time of image acquisition, or based on the distance from the image display control device 3. And the control part 400 outputs the display information for displaying an image on a display screen based on the calculated score. Therefore, for example, the control unit 400 can preferentially display on the display screen of the display device 33 an image showing a subject that is likely to be taken by the photographer during photographing. Therefore, when the user browses the captured image, the user can reduce the trouble of sequentially viewing the stored image until the image focused on the subject that the photographer wanted to capture is displayed. The usability of the image display control device 3 is improved.

  In the first embodiment, by detecting the line of sight, for example, a subject at a location that has been watched by a photographer for a predetermined time or more is extracted as a subject to be photographed. For this reason, for example, it is possible to reduce the trouble of the photographer individually selecting the photographing target and performing a photographing operation a plurality of times with a focus.

<Second Embodiment>
Subsequently, a second embodiment will be described. When a user captures an image at a certain angle of view, a plurality of subjects may fall within the angle of view. FIG. 8 is a diagram illustrating a plurality of subjects at a certain angle of view of the image display control device 3. In the example shown in FIG. 8A, the subject a: a bird, the subject b: a flower, and the subject c: a background are within the angle of view. Then, as shown in FIG. 8B, the distances from the image display control device 3 to the subject a: bird and the subject b: flower are approximately the same about 1.0 m. Subject c: The distance to the background is 100 m or more.

  In this case, it is conceivable to shoot a plurality of images while focusing on each of the plurality of subjects, but the distances to subject a: bird and subject b: flower are approximately the same, about 1.0 m. Therefore, it is not necessary to individually adjust the focal lengths of the lenses for the plurality of subjects at the same distance. For example, an image focused on a plurality of subjects at the same distance in one photographed image. It is possible to obtain

  FIG. 9 shows an example in which a plurality of subjects within the angle of view are photographed by collecting subjects having substantially the same distance from the image display control device 3 into one image. In FIG. 9A, the subject a: a bird and the subject b: a flower are both focused, and the subject c: the background is blurred. In FIG. 9B, the subject c is focused on the background, and the other subjects are blurred. For example, since the subject a: bird and the subject b: flower are approximately the same distance as described above, they can be photographed collectively, and the subject a: bird and the subject b: flower can be captured in one image. An image focused on both can be taken. Further, by collecting and photographing subjects at approximately equal distances, it is possible to shorten the shooting time and to increase the number of shots. In addition, since the photographing time is shortened, the influence of camera shake can be suppressed.

  Or, for example, when the brightness of a plurality of subjects at substantially the same distance is greatly different, it is preferable to adjust the brightness for each subject even when the focal length is the same. There is. In this case, the control unit 400 of the image display control device 3 may adjust the brightness for each of a plurality of subjects at substantially the same distance, and shoot each image. Also in this case, the control unit 400 does not have to execute the process of separately adjusting the focal length for each of the plurality of subjects at substantially the same distance. For example, the control unit 400 applies to the plurality of subjects at substantially the same distance. On the other hand, the focal length may be adjusted once. Therefore, the shooting time can be shortened. In addition, since the photographing time is shortened, the influence of camera shake can be suppressed.

  Therefore, in the second embodiment, when a photographer shoots a certain angle of view, the control unit 400 controls image display control when a plurality of subjects being watched are extracted by detecting the photographer's line of sight. A determination is made as to whether a plurality of subjects include subjects having substantially the same distance from the device 3. Then, when subjects having substantially the same distance from the image display control device 3 are included, the control unit 400 shoots the subjects by, for example, collecting the number of shots or the number of adjustments of the focal length. Reduce time and increase the number of shots.

  FIG. 10 is a diagram illustrating an operation flow of imaging processing according to the second embodiment. In one embodiment, the imaging process starts when an execution instruction for starting the imaging function in the image display control device 3 is input to the control unit 400.

  Note that the processing from S1001 to S1010 in FIG. 10 may correspond to, for example, the processing from S501 to S510 in FIG. That is, for example, in the processing from S1001 to S1010, the control unit 400 may execute the same processing as the processing from S501 to S510 in FIG. For example, in S1001, the control unit 400 may start line-of-sight detection.

  In S1004, when there are a plurality of subjects extracted in S1003 (S1004 is YES), the flow proceeds to S1011. In step S1011, the control unit 400 executes, for example, a shooting efficiency improvement process.

  FIG. 11 is a diagram illustrating an operation flow of the shooting efficiency improving process according to the second embodiment. In one embodiment, when the process proceeds to S1011 in FIG. 10, the shooting efficiency improvement process in FIG. 11 starts. In S1101, the control unit 400 of the image display control device 3 acquires the brightness of each of the plurality of subjects extracted in S1003. In S1102, the control unit 400 acquires distances from the image display control device 3 to each of the plurality of subjects extracted in S1003. In S1103, the control unit 400 compares the distances to each of the plurality of subjects acquired in S1102, and determines in S1104 whether or not there are subjects that are substantially the same distance. Note that the control unit 400 may determine that there are no subjects at substantially the same distance when the plurality of subjects acquired in S1102 do not include subjects whose distance difference is within a predetermined distance, for example. . If there are no subjects at substantially the same distance (S1104 is NO), the flow proceeds to S1105.

  In step S1105, the control unit 400 sets shooting conditions such that auto exposure (AE: automatic stepwise exposure), autofocus (AF: autofocus), and shooting are performed for each of the plurality of subjects extracted in step S1003. In this case, the number of images to be captured is the same as the number of subjects extracted in S1003.

  On the other hand, if there are subjects at approximately the same distance in S1104 (YES in S1104), the flow proceeds to S1106. For example, when the plurality of subjects acquired in S1102 include a plurality of subjects whose distance difference is within a predetermined distance, the control unit 400 may determine that there are subjects at substantially the same distance. . In S1106, the control unit 400 compares the brightness of a plurality of subjects determined to be substantially the same distance in S1104. If the control unit 400 determines in S1107 that the brightness of the plurality of subjects determined to be substantially the same distance is different (NO in S1107), the flow proceeds to S1108. Note that the control unit 400 may determine that the brightness is different when, for example, the difference in brightness between a plurality of subjects determined to be at substantially the same distance is not within a predetermined range. In S1108, for example, the control unit 400 aggregates AF settings for a plurality of subjects at substantially the same distance at one time, and sets shooting conditions so that AE and shooting are performed for each subject.

  On the other hand, when the control unit 400 determines in S1107 that the brightness of the subjects determined to be approximately the same distance is approximately the same (YES in S1107), the flow proceeds to S1109. Note that the control unit 400 may determine that the brightness is substantially the same, for example, when the difference in brightness between a plurality of subjects determined to be approximately the same distance is within a predetermined range. In step S1109, the control unit 400 aggregates AF and AE settings at one time for subjects that are approximately the same distance and have substantially the same brightness, and aggregates shooting into one image. Set the shooting conditions to do. In this case, the number of images to be captured is reduced according to the number of subjects to be captured collectively. For example, as described above, when the control unit 400 sets shooting conditions in S1105, S1108, or S1109, the operation flow ends, and the flow proceeds to S1012.

  In S1012, the control unit 400 calculates a gaze rate for each of the plurality of subjects extracted in S1003, for example. The gaze rate may be calculated, for example, as a ratio of the time when each of the plurality of subjects extracted in S1003 is gazed after a predetermined time elapses after the photographer's line of sight is detected in S1002. . Subsequently, in S1013, the control unit 400 compares the gaze rates with respect to each subject calculated in S1012 and determines whether or not the gaze rates with respect to each of the plurality of subjects extracted in S1003 are approximately the same. For example, in S1013, the control unit 400 may determine that the gaze rates are the same if the calculated differences in the gaze rates for each subject are all equal to or less than a predetermined value.

  If the gaze rate is different in S1013 (S1013 is NO), the flow proceeds to S1014. In S1014, the control unit 400 of the image display control device 3 calculates a score representing the degree of gaze for each of the plurality of subjects extracted in S1003. The calculation of the score representing the degree of gaze may be executed in the same manner as the example described in S513 in FIG. Note that, for example, in S1109 of FIG. 11 described above, the control unit 400 has set the shooting conditions so that AF, AE, and shooting are aggregated for a plurality of subjects having the same brightness and distance. There is. In this case, the control unit 400 may use the score of the subject with the highest gaze degree score among the plurality of subjects that are collectively photographed as the score for the images that are collectively photographed.

  Subsequently, in S1015, the control unit 400 selects a representative subject representing the plurality of subjects extracted in S1003 based on the calculated score of the gaze degree. The representative subject may be, for example, a subject having the highest gaze degree score. In step S1016, the control unit 400 focuses on the selected representative subject. Further, the control unit 400 may display an image focused on the selected representative subject on the finder of the image display control device 3.

  In step S1017, the control unit 400 locks the combined focus. In S1018, for example, when the control unit 400 receives an input for instructing photographing from the photographer, for example, when the photographer depresses the shutter button, the flow proceeds to S1019.

  In step S1019, the control unit 400 of the image display control apparatus 3 captures a plurality of images focused on each of the plurality of subjects extracted in step S1003, for example. Note that, for example, in S1109 of FIG. 11 described above, the control unit 400 has set the shooting conditions so that AF, AE, and shooting are aggregated for a plurality of subjects having the same brightness and distance. There is. In this case, the control unit 400 sets a value by performing one AF and AE for a plurality of subjects for which shooting conditions are set so as to be aggregated in S1109, and takes a single image. You can do it. Further, for example, in S1108, the control unit 400 may have set the shooting conditions so that AF is aggregated for a plurality of subjects having substantially the same distance. In this case, the control unit 400 sets a value by performing AF once for a plurality of subjects for which shooting conditions are set so as to aggregate AF in S1108. On the other hand, AE and image shooting may be performed for each of a plurality of subjects for which shooting conditions are set so as to aggregate AF. Still further, for example, in step S1105, the control unit 400 may set shooting conditions so that AE, AF, and shooting are performed for each of the plurality of subjects extracted in step S1003. In this case, the control unit 400 may take an image by performing AF and AE on each of the plurality of subjects extracted in S1003.

  In step S1020, the control unit 400 stores the obtained image data in the storage unit 420, for example, and the flow returns to step S1001. In one embodiment, the score calculated in S1014 for the subject focused on in the image may be stored in the header area of the image data stored in S1020. In another embodiment, information that associates a plurality of captured images with a score calculated for a subject that is focused on each of the plurality of images may be stored in the storage unit 420. .

  If the gaze rates of a plurality of subjects are approximately the same in S1013 (S1013 is YES), the flow proceeds to S1021. In S1021, the control unit 400 calculates distances from the image display control device 3 to each of the plurality of subjects extracted in S1003. In step S1022, the control unit 400 calculates the score of each subject based on the calculated distance. For example, it is presumed that a subject closer to the image display control device 3 is more likely to be a target that the photographer wants to photograph. Therefore, the control part 400 may calculate a score so that a score may become high in an order from the near one to the far one. For example, the control unit 400 calculates the score for a subject by setting the score of the subject with the longest distance as one point, and assigning a score to the subject that increases one point at a time in order from the subject. Good. When the score is calculated in S1022, the flow proceeds to S1023.

  For example, in S1109 in FIG. 11, the control unit 400 may have set shooting conditions so that AF, AE, and shooting are aggregated for a plurality of subjects that have substantially the same brightness and distance. . In this case, the control unit 400 may calculate one score corresponding to the distances of a plurality of subjects whose brightness and distance are substantially the same for images that are collected and photographed.

  In S1023, the control unit 400 selects a representative subject representing the plurality of subjects extracted in S1003. For example, the representative subject may be the subject with the shortest distance to the subject. As illustrated above, when the score is calculated so that a subject with a short distance has a high score, the representative subject is S1022. It may be the subject with the highest score calculated in. In step S1024, the control unit 400 focuses on the selected representative subject. The control unit 400 may display an image focused on the selected representative subject on the finder of the image display control device 3.

  In S1025, the control unit 400 locks the combined focus. In S1026, for example, when the photographer receives an input for instructing photographing from the photographer by pressing down the shutter button, the flow proceeds to S1027.

  In S1027, the control unit 400 of the image display control device 3 captures a plurality of images focused on each of the plurality of subjects extracted in S1003, for example. For example, in S1109 in FIG. 11, the control unit 400 may have set shooting conditions so that AF, AE, and shooting are aggregated for a plurality of subjects that have substantially the same brightness and distance. . In this case, the control unit 400 sets a value by performing one AF and AE for a plurality of subjects for which shooting conditions are set so as to be aggregated in S1109, and takes a single image. You can do it. Further, for example, in S1108, the control unit 400 may have set the shooting conditions so that AF is aggregated for a plurality of subjects having substantially the same distance. In this case, the control unit 400 sets a value by performing AF once for a plurality of subjects for which shooting conditions are set so as to aggregate AF in S1108. On the other hand, AE and image shooting may be performed for each of a plurality of subjects for which shooting conditions are set so as to aggregate AF. Still further, for example, in step S1105, the control unit 400 may set shooting conditions so that AE, AF, and shooting are performed for each of the plurality of subjects extracted in step S1003. In this case, the control unit 400 may take an image by performing AF and AE on each of the plurality of subjects extracted in S1003.

  In step S1028, the control unit 400 stores the obtained image data in the storage unit 420, for example, and the flow returns to step S1001. In one embodiment, the score calculated in S1022 for the subject focused on in the image may be stored in the header area of the image data saved in S1028. In another embodiment, information that associates a plurality of captured images with a score calculated for a subject that is focused on each of the plurality of images may be stored in the storage unit 420. .

  In the operation flow of FIG. 10, the control unit 400 of the image display control device 3 functions as, for example, the acquisition unit 411 in the processes of S1001 to S1010, S1011, S1015 to S1020, and S1023 to S1028. In the processes of S1012 to S1014 and S1021 to S1022, the control unit 400 of the image display control device 3 functions as the calculation unit 412, for example. In the operation flow of FIG. 11, in the processing of S1101 to S1109, the control unit 400 of the image display control device 3 functions as the acquisition unit 411, for example.

  With the shooting processing according to the second embodiment shown in FIG. 10 described above, a subject is detected from within the angle of view based on the line of sight of the photographer, and an image is shot while focusing on the subject. If there are a plurality of subjects detected from within the angle of view, the control unit 400 includes, for example, subjects that have substantially the same distance from the image display control device 3 in the detected plurality of subjects. Determine whether or not. In addition, when a subject whose distance from the image display control device 3 is substantially the same is included, the control unit 400 determines whether or not the brightness of the subject having the same distance is substantially the same. When there are subjects whose distance and brightness are substantially the same, the control unit 400 collects and shoots these subjects into one image. When the distance from the image display control device 3 is substantially the same but subjects with different brightness are included, the control unit 400 aggregates the execution of AF for these subjects at once, AE is performed on each subject and an image is taken. Therefore, according to the second embodiment, the number of shots or the number of adjustments of the focal length can be reduced, and the shooting time can be shortened and the number of shots can be improved.

  In addition, the image image | photographed by the imaging | photography process which concerns on 2nd Embodiment of FIG. 10 outputs display information by the display information output process of FIG. 6, for example of the display apparatus 33 with which the image display control apparatus 3 is provided. It may be displayed on the display screen. Thereby, the control part 400 can display an image on a display screen based on a score, when the score is calculated. Therefore, the control unit 400 can preferentially display an image of a subject that is likely to be taken by the photographer at the time of shooting on the display screen. Therefore, when browsing the captured image, the user can save time and effort such as sequentially viewing the stored image until the image focused on the subject that the photographer wanted to capture is displayed. This improves the usability of the image display control device 3.

  Further, for example, the above-described operation flows of FIGS. 5, 6, 10, and 11 are examples, and the embodiment is not limited thereto. For example, when possible, these operation flows may be executed by changing the order of processing, may include additional processing, or some processing may be omitted. . For example, the processing of S1101 and S1102 of FIG. 11 may be executed with the order arbitrarily changed. In another embodiment, when there are a plurality of subjects extracted in S504 of FIG. 5, the operation flow may be changed so as to proceed to S513. In this case, for example, the processing of S511, S512, and S520 to S527 may be omitted. In addition, when the shooting efficiency improvement process of S1011 in FIG. 10 is completed, the operation flow may be changed so that the process proceeds to S1014. In this case, for example, the processing of S1012, S1013, S1021 to S1028 may be omitted.

  In the example shown in FIG. 5 or FIG. 10, the image display control device 3 includes, for example, the image sensor 31 and the line-of-sight sensor 32. The image display control device 3 acquires a plurality of images, and a plurality of images obtained. An example of computing the score of was described. In this case, the control unit 400 of the image display control device 3 may function as, for example, the acquisition unit 411 and the calculation unit 412. However, the embodiment is not limited to this. For example, another device executes the operation flow of FIG. 5 or FIG. 10 to obtain a plurality of images whose scores are calculated, and the image display control device 3 receives the obtained plurality of images to display an image. The control device 3 may execute the operation flow of FIG. In this case, the control unit 400 of the image display control device 3 may not include the acquisition unit 411 and the calculation unit 412.

  FIG. 12 is a diagram illustrating a hardware configuration of a computer 1200 for realizing the image display control device 3 according to the embodiment. The hardware configuration for realizing the image display control device 3 in FIG. 12 may include, for example, a processor 1201, a memory 1202, a storage device 1203, a reading device 1204, a communication interface 1205, and an input / output device 1206. Further, a hardware configuration for realizing the image display control device 3 may include, for example, a display device 1211, a line-of-sight sensor 1212, an image processing unit 1213, an image sensor 1214, and a lens driving device 1215. The processor 1201 to the input / output device 1206 and the display device 1211 to the lens driving device 1215 may be connected to each other via a bus 1208, for example.

  The processor 1201 may be, for example, a CPU (Central Processing Unit). For example, the processor 1201 provides a part or all of the functions of each functional unit described above by executing the program 430 describing the procedure of the above-described operation flow using the memory 1202. For example, the control unit 400 of the image display control device 3 may be the processor 1201, and the storage unit 420 may include, for example, a memory 1202, a storage device 1203, and a removable storage medium 1210. For example, the processor 1201 of the image display control device 3 may function as the acquisition unit 411, the calculation unit 412, and the output unit 413 by reading and executing the program 430 stored in the storage device 1203. For example, the program 430 may be stored in the storage device 1203 of the image display control device 3.

  The memory 1202 is a semiconductor memory, for example, and may include a RAM area and a ROM area. Note that RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory. The storage device 1203 is, for example, a hard disk, a semiconductor memory such as a flash memory, or an external storage device.

  The reading device 1204 accesses the removable storage medium 1210 in accordance with instructions from the processor 1201. The removable storage medium 1210 is, for example, a semiconductor device (USB memory or the like), a medium (such as a magnetic disk) to which information is input / output by a magnetic action, or a medium (CD-ROM, etc.) to which information is input / output by an optical action. For example, a DVD). USB is an abbreviation for Universal Serial Bus. CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk. The communication interface 1205 transmits and receives data by connecting to a peripheral device or a network in a wired or wireless manner in accordance with an instruction from the processor 1201. The input / output device 1206 may be, for example, an input device such as a keyboard or a mouse that receives instructions from the user. The input / output device 1206 may be an output device such as an audio device (for example, a speaker).

  In addition, the display device 1211 may be a display, for example, and the display device 1211 may function as a finder when photographing. The display device 33 in FIG. 3 may be the display device 1211, for example. The line-of-sight sensor 1212 may be, for example, an image pickup device such as a camera, or may be a line-of-sight detection device including an infrared light source and a light receiving element. The line-of-sight sensor 32 in FIG. 3 may be the line-of-sight sensor 1212, for example. The image processing unit 1213 may be an image processing processor such as an ISP (image signal processor). And the control part 400 of the image display control apparatus 3 may be the image process part 1213, for example. The image sensor 1214 may be a camera using, for example, a CCD or a CMOS. The image sensor 31 in FIG. 3 may be an image sensor 1214, for example. The lens driving device 1215 may be, for example, an actuator for driving the lens to change the focal point.

Each program according to the embodiment may be provided to the image display control device 3 in the following form, for example.
(1) Installed in advance in the storage device 1203.
(2) Provided by the removable storage medium 1210.
(3) Provided via a communication interface 1205 from a server such as a program server.

  As described above, some embodiments may be implemented by the processor 1201 included in the image display control device 3 reading and executing the program 430. However, the embodiment is not limited to this. For example, some or all of the above-described functional units included in the image display control device 3 may be implemented as hardware such as FPGA and SoC. Note that FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip. In this case, the control unit 400 may be a circuit implemented as hardware such as FPGA and SoC.

  As another embodiment, for example, the image display control device 3 receives a plurality of images acquired by another device executing the operation flow of FIG. 5 or 10, and the image display control device 3 The operation flow of FIG. 11 may be executed. In this case, the image display control device 3 may not include the display device 1211, the line-of-sight sensor 1212, the image processing unit 1213, the image sensor 1214, and the lens driving device 1215, for example.

  Several embodiments, including those described above, will be understood by those skilled in the art as including various modifications and alternatives of the above-described embodiments. For example, various embodiments may be embodied by modifying components. Various embodiments may be implemented by appropriately combining a plurality of components disclosed in the above-described embodiments. Further, various embodiments may be implemented by deleting or replacing some components from all the components shown in the embodiments, or adding some components to the components shown in the embodiments. May be.

3 Image Display Control Device 400 Control Unit 411 Acquisition Unit 412 Calculation Unit 413 Output Unit 420 Storage Unit 1200 Computer 1201 Processor 1202 Memory 1203 Storage Device 1204 Reading Device 1205 Communication Interface 1206 Input / Output Device 1208 Bus 1211 Display Device 1212 Eye Sensor 1213 Image Processing 1214 Image sensor 1215 Lens drive device

Claims (6)

A plurality of images obtained by focusing on each of a plurality of subjects detected based on the line of sight of the photographer according to a signal instructing photographing is sent to each of the plurality of subjects by the photographer. An output unit that outputs display information for display based on the degree of gaze required from the line of sight toward
An image display control device.   The degree of gaze of the photographer is a gaze time that represents a time during which the photographer turned the line of sight toward each of the plurality of subjects during the photographing, and the photographer directed to each of the plurality of subjects during the photographing. And a gaze rate that represents a proportion of time during which the line of sight was directed, and at least one of a gaze order that represents the order in which the photographer turned the line of sight toward each of the plurality of subjects during the shooting. The image display control device according to claim 1. When the plurality of subjects includes a plurality of subjects whose difference in distance from the image display control device is within a predetermined distance, the difference in distance from the image display control device is within a predetermined distance. An acquisition unit that collects focal lengths for a plurality of subjects at once and acquires the plurality of images;
The image display control device according to claim 1, further comprising:   The acquisition unit determines the distance from the image display control device when the difference in brightness of a plurality of subjects whose difference from the image display control device is within a predetermined distance is within a predetermined range. The image display control apparatus according to claim 3, wherein a plurality of subjects whose differences are within a predetermined distance are aggregated into one image, and the plurality of images are acquired. A plurality of images obtained by focusing on each of a plurality of subjects detected based on the line of sight of the photographer according to a signal instructing photographing is sent to each of the plurality of subjects by the photographer. Output display information for display based on the degree of gaze required from the line of sight
An image display control program for causing an image display control device to execute processing. A plurality of images obtained by focusing on each of a plurality of subjects detected based on the line of sight of the photographer according to a signal instructing photographing is sent to each of the plurality of subjects by the photographer. A step of outputting display information for display based on the degree of gaze required from the line of sight toward
An image display control method executed by the image display control device.