JP7630272B2 - Image processing device and control method thereof, imaging device, program, and storage medium - Google Patents

Description

The present invention relates to subject detection technology in image processing devices and imaging devices.

Conventionally, digital cameras and other devices have been equipped with a tracking AF mode that detects human faces and eyes from images continuously output from an image sensor, and continuously optimizes the focus and exposure conditions for the detected human faces or eyes.

In recent years, it has become possible to use machine learning techniques to detect not only human faces and eyes, but also various other types of subjects (Patent Document 1). With this technique, a specific subject is detected by inputting to a detector both an image and dictionary data that has been trained to detect the subject to be detected. By changing the dictionary data input to the detector, it is possible to detect different types of subjects within an image.

As the number of types of subjects to be detected increases, the number of types of corresponding dictionary data also increases. In devices such as digital cameras, where the number of logical operation circuits constituting the detector and the processing power are limited, a control method can be considered that realizes detection of multiple types of subjects by switching dictionary data every frame. Note that, here, one frame represents the time or cycle until the execution and completion of operation processing and camera control for the image output from the image sensor.

JP 2010-154438 A

However, a control method that switches dictionary data for each frame in which an image is output alone will not allow a detected subject to be detected in a frame in which dictionary data corresponding to that subject is not input. This creates a problem in that stable subject tracking cannot be performed.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an imaging device that can detect multiple types of subjects and can stably track the subjects.

The image processing device of the present invention comprises a memory means for storing multiple dictionary data for detecting multiple different subjects from an image, a detection means for detecting a subject corresponding to a portion of the dictionary data from among the multiple dictionary data for each frame of a multiple frame image obtained by an imaging means, and a switching means for switching the dictionary data used by the detection means in the multiple frames depending on the result of the subject detection by the detection means, wherein the switching means uses the dictionary data for which the corresponding subject has been detected by the detection means in subsequent frames at a shorter detection period than before the subject was detected, and switches the dictionary data used by the detection means so that the combination of dictionary data used by the detection means is switched periodically for each frame .

The present invention makes it possible to provide an imaging device that can detect multiple types of subjects and can stably track the subjects.

1 is a side sectional view showing the configuration of a digital single-lens camera that is a first embodiment of an image pickup apparatus of the present invention. FIG. 2 is a block diagram of a control system of the imaging apparatus. 4 is a flowchart for explaining the operation of one frame of the imaging apparatus. 11 is a flowchart of a dictionary data priority calculation process according to the first embodiment. 5 is a flowchart for determining a dictionary data switching control method according to the first embodiment. 6A and 6B are diagrams showing examples of detection periods of each dictionary data. FIG. 13 is a diagram showing an example of dictionary data control assuming a specific case. 11A and 11B are diagrams showing examples of detection periods of each dictionary data when local dictionary data is defined. 13 is a diagram showing an example of dictionary data control assuming a specific case where local dictionary data is defined. FIG. 13A and 13B are diagrams showing an example of a priority table and a detection period of dictionary data according to the second embodiment. 13A and 13B are diagrams showing examples of dictionary data control assuming specific cases in the second embodiment. FIG. 13 is a diagram showing data on subject detection frequency according to the third embodiment. FIG. 13 is a diagram showing an example of a detection period of dictionary data according to the third embodiment. 11 is a flowchart showing the contents of a continuity check process. 13A and 13B are diagrams showing problems that arise when continuous shooting is started, etc. FIG. 16 is a diagram showing a control concept for solving the problem shown in FIG. 15 . 17 is a flowchart showing a control for realizing the control concept of FIG. 16 . 13A and 13B are diagrams showing the positional relationship of subjects when the subject is reselected as the main subject.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
Fig. 1 is a side cross-sectional view showing the configuration of a digital single-lens camera (hereinafter, also referred to simply as camera) 100 which is a first embodiment of an image pickup apparatus of the present invention. Fig. 2 is a block diagram showing the electrical configuration of the camera 100 of Fig. 1.

As shown in FIG. 1, the camera 100 of this embodiment has a detachable, interchangeable lens unit 120 attached to the front side (subject side) of the camera body 101. The lens unit 120 has a focus lens 121, an aperture 122, etc., and is electrically connected to the camera body 101 via a mount contact portion 123. This makes it possible to adjust the amount of light taken into the camera body 101 and the focal position. The focus lens 121 can also be manually adjusted by the user.

The image sensor 104 is composed of a CCD sensor, a CMOS sensor, or the like, and includes an infrared cut filter, a low-pass filter, and the like. When shooting, the image sensor 104 photoelectrically converts the subject image formed through the shooting optical system of the lens unit 120, and transmits a signal for generating a shot image to the calculation device 102. The calculation device 102 generates a shot image from the received signal, stores it in the image storage unit 107, and displays it on the display unit 105 such as an LCD. The shutter 103 blocks light from the image sensor 104 when shooting is not being performed, and opens when shooting to expose the image sensor 104.

Next, the configuration related to control will be explained using FIG. 2. The calculation device 102 is equipped with a multi-core CPU capable of parallel processing of multiple tasks, RAM, and ROM, as well as dedicated circuits for executing specific calculation processes at high speed. With this hardware, the calculation device 102 is made up of a control unit 201, a main subject calculation unit 202, a tracking calculation unit 203, a focus calculation unit 204, and an exposure calculation unit 205. The control unit 201 controls each part of the camera body 101 and the lens unit 120.

The main subject calculation unit 202 is configured with a dictionary data priority calculation unit 211, a dictionary data switching control unit 212, a detector 213, and a main subject determination unit 214. The detector 213 performs processing to detect specific areas (e.g., a human face or eyes, or a dog's face or eyes) from an image. There are cases where no specific areas are detected, and cases where multiple specific areas are detected. The detection method may be any known method such as AdaBoost or a convolutional neural network. In addition, the implementation form may be a program running on a CPU, dedicated hardware, or a combination of these. In addition, the type of detected subject can be changed by switching the dictionary data input to the detector 213.

The dictionary data is data in which, for example, the characteristics of the corresponding subject are registered, and control commands for the logic circuit are written for each type of subject. In this embodiment, the dictionary data for each subject is stored in the ROM in the calculation device 102 (multiple types of dictionary data are stored). Since the dictionary data exists for each type of subject and each specific area, different types of subjects can be detected by switching the dictionary data. For example, the dictionary data priority calculation unit 211 calculates the priority of the dictionary data for each predetermined period, such as for each frame, and the dictionary data switching control unit 212 determines the dictionary data to be input to the detector 213 based on the calculation result. The subject detection result obtained from the detector 213 is sent to the main subject determination unit 214, and the main subject is determined from the detected subjects and the main subject area is set. The main subject is determined using a known calculation method based on the size, position, reliability of the detection result, etc. If the detector 213 does not detect a subject or a specific area, the main subject area to be the main subject is determined based on past detection results, feature amounts such as edges of the target frame, subject distance, etc.

In this embodiment, the detector 213 is a detector that performs subject detection using a convolutional neural network (CNN), and the dictionary data that the detector uses to detect each subject is learned parameters that have been generated in advance by machine learning of the CNN in an external device (PC) or the imaging device 100.

The machine learning of the CNN may be performed by any method. For example, a specific computer such as a server may perform machine learning of the CNN, and the imaging device 100 may obtain the trained CNN from the specific computer. For example, the CNN of the subject detection unit 204 may be trained by a specific computer performing supervised learning using the image data for training as input and the position of the subject corresponding to the image data for training as teacher data. The CNN of the dictionary estimation unit 205 may be trained by a specific computer performing supervised learning using the image data for training as input and dictionary data corresponding to the subject of the image data for training as teacher data. As a result, trained parameters of the CNN are generated. The CNN may be trained by the imaging device 100 or the image processing device described above.

The detector 213 used in this embodiment has the processing power to detect multiple types of subjects per frame, but does not always detect all types of subjects targeted by the camera of this embodiment within one frame. In other words, this embodiment does not necessarily limit the processing power of the detector 213, but in order to save processing speed and bus bandwidth, it applies detection of fewer than all types of subjects by switching dictionaries according to predetermined settings.

The tracking calculation unit 203 tracks the main subject region based on the detection information of the main subject. The focus control unit 204 calculates the control value of the focus lens 121 for focusing on the main subject region. The exposure calculation unit 205 calculates the control values of the aperture 122 and the image sensor 104 for achieving proper exposure for the main subject region.

The operation unit 106 includes a release switch and a mode dial, and the control unit 201 can receive shooting instructions and mode change instructions from the user through the operation unit 106. This completes the configuration of the camera 100 according to the first embodiment of the present invention.

Next, the operation of one frame of the camera 100 in this embodiment will be described with reference to FIG. 3. The operation of the flowchart in FIG. 3 is repeatedly executed for each frame.

In step S301, the control unit 201 performs a process of reading out pixel signals from the image sensor 104. Image data is generated from the read out pixel signals. After storing the pixel signals and the generated image data in RAM, the process proceeds to step S302.

In step S302, proceed to step S306 only if it is the first frame. Otherwise, proceed to step S303.

In step S303, the tracking calculation unit 203 performs a tracking process to calculate the position in the current frame of the main subject area set in the previous frame using the tracking reference information generated in step S310 (described later) for the previous frame. Any known method may be used as the algorithm used in the tracking process. For example, the tracking process can be performed by searching for an area in the current frame where the feature amounts are close to each other, using the feature amount extracted from the main subject area set in the previous frame by a predetermined feature extraction means as the tracking reference information. In addition, the implementation form of the algorithm may be a program running on a CPU, dedicated hardware, or a combination of these. After the tracking process is completed, the tracking result (the position and size of the main subject area in the current frame) is saved in the RAM in the calculation device 102, and the process proceeds to S304.

In step S304, the focus calculation unit 204 uses the tracking result generated in step S303 and the signal information and image data generated in step S301 to calculate a control value for the focus lens 121 so that the tracking result is in focus. The focus lens control value may be calculated using any known method, such as a contrast method or a phase difference detection method. When calculation of the focus lens control value is complete, the focus calculation unit 204 notifies the control unit 201 of the focus lens control value. The control unit 201 controls the focus lens 121 through the mount contact unit 123 based on the notified control value. After the above processing is completed, proceed to step S305.

In step S305, the exposure calculation unit 205 uses the tracking result generated in step S303 and the signal information and image data generated in step S301 to calculate control values for the image sensor 104 and the aperture 122 so that the exposure of the tracking result is appropriate. Any known method may be used to calculate the control values for the image sensor 104 and the aperture 122. When the calculation of the control values for the image sensor 104 and the aperture 122 is completed, the exposure calculation unit 205 notifies the control unit 201 of the control values for the image sensor 104 and the aperture 122. The control unit 201 controls the image sensor 104 based on the notified control values for the image sensor 104 and the aperture 122, and also controls the aperture 122 through the mount contact unit 123. After the above processing is completed, proceed to step S306.

In step S306, the control unit 201 reads the state of the operation unit 106, and if the release switch is pressed, the process proceeds to step S307. Otherwise, the process proceeds to step S308.

In step S307, the control unit 201 performs imaging processing for a still image. Based on the control value calculated in step S305, the shutter 103 is driven to expose the image sensor 104. Image data is generated from pixel signals read from the exposed image sensor 104. The generated image data is stored in an external storage medium such as an SD card. After the above processing is completed, the process proceeds to step S308.

In step S308, the priority of each dictionary data is calculated. Details of this process will be described later with reference to FIG. 4. Next, proceed to step S309.

In step S309, a dictionary data switching control method is determined based on the calculated priority of each dictionary data. Details of this process will be described later with reference to FIG. 5. Next, the process proceeds to step S310.

In step S310, the image data generated in step S301 and the dictionary data are input to the detector 213, and a specific area of the subject corresponding to the dictionary data is detected. The input dictionary data follows the dictionary data switching control method determined in step S309. Next, proceed to step S311.

In step S311, the main subject region is determined from the detection result obtained in step S310, for example, using the method described above. The main subject region is determined using a known calculation method based on the part of the detection result (e.g., whole body, face, eyes, etc.), size, position, and reliability of the detection result. Once information on the determined main subject region is saved in the RAM in the computing device 102, the process proceeds to step S312.

In step S312, the tracking calculation unit 203 generates tracking reference information to be used in the tracking process of step S303 for the next frame based on the information on the main subject region generated in step S311 and the image data generated in step S301. The generated tracking reference information is then stored in the RAM in the calculation device 102.

When step S312 is completed, the operation of one frame of the camera 100 is completed. As long as the camera 100 continues to operate, the process returns to step S301 and repeats the process described up to this point. This concludes the description of the operation of one frame of the camera 100 in this embodiment.

Next, the method for calculating the priority of dictionary data in step S308 of FIG. 3 will be explained using FIG. 4.

In step S400, the priority of each dictionary data set in the previous frame is initialized.

In step S401, it is determined whether a main subject region has been output in the previous frame by referring to main subject region information stored in the RAM in the calculation device 102. If a main subject region has been detected, the process proceeds to step S402, and if no subject has been detected, the process proceeds to step S405.

In step S402, the priority of the dictionary data corresponding to the main subject detected in the previous frame is set to "high." In FIG. 4, the priority of the dictionary data is set to two levels, "high" and "low." The method of setting the priority differs depending on the number of dictionary data and the processing power of the detector (types and number of times that can be detected per frame).

In step S403, it is determined whether local dictionary data is defined for the main subject. Depending on the subject, the detection area may be defined as a whole area and a local area that is a part of the whole area. Specifically, for example, the relationship is as follows:

(For humans) Whole area: Face Local area: Pupils (For dogs) Whole area: Whole body/face Local area: PupilsIn this embodiment, detection of a local area (specific area) based on the dictionary data is not performed from the entire captured image, but is performed by inputting an image obtained by cutting out the whole area of the detected subject and its vicinity from the original captured image to the detector 213. In other words, if the target subject is a person, the pupils, which are a local area, are detected only after the human face is detected. If there is corresponding local dictionary data, the process proceeds to step S404, and if there is no local dictionary data, the process proceeds to step S405.

In step S404, the priority of the local dictionary data corresponding to the main subject is set to "high," and the process proceeds to step S405.

In step S405, it is determined whether there is any dictionary data for which the priority has not been determined. If there is any dictionary data for which the priority has not been determined, the process proceeds to step S406; if not, the dictionary data priority calculation process in step S308 ends.

In step S406, the priority of dictionary data for which the priority has not been determined is set to "low."

Next, the method for determining the dictionary data switching control method in step S309 of FIG. 3 will be explained using FIG. 5.

In step S500, the detection period of each dictionary data determined in the previous frame is initialized. Here, the detection period is a parameter [frame/detection] indicating how many frames a single detection result is obtained. The unit of the detection period is not limited to this, and may be set, for example, by time [ms/detection]. The method of setting the dictionary data used by the detector 213 to detect the subject for multiple frames is not limited to the above. For example, a plurality of data tables describing which combination of dictionary data the detector 213 uses in each frame may be stored in advance in the ROM in the computing device 102. The control unit 201 selects a table according to a user's setting of which subject to prioritize for detection of the subject (person priority, animal (dog, cat, bird) priority, vehicle (2-wheel, 4-wheel) priority, etc.), and expands it in the RAM. The detector 213 refers to the dictionary data in the order of the table stored in the RAM, and detects the subject corresponding to each dictionary data. The table indicating the input order of the dictionary data expanded in the RAM is rewritten at any time according to the detection situation of the subject. For example, if a specific subject such as a bird is detected, the dictionary data corresponding to the bird's eyes is rewritten so that they are detected with priority, so that the bird's eyes can be detected next.

In step S501, it is determined whether there is any dictionary data with a "high" priority for which the detection period has not been set. If such dictionary data is present, the process proceeds to step S502, where the detection period for that dictionary is set to 1 [frame/detection] (selected as dictionary data for which detection results are obtained for each frame). Steps S501 and S502 are repeatedly executed until detection periods have been set for all dictionary data with a "high" priority. Next, the process proceeds to step S503.

In step S503, the detection period for the remaining dictionary data with a "low" priority is set to multiple frames. Specific examples of detection periods for dictionary data with a "low" priority will be described later with reference to Figures 6 and 7. Next, the process proceeds to step S504.

In step S504, a control schedule for the dictionary data is determined based on the detection period of each dictionary data and the number of times detection is possible per frame determined in the previous steps. Specific examples of the control schedule for the dictionary data will be described later with reference to Figures 6 and 7.

Next, a specific example of dictionary data control will be described with reference to Figs. 6 and 7. Fig. 6(a) shows the types of dictionary data, detection areas, and detector processing constraints in the control case assumed in this embodiment. In this embodiment, six types of dictionary data are prepared to detect subjects classified as people, dogs, cats, birds, four-wheeled vehicles, and two-wheeled vehicles. In this embodiment, the logic circuit block that serves as the detector is one block, and detection can be performed three times in the time it takes to process one frame of image signals. In other words, it is possible to process three types of dictionaries per frame. Four-wheeled vehicles refer to dictionary data for detecting four-wheeled vehicles, and examples of vehicles include racing vehicles such as rally cars with four wheels, as well as general passenger cars. Two-wheeled vehicles refer to dictionary data for detecting two-wheeled vehicles, and examples of vehicles include motorbikes and bicycles.

Under the conditions in FIG. 6(a), the priority and detection period of each dictionary data are determined by steps S308 and S309 in FIG. 4 and FIG. 5. The results are shown in FIG. 6(b). FIG. 6(b) shows the detection period of each dictionary determined according to the detected main subject.

If no subject is detected, the detection period for all dictionary data is set to 2 [frame/detection]. On the other hand, if a person is detected as the subject, the detection period for only the dictionary data for people is set to 1 [frame/detection]. Similarly, if other subjects are detected, the detection period for only the dictionary data corresponding to the detected subject is set to 1 [frame/detection].

Next, an example of dictionary data control assuming a case where a dog 703 is detected in the middle of consecutive frames is shown using FIG. 7. In FIG. 7, 701 represents the dictionary data scheduled for each frame, and 702 represents the captured image input to the detector 213.

In the example of FIG. 7, since no subject is detected in the first to fifth frames, the detection cycle for all dictionary data is set to 2 [frame/detection]. In the fourth frame, a dog 703 appears in the frame, and in the middle of the fifth frame, a dog's face 704 is output as a detection result, and the dog is determined to be the main subject. From the sixth frame onwards, the detection cycle for the dog dictionary data is controlled at 1 [frame/detection], and the other dictionary data (people, cats, birds, four-wheeled vehicles, two-wheeled vehicles) is controlled at 2 [frame/detection]. From the sixth frame onwards, dictionary data corresponding to the detected main subject is input to the detector 213 in all frames (every frame from this frame onwards), making it possible to stably track the subject. In addition, since dictionary data other than the main subject is also input to the detector, even if a type of subject other than the main subject appears in the captured image, it is possible to detect it and change the main subject to another type of subject.

Next, an example of dictionary data control when local dictionary data is present will be described. Figure 8(a) shows the types of dictionary data and the processing constraints of the detector. Figure 8(b) shows the detection cycles of each dictionary data and local dictionary data determined by steps S308 and S309 in Figures 4 and 5. When a subject for which local dictionary data is defined is detected as the main subject, the local dictionary data is also set to a "high" priority and is controlled at a detection cycle of 1 [frame/detection]. Therefore, the detection cycle of dictionary data corresponding to subjects other than the main subject is relatively reduced.

Figure 9 shows an example of dictionary data control using the dictionary data detection cycle determined by Figure 8. A dog is detected in the fifth frame shown in Figure 9, and the dictionary data control method is changed from the sixth frame onwards. From the sixth frame onwards, the corresponding local dictionary data is also input to the detector at a detection cycle of 1 [frame/detection], and the dog's local region, the eye 905, is detected.

As described above, in this embodiment, in a device with a small number of logic operation circuits constituting a detector compared to the number of dictionary data and with poor processing power, the priority of each dictionary data to be used for each frame is dynamically calculated according to the subject detection situation. Then, by changing the control method of the dictionary data, it is possible to detect multiple types of subjects, while enabling stable subject tracking after a specific subject is detected.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, the priority of each dictionary data other than the detected main subject is relatively lowered uniformly. In this embodiment, the priority is calculated for each dictionary data other than the main subject, and it is determined which dictionary data is to be preferentially input to the detector 213. It is preferable to determine the priority of each dictionary data in accordance with an actual use case.

For example, if a dog is the main subject, it is thought that the subject that is most often photographed together with a dog is a person. Possible scenes include a walk, a person holding a dog, or a person playing with a dog, so the priority of the dictionary data for people should be set high. On the other hand, it is thought that there are few use cases in which a dog is photographed together with a four-wheeled or two-wheeled vehicle, so the priority of the dictionary data for four-wheeled or two-wheeled vehicles should be set low.

In actual control, based on this concept, the priority of each dictionary data is prepared in advance as a design table stored in ROM, and the priority of the dictionary data is determined by referring to this table depending on the state of the detected subject.

Figure 10(a) shows a specific example of a dictionary data priority table. Here, the priority of dictionary data is expressed in five stages from 5 (high) to 1 (low). Based on this table, the detection period for each dictionary data is calculated and a dictionary data switching control method is determined. Figure 10(b) shows an example of the detection period for each dictionary data determined based on the priority in Figure 10(a).

Next, FIG. 11 shows an example of dictionary data control assuming a case where a dog 1103 is detected in the middle of consecutive frames. A dog 1103 is detected as a subject in the fifth frame, and dictionary data control is changed from the sixth frame onwards. Dictionary data corresponding to the dog's eyes, which are a local region of the dog, is input to the detector from the sixth frame onwards, and the dog's eyes 1105 are detected. The detection cycle for dictionary data other than the dog is scheduled according to FIG. 10(b).

As described above, in this embodiment, the priority of each dictionary data is determined in accordance with the actual use case. This makes it possible to detect various types of subjects more quickly and to change the main subject more quickly, even in a device with a limited number of logic operation circuits constituting the detector and limited processing power.

Third Embodiment
Next, a third embodiment of the present invention will be described. In the third embodiment, the priority calculation step for each dictionary data shown in Fig. 3 is performed according to the past detection history. This embodiment differs from the first embodiment only in steps S308, S309, and S311 in Fig. 3, and the other processes are the same as those in the first embodiment.

In the process of determining the main subject from among the detected subjects in step S311, a process is added in which the detection information of subjects detected in the same frame (simultaneously) as the main subject is accumulated for each subject type and stored as subject detection frequency data.

Figure 12 is a bar graph showing an example of subject detection frequency data for each subject type, and indicates the detection frequency of other types of subjects being detected in the same frame when a specific subject is detected as the main subject. As in the first embodiment, the subjects to be detected are people, dogs, cats, birds, four-wheeled vehicles, and two-wheeled vehicles, and the bar graph shows the detection frequency of other subjects when each of these subjects is detected as the main subject. Here, the detection frequency of the type of subject itself detected as the main subject is not included, and is therefore indicated as N/A (Not Applicable).

Next, in step S308, the subject detection frequency data accumulated in the previous step S311 is referenced, and the priority of each dictionary data for the current subject detection is determined according to the subject type detected as the main subject at that time. The priority of dictionary data for subject types that are detected frequently is set to high, and the priority of dictionary data for subject types that are detected infrequently is set to low.

Next, in step S309, the detection period for each dictionary data is determined based on the priority of the dictionary data that has already been determined. FIG. 13 shows an example of the determined detection period for each dictionary data.

As described above, in this embodiment, the priority of each dictionary data is determined in accordance with past subject detection frequency data, making it possible to perform dictionary data control suited to the user. Therefore, even in a device with a limited number of logic operation circuits constituting a detector and limited processing power, it is possible to detect various types of subjects more quickly and to change the main subject more quickly.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In subject detection, in order to prevent erroneous detection, an algorithm is used in which a subject is detected consecutively a certain number of times or more before the detection result is adopted (hereinafter, referred to as continuity check).

However, in shooting modes where immediacy is required for display and tracking control during continuous shooting, continuity checks are not performed, and if a subject is detected even once, the result is immediately adopted as the main subject and reflected in tracking control, dictionary data control, and frame display.

However, there is a possibility of false positives if only one detection result is made. For example, there is a possibility of false positives, such as detecting part of a two-wheeled vehicle as a dog. Even if there is a false positive, if the target subject continues to be detected continuously, there will be no major problems with tracking control. However, false positives often result in an unstable detection state, increasing the risk of problems occurring with subject tracking.

In this embodiment, therefore, in a shooting mode that does not perform a continuity check, such as a continuous shooting mode, all types of dictionary data continue to operate even after a subject is detected once and adopted as the main subject, and if it is detected continuously a certain number of times or more, it is re-adopted as the correct main subject. The tracking control, dictionary data control, and frame display control are also changed to match the re-adopted main subject.

The above continuity check is performed in the main subject determination process of S311 in FIG. 3. The continuity check process is described in FIG. 14. As explained in FIG. 3, the flowchart in FIG. 3 is repeated for each frame, and therefore the operation of the flowchart in FIG. 14 is also performed for each frame.

In step S1401, it is checked whether or not there is a subject detection result in the input image. If there is a subject detection result, the process proceeds to step S1402. If there is no subject detection result, the number of consecutive detections is initialized to 0 in step S1403, and the detection result is not adopted (step S1407).

Next, in step S1402, it is checked whether the number of consecutive detections is 0 or whether the subject ID detected in the previous frame is the same as the subject ID detected in the current frame. Correlation processing between subjects between frames is performed before the continuity check processing, and subjects are linked by subject ID. If the number of consecutive detections is 0, this is the first detection result, so the process proceeds to step S1404. Alternatively, if the previous subject ID and the current subject ID match, the process similarly proceeds to step S1404. For example, even if the number of consecutive detections is 1 or more, if the condition that the previous subject ID and the current subject ID match is met, the process proceeds to step S1404. If none of the above conditions are met, the process proceeds to step S1403.

In step S1403, the number of consecutive detections is initialized to 0, and in step S1407, the detection result in this case is not adopted.

In step S1404, the number of consecutive detections is incremented by 1, and the process proceeds to step S1405. The number of consecutive detections is stored in a memory or the like each time the operation in FIG. 14 is performed once. In step S1402 in FIG. 14, if the condition that the immediately preceding subject ID and the current subject ID match continues for multiple frames, the number of consecutive detections is counted up in the memory.

Next, in step S1405, it is checked whether the number of consecutive detections is equal to or greater than the threshold value for the number of consecutive detections. In a shooting mode that requires immediacy, such as continuous shooting, the threshold value for the number of consecutive detections is set to 1 (continuous shooting mode), and in other cases, the threshold value for the number of consecutive detections is set to a number greater than 1 (normal mode). If the number of consecutive detections exceeds the threshold value for consecutive detections, the process proceeds to step S1406, where the detection result at that time is adopted and is adopted as the main subject after the continuity check process in FIG. 14.

If the number of consecutive detections is less than the threshold number of consecutive detections, proceed to step S1407 and do not use the detection result at that time.

Next, using Figure 15, we will explain the issues that this embodiment addresses when starting continuous shooting, etc.

In FIG. 15, images 1501 and 1502 show input images of two wheels, and frame 1503 represents the AF frame. The detection and tracking states for each image frame are also displayed below the input image. In input image 1501, when continuous shooting begins without a main subject, subject detection is performed in the AF frame 1503, and thereafter the detected subject is tracked across the entire angle of view. Data 1504 represents the dictionary data input in each image frame. In this example, it is assumed that three dictionary data can be set for one frame.

In Figure 15, part of a two-wheeled vehicle is erroneously detected as a dog 1505 in the first frame, and from the second frame onwards, the detection result of the dog 1505 is selected as the main subject and tracking is performed. The correct detection result in this image is two wheels, but from the second frame onwards, control is performed to increase the priority of the dictionary data for dogs, so dictionary data for two-wheeled and four-wheeled vehicles is not set. Therefore, the target subject cannot be re-detected as two wheels. The control concept in this embodiment that solves this problem is shown in Figure 16.

In FIG. 16, even if part of two wheels is detected as a dog 1505 in the first frame, dictionary data control is not performed according to the dog selected as the main subject for a certain period from the second frame onwards, and dictionary data control is performed in the same way as when there is no main subject (dictionary data of all types is set). In FIG. 16, the detection result of two wheels 1506, which is a correct detection, is obtained multiple times from the second frame onwards, and the number of consecutive detections exceeds the threshold for the number of consecutive detections (set to four times in FIG. 16) in the eighth frame. Then, from the next ninth frame, the main subject is changed from the dog to two wheels, and dictionary data control is also changed to match the two wheels. By re-adopting the new subject detection results that have been detected multiple times as the main subject, it is possible to correct erroneous detections while maintaining immediacy during continuous shooting, and to reduce the risk of erroneous tracking.

Next, FIG. 17 shows a control flow for realizing the control concept of FIG. 16. The control flow of FIG. 17 is performed in the main subject determination process of step S311 of FIG. 3. As explained in FIG. 3, the flowchart of FIG. 3 is repeatedly performed for each frame, and therefore the operation of the flowchart of FIG. 17 is also performed for each frame.

In FIG. 17, step S1701 determines whether or not a main subject is already present. If a main subject is already present, the process ends. If no main subject is present, the process proceeds to step S1702, where it is determined whether or not the continuous shooting mode is selected.

If the continuous shooting mode is not selected, the continuity check can be performed in normal mode, and the program proceeds to the continuity check in step S1706 (normal mode: number of continuous detections = 4). If the continuous shooting mode is selected, a check is made in step S1703 to see if there is a tentative main subject. A tentative main subject here refers to a subject that is selected as the main subject without going through a continuity check (normal mode: number of continuous detections = 4), such as the dog that was the main subject in frames 2 to 8 in FIG. 16. If this subject is the tentative main subject, it will be a tracking target, but the dictionary data control is the same as the dictionary data control when there is no main subject, in which all types of dictionary data are set uniformly.

If there is no provisional subject in step S1703, a continuity check is performed in step S1704 (continuous shooting mode: number of continuous detections = 1), and if there is a detection result in step S1705, it is adopted as the provisional main subject.

Next, in step S1709, the number of provisional main subject frames is initialized to 0, and processing ends. The number of provisional main subject frames is the number of image frames counted since the provisional main subject was set.

Also, if a provisional subject is found in step S1703, proceed to step S1708.

In step S1708, it is determined whether the number of provisional main subject frames exceeds the threshold value for provisional main subject frames. If the number of provisional main subject frames exceeds the threshold value for provisional main subject frames, it is determined that there is no subject more likely than the provisional main subject, and in step S1710, the provisional main subject is changed to the main subject and the process ends. Note that in this flowchart, the number of provisional main subject frames is stored in a memory or the like each time the operation in FIG. 17 is performed once, and the number of provisional main subject frames is counted up.

If the number of tentative main subject frames is equal to or less than the tentative main subject frame threshold, proceed to step S1706. The tentative main subject frame threshold is set to limit the control state by the tentative main subject to a certain period of time, and its value is set to at least equal to or greater than the threshold for the number of consecutive detections.

Next, a continuity check (normal mode) is performed in step S1706, and if a detection result that exceeds the threshold for the number of consecutive detections is obtained, in step S1707, that detection result is adopted as the main subject, and the process ends.

The above-mentioned series of controls can also be applied when the main subject being tracked during continuous shooting is lost and a new subject needs to be detected again.

Furthermore, only when the detection area of the subject to be reselected and its neighboring area overlap with the detection area of the provisional subject, the subject is reselected as the main subject. Figure 18 shows the positional relationship of the subjects. Reference numeral 1801 denotes the image area, 1802 denotes the detection area of the provisional main subject, 1803 denotes the detection area of the subject to be reselected, and 1804 denotes the neighboring area of detection area 1803.

In FIG. 18(A), there is no area that overlaps with detection area 1802, so detection area 1803 is not selected as the new main subject.

In Figures 18(B) and 18(C), there is an area that overlaps with detection area 1802, so detection area 1803 is selected as the new main subject. Nearby area 1804 is set depending on the type of subject to be reselected and the amount of movement up to that point. For example, if the subject to be reselected is an animal, it is expected that it will move quickly and move a lot over several frames, so nearby area 1804 is set relatively wide. On the other hand, if the subject to be reselected is a person, it is unlikely that they will move a lot over several frames, so one possible setting method is to make the nearby area relatively narrow.

This reduces the risk of accidentally selecting a different subject in the surrounding area.

As described above, in this embodiment, in a shooting mode that does not perform a continuity check, such as continuous shooting mode, all types of dictionary data continue to operate even after a subject is detected once and adopted as the main subject, and if it is detected continuously a certain number of times or more, it is re-adopted as the correct main subject. The tracking control, dictionary data control, and frame display control are also changed to match the re-adopted main subject. In this way, it is possible to suppress a decrease in tracking performance due to erroneous detection while ensuring the immediacy of tracking control and frame display control required in continuous shooting mode.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

The present invention is not limited to use with digital cameras or digital video cameras, but can also be installed in imaging devices such as surveillance cameras, web cameras, and mobile phones.

102: Calculation device, 104: Image sensor, 202: Main subject calculation unit, 203: Tracking calculation unit, 211: Dictionary data priority calculation unit, 212: Dictionary data switching control unit, 213: Detector, 214: Main subject determination unit

Claims (18)

A storage means for storing a plurality of dictionary data for detecting a plurality of different subjects from an image;
a detection means for detecting an object corresponding to a part of the dictionary data by using a part of the plurality of dictionary data for each frame of a plurality of frames of images obtained by an imaging means;
a switching means for switching dictionary data used by the detection means in the plurality of frames in response to a result of the object detection by the detection means;
Equipped with
The image processing device is characterized in that the switching means uses dictionary data corresponding to a subject detected by the detection means in subsequent frames at a shorter detection period than before the subject was detected, and switches the dictionary data used by the detection means so that the combination of dictionary data used by the detection means is switched periodically for each frame . The image processing device according to claim 1, characterized in that the detection means detects a main subject from the image. 3. The image processing apparatus according to claim 2, wherein the main subject is a subject on which the image capturing means adjusts at least one of focus and exposure.
4. The image processing device according to claim 1, wherein the storage means stores a plurality of types of tables for switching between the plurality of dictionary data for each frame, and the switching means switches the dictionary data to be used by the detection means using a table that has been set in advance by a user from among the plurality of types of tables . The image processing device according to any one of claims 1 to 4, characterized in that the switching means switches the dictionary data used by the detection means to use second dictionary data corresponding to the detection of a local area of the subject corresponding to the first dictionary data in response to the detection of a corresponding subject using the first dictionary data. 6. The image processing apparatus according to claim 5, wherein the switching means switches the dictionary data used by the detection means so that, when the second dictionary data is present, the second dictionary data is preferentially used together with the first dictionary data. 7. The image processing apparatus according to claim 1, wherein the switching means has a priority table for determining priorities of the plurality of dictionary data in accordance with the subject detected by the detection means. 8. The image processing apparatus according to claim 7 , wherein the switching means determines the priorities of the plurality of dictionary data based on the priority table. 9. The image processing apparatus according to claim 1, wherein the switching means stores frequency data that is a cumulative count of the frequency of a type of object being detected simultaneously with a specific object. 10. The image processing apparatus according to claim 9 , wherein the switching means determines priorities of the plurality of dictionary data based on the frequency data. 11. The image processing apparatus according to claim 1, wherein the plurality of dictionary data are convolutional neural networks trained by machine learning. The image processing device according to any one of claims 1 to 11, characterized in that the switching means performs switching control of dictionary data so that, even if a first subject is detected and selected as a main subject in a specific shooting mode, if a second subject other than the first subject is detected consecutively a predetermined number of times or more , the second subject is re-selected as the main subject. 13. The image processing apparatus according to claim 12 , wherein the switching means performs the switching control even when the main subject being tracked is lost in the specific photographing mode and the subject is redetected. 13 . The image processing device according to claim 12 , wherein the switching means reselects the main subject when the detection area of the second subject and an area including the vicinity thereof overlap with the detection area of the first subject. An imaging means for capturing an image;
An image processing device according to any one of claims 1 to 14 ,
An imaging device comprising: a detection step of detecting, for each frame of a plurality of frames of images obtained by the imaging means, a subject corresponding to a portion of a plurality of dictionary data for detecting a plurality of different subjects from the image, using the portion of dictionary data;
a switching step of switching dictionary data used in the detection step in the plurality of frames according to a result of the detection of the object in the detection step;
having
A control method for an image processing device, characterized in that in the switching process, dictionary data corresponding to a subject detected in the detection process is used in subsequent frames at a shorter detection period than before the subject was detected, and the dictionary data used in the detection process is switched so that the combination of dictionary data used in the detection process is switched periodically for each frame . A program for causing a computer to execute each step of the control method according to claim 16 . A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 16 .

Images