JP6685129B2 - Image processing apparatus, program, and image processing method - Google Patents

Description

  The present invention relates to an image processing device, a program, and an image processing method, and more particularly, to an image processing device, a program, and an image processing method for creating a photo album from a plurality of images taken by, for example, a digital camera.

  In recent years, a service for automatically laying out images taken by a user to create an album has started to be provided. Here, the automatic layout means that a part or all of the functions related to album creation are automatically processed. For example, (1) images are automatically selected for publication in the album, (2) templates to be used in the album are automatically determined, and (3) images are automatically assigned to the templates.

  When the selected image is an image including a person, it is preferable that the image includes an important person. Many users may want to create an album with "a souvenir picture of themselves" rather than "a picture of a passerby by chance," even if the composition and background of the person are the same. Is high. There is a personal recognition technique as a technique necessary for automatically making such a judgment.

  For example, Patent Document 1 discloses a configuration in which a dictionary is newly created and matching processing and registration processing are sequentially performed on the detected face information. Patent Document 1 describes that a dictionary that is optimal for each user is formed by newly creating a dictionary in this way.

JP, 2014-44606, A

  However, when the contents of the face dictionary are sequentially updated as in Patent Document 1, there are cases where the “reproducibility” of the personal recognition result cannot be maintained. For example, when face detection is performed by multithreads, the start / end timing of each thread varies depending on the task usage status. As a result, even if the personal recognition process itself is running in only one thread, the personal recognition result changes if the input order of the face information input to the one thread varies.

  A change in the personal recognition result leads to a change in the automatic layout result. When a user creates an album using the same image many times, it is not preferable that an album with a different layout result is created each time.

  The present invention has been made in view of the above conventional example, and an object thereof is to provide an image processing apparatus, an image processing method, and a program that can realize the same layout configuration for the same image.

  In order to achieve the above object, the image processing apparatus of the present invention includes the following configurations.

That is, in an image processing apparatus that processes a plurality of image data, an acquisition unit that acquires a plurality of image data and information of a face portion of each of the plurality of image data acquired by the acquisition unit are processed by multithread processing. a detecting means for detecting that, on the basis of being the sort result obtained by sorting the information of the detected portion of the plurality of image data each face by said detecting means, the portion of the plurality of image data each face determining means for determining the order of execution of recognition processing for information, in accordance with the execution order determined by said determining means, and having an execution unit configured to execute the recognition processing for the portion of the detected face.

  According to another aspect of the present invention, there is provided a program that causes a computer to function as each unit of the apparatus having the above configuration.

Looking further present invention from another aspect, there is provided an image processing method in an image processing apparatus for processing a plurality of image data, an acquisition step of acquiring a plurality of image data, the acquired plurality of image data for each A detection step of detecting face part information by multithread processing, and a plurality of images based on a sorting result obtained by sorting the face part information of each of the detected plurality of image data a determining step of determining the order of execution of recognition processing for the data information portion of each face, in accordance with the execution order in which the determined, to have an execution step of executing the recognition processing for the portion of the detected face A characteristic image processing method is provided.

  Therefore, according to the present invention, the same layout configuration can be obtained by using the same image data when the specific processing is executed by using the information obtained by detecting the specific portion of each image. .

It is a block diagram which shows the hardware constitutions of an image processing apparatus. It is a block diagram showing a software configuration of an album creating application. It is a figure which shows the display screen which an album production application provides. It is a flowchart which shows an automatic layout process. It is a figure which shows the image analysis information distinguished for every ID. It is a figure which shows the result of the scene division of an image data group. It is a figure which shows the average value and standard deviation for every scene. It is a figure which shows the concept of a scoring axis. It is a figure for demonstrating selection of image data. It is a figure which shows the template group used for the layout of image data. 4 is a block diagram showing a detailed configuration of a person information analysis unit 205. FIG. It is a figure which shows the example of a data table. 6 is a flowchart showing face detection processing executed by a person information analysis unit according to the first embodiment. 6 is a flowchart showing a personal recognition process executed by a person information analysis unit according to the first embodiment. It is a block diagram which shows the detailed structure of an image scoring part. It is a flowchart which shows the scoring process after the scene classification which an image scoring part performs. It is a figure which shows the selected template. 9 is a flowchart showing processing executed by an automatic layout processing unit of the album creating application according to the second embodiment. 9 is a flowchart for executing a process in page units according to the second embodiment by multithreading. It is a figure which shows the example of a 2-page spread album. 8 is a flowchart showing details of album replacement processing according to the second embodiment. 9 is a row chart showing face detection processing executed by a person information analysis unit according to the third embodiment. 9 is a flowchart showing a personal recognition process executed by a personal information analysis unit according to the third embodiment.

  Hereinafter, embodiments of the present invention will be described more specifically and in detail with reference to the accompanying drawings. However, the relative arrangement and the like of the constituent elements described in this embodiment are not intended to limit the scope of the present invention thereto only unless otherwise specified.

  In this specification, "recording" (sometimes referred to as "printing") is not limited to the case of forming significant information such as characters and figures, but also meaningless. Furthermore, the case where an image, a pattern, a pattern, or the like is widely formed on a recording medium or the medium is processed, regardless of whether or not it is actualized so that it can be visually perceived by humans.

  The term "recording medium" means not only paper used in a general recording apparatus but also a wide range of materials such as cloth, plastic film, metal plate, glass, ceramics, wood, and leather that can accept ink. I shall.

<Embodiment 1>
In this embodiment, an example in which an album creating application (hereinafter, application) is operated on the image processing apparatus to generate an automatic layout will be described. The feature of this embodiment is that the face detection process can be executed by multithread.

  As pointed out in the above-mentioned problem, when the contents of the face dictionary are sequentially updated according to the collation result of the face dictionary and the detected face information, the "reproducibility" of the personal recognition result may not be maintained. The reproducibility of the personal recognition result largely depends on the face dictionary formation process. It is a problem of clustering that the contents of the dictionary change due to the change of the input order of the information to be collated / registered.

  For example, among the face information A, B, and C, A and C are not similar to each other, and B is similar to A and C to some extent. If the personal recognition processing is executed in the order of the face information A, B, and C, B is collated with the dictionary while A is registered in the dictionary, and it is determined that the face information A and B are similar. On the contrary, if the personal recognition processing is performed in the order of face information C, B, and A, B is collated with the dictionary while C is registered in the dictionary, and it is determined that the face information B and C are similar. . In this way, the face information B can be judged as the same person by either A or C depending on the existing dictionary information. When the face detection operates in a multi-threaded manner, the detection order of the face information is different each time. Therefore, in the configuration of Patent Document 1, the face recognition result of each face information is changed according to the change of the detection order. In this embodiment, the problem is solved and the "reproducibility" of the personal recognition result is improved.

  FIG. 1 is a block diagram showing the hardware configuration of the image processing apparatus. The image processing device may be, for example, a PC, a smartphone, or the like, and is a PC in this embodiment.

  A CPU (Central Processing Unit / Processor) 101 centrally controls the image processing apparatus 100, and implements the operation of the present embodiment by, for example, reading a program stored in a ROM 102 into a RAM 103 and executing the program. Although the number of CPUs is one in FIG. 1, it may be composed of a plurality of CPUs. The ROM 102 is a general-purpose ROM, and stores, for example, a program executed by the CPU 101. The RAM 103 is a general-purpose RAM and is used, for example, as a working memory for temporarily storing various information when the CPU 101 executes a program. The HDD (hard disk) 104 is a storage medium (storage unit) for storing a database holding processing results such as image files, image data, and image analysis, a template used by an album creating application, and the like.

  The display 105 displays the layout result of the user interface (UI) and image data of the present embodiment to the user. The keyboard 106 and the pointing device 107 receive an instruction operation from a user. The display 105 may have a touch sensor function. The keyboard 106 is used, for example, when the user receives an input such as the number of spreads of an album to be created from the user on the UI displayed on the display 105. The pointing device 107 is used, for example, when the user clicks a button on the UI displayed on the display 105.

  The data communication unit 108 communicates with an external device via a wired or wireless network. The data communication unit 108 transmits, for example, the data laid out by the automatic layout function to a printer or a server that can communicate with the image processing apparatus 100. The data bus 109 connects the blocks of FIG. 1 so that they can communicate with each other.

  The album creating application according to the present embodiment is stored in the HDD 104, and the user double-clicks the icon of the application displayed on the display 105 by the pointing device 107 to start it, as described later.

  FIG. 2 is a block diagram showing the software configuration of the album creating application. FIG. 2 particularly shows a software block diagram relating to the automatic layout processing unit that executes the automatic layout function. The album creating condition designating unit 201 outputs an album creating condition according to a user's UI operation described later by the pointing device 107 to an automatic layout processing unit surrounded by a thick solid line frame in FIG.

  The image acquisition unit 202 acquires, from the HDD 104, the image data group designated by the album creation condition designation unit 201. The image data group may be acquired from a server or SNS on the network via the data communication unit 108. The acquired image data group is sorted in order of shooting date and time and serial numbers are associated with each other. The use of the serial number facilitates the identification of information when processing the designated image data. The image conversion unit 203 converts the image data used for the subsequent processing into a desired number of pixels and color information. In the present embodiment, the number of pixels of 420 pixels on the short side and sRGB color information are converted into analysis image data.

  The image analysis unit 204 executes acquisition of a characteristic amount described below from the analyzed image data. The person information analysis unit 205 executes face detection and individual recognition processing. The image analysis unit 204 also executes acquisition of data associated with the image data acquired from the HDD 104, for example, acquisition of the shooting date and time from the Exif information. The image classification unit 206 uses the shooting date / time information, the number of shots, detected face information, and the like for the image data group to perform scene division and scene classification described below. A scene is a shooting scene such as a trip, daily life, or wedding. The image scoring unit 208 performs scoring so that image data suitable for the layout has a high score.

  The hero information input unit 207 inputs the ID (identification information) of the hero instructed by the album creating condition designation unit 201 to the image score unit 208. The image scoring unit 208 is configured to add a higher score to the image data including the hero ID input from the hero information input unit 207.

  The two-page spread allocation unit 210 divides (groups) the image data group and allocates it to each two-page spread of the album. The spread number input unit 209 inputs the spread number of the album specified by the album creating condition specifying unit 201 to the spread allocation unit 210. The spread allocation unit 210 divides the image data group according to the input number of spreads, and allocates a part of the image data group (also referred to as a division or an image group) to each spread. The image selecting unit 211 attaches the image data for the number of slots designated by the album creating condition designating unit 201 to the image scoring unit 208 from a part of the image data group assigned to each facing spread by the facing spread allocating unit 210. Make a selection based on your score.

  The image layout unit 213 determines the layout of image data. The template input unit 212 inputs a plurality of templates corresponding to the template information designated by the album creating condition designation unit 201 to the image layout unit 213. The image layout unit 213 selects a template suitable for the image data selected by the image selection unit 211 from the plurality of templates input from the template input unit 212, and determines the layout of the image data. The layout information output unit 216 outputs layout information to be displayed on the display 105 according to the image data layout determined by the image layout unit 213. The layout information is, for example, bitmap data obtained by laying out the image data selected by the image selection unit 211 on the selected template.

  The image correction unit 215 executes dodging correction, red-eye correction, and contrast correction. The image correction condition input unit 214 inputs the ON / OFF condition of the image correction designated by the album creating condition designation unit 201 to the image correction unit 215. The image correction unit 215 executes the correction on the image data when the image correction condition is ON, and does not execute the correction when the image correction condition is OFF. The image correction unit 215 executes correction on the image data input from the image conversion unit 203 according to ON / OFF of correction. The number of pixels of the image data input from the image conversion unit 203 to the image correction unit 215 can be changed according to the layout size determined by the image layout unit 213.

  When the album creating application is installed in the image processing apparatus 100, a start icon is displayed on the top screen (desktop) of the OS (operating system) operating on the image processing apparatus 100. When the user double-clicks the activation icon on the display 105 displayed on the display 105 with the pointing device 107, the album creating application program stored in the HDD 104 is loaded into the ROM 102. When the program of the ROM 102 is read out to the RAM 103 by the CPU 101 and executed, the album creating application is activated.

  FIG. 3 is a diagram showing an example of the display screen 301 provided by the album creating application.

  The display screen 301 is displayed on the display 105. The user sets album creating conditions, which will be described later, via the display screen 301, and the album creating condition designating unit 201 acquires the contents set by the user.

  A pass box 302 on the display screen 301 displays the storage location (pass) in the HDD 104 of the image data group for which the album is to be created. The folder selection button 303 displays a folder including a group of image data to be album-created in a tree structure so that the folder can be selected by the user when the user clicks on the pointing device 107. A folder path including the image data group selected by the user is displayed in the path box 302.

  The hero designation icon 304 is an icon for the user to designate the hero, and a face image of a person is displayed as the icon. A plurality of different face image icons are displayed side by side on the main character designation icon 304, and the user can select by clicking with the pointing device 107. Note that each face image is registered in association with the personal ID. The hero designating icon 304 is used to identify the hero who is the central person from the persons shown in the image represented by the image data to be analyzed. The face image is, for example, previously detected from the image data and registered in the face database in association with the personal ID. The hero designating icon 304 can be automatically set or a face image of a person selected by the user from the face images of the persons registered in the face database, for example.

  The spread number box 305 receives the setting of the number of spreads of the album from the user. The user directly inputs a number into the two-page spread box 305 via the keyboard 106 or inputs a number from the list into the two-page spread box 305 using the pointing device 107.

  The template designation icon 306 displays an illustration image for each taste (pop tone, thick tone, etc.) of the template. A plurality of template icons are displayed side by side on the template designation icon 306, and the user can select by clicking with the pointing device 107. A check box 307 accepts an ON / OFF designation of image correction from the user. The checked state is a state where the image correction ON is designated, and the unchecked state is a state where the image correction OFF is designated. Although the UI configuration screen 301 is configured to be able to set ON / OFF collectively for a plurality of correction processes, it may be configured to be able to set ON / OFF for each individual correction process.

  When the user presses the OK button 308, the album creating condition designating unit 201 acquires the contents set on the display screen 301. The album creating condition specifying unit 201 outputs the acquired setting contents to the automatic layout processing unit of the album creating application. At that time, the path entered in the path box 302 is transmitted to the image acquisition unit 202. The personal ID of the hero selected by the hero designation icon 304 is transmitted to the hero information input unit 207. The number of spreads input to the number-of-spreads box 305 is transmitted to the number-of-spreads input unit 209. The template information selected by the template designation icon 306 is transmitted to the template input unit 212. The image correction ON / OFF designation of the image correction check box 307 is transmitted to the image correction condition input unit 214. The reset button 309 on the display screen 301 is a button for resetting each setting information on the display screen 301.

  FIG. 4 is a flowchart showing the processing of the automatic layout processing unit of the album creating application. Note that the flowchart of the present application is realized, for example, by the CPU 101 reading a program stored in the HDD 104 into the ROM 102 or the RAM 103 and executing the program.

  Here, the automatic layout process will be described with reference to FIG.

  In S401, the image conversion unit 203 generates analysis image data. The image converting unit 203 converts each image data of the image data group of the HDD 104 designated by the album creating condition designating unit 201 into analysis image data having a desired number of pixels and color information. In this embodiment, it is converted into analysis image data of 420 pixels on the short side and color information of sRGB.

  In S402, the image analysis unit 204 analyzes and acquires the image feature amount. The image analysis unit 204 acquires the shooting date and time from, for example, Exif information associated with each image data acquired from the HDD 104. Further, the image analysis unit 204 acquires the feature amount from the analysis image data generated in S401. The feature amount is, for example, focus. A Sobel filter is generally known as an edge detection method. Edge detection is performed with a Sobel filter, and the inclination of the edge (luminance gradient) is calculated by dividing the brightness difference between the start point and the end point of the edge by the distance between the start point and the end point. From the result of calculating the average inclination of the edges in the image, it can be considered that the image having a large average inclination is more in focus than the image having a small average inclination. Then, by setting a plurality of thresholds having different values with respect to the inclination, it is possible to output the evaluation value of the focus amount by determining which threshold or more. In this embodiment, two different thresholds are set in advance, and the focus amount is determined in three stages of ◯ Δ ×. For example, the focus inclination desired to be adopted in the album is determined as ◯, the allowable focus inclination is determined as Δ, the unacceptable inclination is determined as ×, and each threshold value is set in advance. The threshold setting may be provided by, for example, the creator of the album creating application, or may be settable on the user interface.

  In S403, the person information analysis unit 205 performs face detection on the analysis image data generated in S401. Here, a known method can be used for the face detection processing, and for example, Adaboost that creates a strong classifier from a plurality of prepared weak classifiers is used. In this embodiment, a face image of a person (object) is detected by a strong classifier created by Adaboost.

  The person information analysis unit 205 extracts the face image and acquires the upper left coordinate value and the lower right coordinate value of the detected position of the face image. By having these two types of coordinates, the image analysis unit 204 can acquire the position of the face image and the size of the face image. Here, not only the face but also an animal such as a dog or a cat, an object such as a flower, food, a building, a figurine or the like may be used as a detection target instead of the face to create a strong discriminator by Adaboost. Accordingly, the image analysis unit 204 can detect an object other than the face image.

  In step S404, the person information analysis unit 205 performs personal recognition by comparing the face image detected in step S403 with the similarity of the representative face image stored in the face dictionary database for each individual ID. The person information analysis unit 205 sets an ID having a similarity higher than a threshold value and having the highest similarity as an ID of the detected face image. If the highest similarity is less than the threshold value, the person information analysis unit 205 registers the extracted face image as a new face in the face dictionary database as a new personal ID. Here, not only a face, but also an object such as an animal such as a dog or a cat, a flower, food, a building, or a figurine may be a recognition target instead of the face. For example, when the person information analysis unit 205 targets a dog, it is possible to compare the similarities so as to identify the dog breed.

  The detailed configuration of the person information analysis unit 205, which is one of the characteristic configurations of this embodiment, will be described. The person information analysis unit 205 executes the face detection processing in S403 and the individual recognition processing in S404 of FIG.

  FIG. 11 is a block diagram showing a detailed configuration of the person information analysis unit 205. In the present embodiment, it is assumed that 1401 to 1403 of FIG. 11 are implemented by multithread and 1404 to 1406 are implemented by single thread. Further, in the present application, even if the detection order of the face information is different each time due to the face detection being performed by the multithread, by executing 1403 to 1405, the "reproducibility" of the personal recognition result is obtained. It becomes possible to improve.

  The image information acquisition unit 1401 acquires information on image data input by the user. The image information acquisition unit 1401 can also acquire the image analysis information acquired by the image analysis unit 204. By using the information acquired by the image analysis unit 204 in the personal information analysis unit 205, it is possible to eliminate the duplication of the processing executed in both of the partial color information acquisition and the like, and to speed up the processing.

  The face information acquisition unit 1402 detects a face included in the image from the information of the image data acquired by the image information acquisition unit 1401 and acquires the detection result. The detection result includes the number of faces, position information of each face, size information of each face, rotation angle of each face, orientation of each face, positional relationship between faces, detection reliability of each face, and the like. The detection reliability is a numerical value of the degree of “face-likeness” with respect to the area determined to be a face in the face detection result. Further, as the detection result, the pixel value of the detected face area can also be acquired. In the present embodiment, the pixel value of the detected face area is obtained after normalizing the information for personal recognition processing described later.

  The face information numbering unit 1403 assigns serial numbers to the plurality of face information acquired by the face information acquisition unit 1402. In the present embodiment, the image serial number, which is information between image data, and the face serial number, which is information in the image data, are managed in association with each piece of acquired face information. The image serial number uses the serial number associated with the image data. As for the face serial number, each detected face is sorted by the detection reliability acquired at the time of face detection in S403, and each sorted face is numbered.

  For example, the number of faces in the image (the number of faces included in the image data) is three (face A, face B, face C), and the detection reliability of each face is face A = 80, face B = 70, face C = When it is 90, 1 is assigned to face A, 2 is assigned to face B, and 0 is assigned to face C. It should be noted that the higher the face detection reliability value, the more the face is judged to be more likely. By performing the sorting based on the face detection reliability, it is possible to improve the dictionary accuracy of the personal recognition process in S404 described later. In addition, a threshold value may be set for the detection reliability after performing the sort. For example, when the face detection threshold is set to 75, the number of detected faces after the face detection in S403 is 3, but the face B = 70 is not regarded as a face by the threshold processing, and the total number of faces is 2. Further, from the face orientation information obtained during face detection in S403, it is possible not to number detected faces depending on the face orientation. For example, when face A: front face, face B: front face, face C: side face, and only the faces facing the front are targeted for numbering, the number of faces detected after the execution of detection was three, but the face direction is used. The total number of faces becomes 2 due to the filtering process.

  The information management unit 1404 manages a plurality of face information acquired by the face information acquisition unit 1402 and the image serial number and the face serial number acquired by the face information numbering unit 1403 in association with each other. The information management unit 1404 sequentially stores face related information (combination information of face information and serial number) in the ROM 102 or the RAM 103. When the storable areas of the ROM 102 and the RAM 103 are reduced due to the face-related information to be stored, or when the storage areas of the ROM 102 and the RAM 103 have other data to be allocated, they can be stored in the HDD 104. Further, in the present embodiment, as the storage information, not only the information of each face is individually held, but also a data table for managing the stored face group can be stored.

  FIG. 12A is a diagram showing an example of the data table.

  As shown in FIG. 12A, when the processes of S1205 and S1206 of FIG. 12 described later are completed, the image serial number and the face serial number are linked and managed. The data table is stored in the order of the face-related information acquired by the information management unit 1404, but the image serial numbers may not be stored in ascending order depending on the system configuration and the performance of the CPU 101. In that case, the missing number may be stored in the data table, or the missing number may be skipped. In this embodiment, when the face information is not detected from the image data, the missing number is described (the image serial number is described and the face serial number is “-” stored). Further, in the present embodiment, a plurality of faces are detected from the image data, but due to factors such as the face orientation not being in front, the face information removed after the face detection reliability sorting is skipped. I do. The information stored in the data table can uniquely identify each face information.

  The order control unit 1405 acquires a plurality of face-related information managed by the information management unit 1404, and performs a sorting process using the serial number of the face-related information. Note that this sort processing refers to both sort processing using an image serial number and sort processing using a face serial number. When the face-related information is stored in the ROM 102 and the RAM 103, necessary information is read from the ROM 102 and the RAM 103 and sorted. Similarly, when stored in the HDD 104, necessary information is read from the HDD 104 and sorted. This necessary information may be all the information in the face-related information, or may be only the data table (FIG. 12A) that manages the face group.

  In this embodiment, the image serial numbers are sorted in ascending order, and the face serial numbers are sorted in ascending order within the same image serial number. As a result of these two ascending order sorts, FIG. 12B is obtained. It should be noted that the order control unit 1405 absorbs the missing number or the skipped portion stored in the data table shown in FIG. 12A, and only the data to which the image serial number and the face serial number are assigned is set as “personal”. Recognition execution order ”is assigned.

  The personal recognition information acquisition unit 1406 executes the personal recognition processing in the order of the face-related information sorted by the order control unit 1405. Individual recognition processing is roughly classified into "similarity calculation processing" and "dictionary registration processing". The personal recognition information acquisition unit 1406 first creates a new face dictionary. Part or all of the registered face information is recorded in the face dictionary. In other words, even if the detection order of face information is different each time due to the face detection being performed in multithread, the personal recognition processing is executed in the order of the sorted face-related information. "Reproducibility" of results can be realized.

  The personal recognition information acquisition unit 1406 first registers the face information input to the first person. The face information in the personal recognition execution order “0” in FIG. 12B corresponds to that. At registration, the face-related information is associated with the personal ID. In the present embodiment, if there is no face registered in the dictionary, personal ID = 0 is given to newly registered face information. In this embodiment, the dictionary is newly created and the first face information is registered, but it does not matter whether the dictionary is in the initial state. For example, face information may already be registered in the dictionary.

  Next, the personal recognition information acquisition unit 1406 collates the face information input for the second person with the face dictionary.

  Here, with respect to the face information of the personal recognition execution order “1” in FIG. 12B, the similarity with the face dictionary including the face information of the personal ID = 0 is calculated. Then, regarding the face information of the personal recognition execution order “1”, the similarity with the personal ID = 0 is determined, and it is determined whether the person is regarded as the same person as the personal ID = 0 or a new personal ID = 1 should be assigned. After the determination is completed, the face information of the personal recognition execution order “1” is also linked to the personal ID and registered in the dictionary. In this way, the face dictionary is updated with the face information acquired from the image data input by the user. It should be noted that in the present embodiment, the registration process is performed for all face images, but it is also possible to limit the face information to be registered by determining the representative face information for a certain personal ID.

  The details of the personal information analysis unit 205 have been described above. By executing the face detection processing in S403 and the personal recognition processing in S404 using this configuration, the order in which the personal recognition processing is performed and the face dictionary registration order is changed. Reproducibility is enhanced. Since the face detection processing is executed in multithreads, the order of the face-related information input to the information management unit 1404 may be different for each face detection processing even when the same image data group is handled. There is. However, the order control unit 1405 can perform a sort process to give a fixed personal recognition execution order to each face-related information at any time. Therefore, it is possible to unify the order of the face-related information input to the information management unit 1404.

  FIG. 13 is a flowchart showing the face detection processing executed by the person information analysis unit 205.

  In this embodiment, a multi-thread is executed during face detection processing. The process shown in FIG. 13 is a flowchart of the face detection process executed by one thread. In this embodiment, the image data group input by the user for creating an album is divided into equal parts and executed in each thread (each core that executes each thread). For example, when the user specifies that 120 image data are used for creating an album, and three cores for face detection can be used at the time of executing the process, one core stores 40 image data. deal with. In this case, the number of image data (N) assigned to one thread is N = 40. The designation of the number of cores can be adaptively changed by the instruction of the OS. The distribution of image data does not have to be evenly divided, and it is possible to adaptively change the progress of each thread.

  In this process, a parameter n indicating the number of processed images, which will be described later, is initialized to "0". Further, the parameter m indicating the number of detected faces for each image data is also initialized to "0".

  In S1501, the parameter n indicating the number of processed images is incremented by "+1". In step S1502, the image information acquisition unit 1401 acquires information about one piece of image data from the image group assigned to the core. At this time, the image information acquisition unit 1401 can also acquire the image analysis information acquired by the image analysis unit 204. In step S1503, the face information acquisition unit 1402 detects the face area in the image data from the image data information and the image analysis information acquired by the image information acquisition unit 1401. Further, in the present embodiment, the sorting process is executed on the plurality of pieces of acquired face information with the face detection reliability.

  In step S1504, the parameter m indicating the number of detected faces to be processed out of the number of detected faces (M) acquired in step S1503 is incremented by "+1". When a plurality of faces are detected from one image data, the processes of S1505 and S1506 are executed in order from the face information with the highest detection reliability.

  In step S1505, the face information numbering unit 1403 associates the detected face information with an image serial number. In the present embodiment, each image data is associated with face information by using a serial number assigned in the order of shooting date and time of the image data as an image serial number. At that time, the value of the parameter n in S1501 and the image serial number associated with the face information in S1505 do not necessarily match. For example, when the image serial numbers processed by a certain core are 4, 8, and 9, parameters may be sequentially assigned from 1 as the parameter n in S1501. In the present embodiment, as the parameter n in S1501, a value newly assigned to each core (that is, a value that does not match the image serial number) is applied.

  In step S1506, the face information numbering unit 1403 associates the detected face information with a face serial number. In the present embodiment, since the sort processing is executed on the plurality of face information acquired in S1503, the face serial numbers are linked in ascending order from the face information having the high face reliability.

  In S1507, the information management unit 1404 stores the face information acquired in S1503, the image serial number of each face information acquired in S1505, and the face-related information including the face serial number of each face information acquired in S1506. In this embodiment, the face-related information is stored in the RAM 103. When the storable area of the RAM 103 becomes small, the storage destination is moved to the HDD 104 or the web server and stored. In addition, a data table that is a correspondence table of the face-related information that is stored is also stored in the RAM 103.

  The contents of the data table (for example, FIG. 12A) are updated during the processing of S1504 to S1507. Note that FIG. 12A before sorting is described as the update target in the present embodiment, but the data table after sorting may be updated.

  In S1507a, it is checked whether or not the process for the number of detected faces (M) is completed, that is, whether m = M, and if m ≠ M, the process returns to S1504, and if m = M, the process is performed. Advances to S1507b. Although not shown, when no face is detected from the image data, that is, when M = 0, the processes of S1504 to S1507 are not executed and the process proceeds to the next image data. For example, when two pieces of face information are detected from the image data of the image serial number “0”, the processes of S1504 to S1507 are executed twice. As a result, "image serial number: 0-face serial number 0" and "image serial number: 0-face serial number 1" in FIG. 12A are stored.

  In S1507b, it is checked whether or not the process regarding the image data of the assigned image data number (N) is completed, that is, whether n = N, and if n ≠ N, the process returns to S1501 and n = If N, the process ends.

  Although the data table is shared by the threads in this embodiment, a separate data table may be created for each thread. If you have individual data tables, you need to integrate the tables. When sharing the data table, it is necessary to avoid access collisions in which the data write timings overlap between threads. Similarly, in the table integration process, it is necessary to prevent a mismatch of the results due to the difference between the write timing in each thread and the read timing during integration. Locking between threads is executed as a method to prevent such conflicts and inconsistencies in results. For example. If there is data to share between threads, lock write permission before one thread writes. Other threads wait without writing if the write permission is locked.

  Locks do not cause access collisions between threads, but deadlocks occur when threads are locked at the same timing. A deadlock can be prevented by introducing a control layer that controls the order of lock execution. Even if the locking timings of the threads are the same, the control layer arranges the execution order, so deadlock does not occur. Further, each core executes the processing of FIG. 13, and as a result, the data table of FIG. 12A is generated.

  FIG. 14 is a flowchart showing a personal recognition process executed by the personal information analysis unit 205. In the present embodiment, the face detection and the individual recognition process that operate in multithread are executed in parallel. The individual recognition process itself is not executed by multiple cores. FIG. 14 shows personal recognition processing executed by one thread.

  In step S1508, the order control unit 1405 checks whether or not there is update information in the data table (for example, FIG. 12A) managed by the information management unit 1404 and the face related information group. Here, unless there is update information in the data table and face-related information group managed by the information management unit 1404, the process proceeds to S1509 and waits for the process. In this embodiment, the standby condition is set to 10 seconds, and after 10 seconds have elapsed, the standby is ended and the process returns to S1508. The waiting end condition may be a designated number of seconds, or the waiting may be ended by using the data update information stored in the RAM 103 or the HDD 104 or on the web server as a trigger. On the other hand, if the order control unit 1405 determines in step S1508 that the data table and the face-related information group have been updated, the process proceeds to step S1510.

  In step S1510, the order control unit 1405 reads the data table stored in the information management unit 1404 (for example, FIG. 12A) and sorts in ascending order based on the image serial number. In the present embodiment, the image serial numbers correspond to the shooting date and time of the image groups input by the user for creating the album. By the ascending order sorting, the face-related information is sorted in the order of the shooting date and time of the image data in which the faces are shown.

  In step S1511, the order control unit 1405 reads the data table (for example, FIG. 12A) stored in the information management unit 1404, and performs ascending sort using the face serial number within the same image serial number. Note that the sorting criterion does not have to be the ascending order.

  In S1512, the order control unit 1405 adds information on the personal recognition execution order to the data table. The added personal recognition execution order is used in S1513 to determine whether to execute the personal recognition in S1514. For example, assume that the state of the data table confirmed to be updated in S1509 is the state shown in FIG. After executing the sort processing of S1510 and S1511, the personal recognition execution order is added in S1512, and the data table as shown in FIG. 12B can be acquired. In this way, the personal recognition execution order is determined according to the ascending order of the image serial numbers and the sorting result of the face serial numbers.

  Note that, in the example shown in FIG. 12A, the face-related information of the image serial number 3 is a missing number, and thus is not regarded as the target of the personal recognition execution order as shown in FIG. 12B.

  Here, for the sake of explanation, it is assumed that the update confirmed in S1509 is in the state of the data table as shown in FIG. In this case, the data table of FIG. 12C is subjected to the sort process in S1510 and S1511. However, in the example shown in FIG. 12C, the face detection result of the image data having the image serial number 6 is not described in the data table. In the multi-thread processing, even if the image data is evenly distributed to each core, there is a difference in the processing end time of each core.

  In the example of FIG. 12C, the core A has serial numbers 0, 3, 6, 9 ..., the core B has serial numbers 1, 4, 7, 10 ..., and the core C has serial numbers 2, 5, 8 ,. It is assumed that the image data of 11 ... Is distributed and the face detection processing flow is executed in each core. The cores A, B, and C do not end the face detection processing on the image data with serial numbers 0, 1, and 2 at the same time, and the execution end timing depends on the difference in contents included in each image data and the state of the terminal at the time of execution. Makes a difference. The execution time in each thread is not reproducible, and it is difficult to predict the processing end time. In FIG. 12C, since the core B processes the image data earlier than the cores A and C, the core B corresponds to the image serial number 7 before the face detection result of the image data having the image serial numbers 6 and 8 is obtained. Face information of the created image data is stored.

  The data table shown in FIG. 12D can be acquired by the sorting process and the individual recognition execution order addition process for the data table shown in FIG. 12C. In the example shown in FIG. 12D, the personal recognition execution order is not determined for the face-related information acquired from the image data having the image serial number 7. In the present embodiment, the personal recognition execution order is determined according to the ascending order sorting result of the image serial number and the face serial number. From the state shown in FIG. 12C, the number of faces in the image data having the image serial number 6 is unknown. Therefore, in S1512, the face-related information acquired from the image data having the image serial number 7 ends with the individual recognition execution order being undetermined until the next table update.

  In step S1513, the order control unit 1405 determines whether there is face-related information for which the personal recognition information acquisition unit 1406 should perform personal recognition based on the personal recognition execution order. This determination is specifically realized by determining whether or not there is a missing image serial number on the data table. For example, when the data table is in the state shown in FIG. 12B, the image serial numbers 1 to 5 are continuously arranged. Therefore, it is determined that all face-related information corresponding to FIG. 12B can be personally recognized in the personal recognition execution order. That is, it is determined that there is face-related information for which personal recognition should be performed. On the other hand, in the state shown in FIG. 12D, the image serial numbers 1 to 5 are continuously arranged. Therefore, it is determined that the face-related information corresponding to the image serial numbers 1-5 corresponding to FIG. 12D can perform the personal recognition in the personal recognition execution order. After that, for example, even if the face detection of the image serial number 8 is completed, the face detection of the image serial number 6 is unprocessed, and there is a gap in the image serial number. Therefore, until the face detection of the image serial number 6 is processed, the order control unit 1405 determines that there is no face-related information for which personal recognition should be performed (S1513-No). The reproducibility of the execution order of the individual recognition is enhanced by the definition of the order in S1512 and the execution propriety determination in S1513. If it is determined in S1513 that there is no good face information for which personal recognition is performed, the process returns to S1509 and waits for information update.

  In step S1514, the personal recognition information acquisition unit 1406 performs personal recognition on the executable face-related information. In step S1514, the degree of similarity with the face-related information that can be executed for the face dictionary is calculated. Here, if the similarity exceeds the threshold, the same personal ID as the registered personal ID that is similar is added to the face-related information. If the similarity is below the threshold, the dictionary does not A new unregistered personal ID is added to the face-related information. Then, as for face-related information, a part or all of the information is stored in the face dictionary, and the face dictionary is updated.

  As described above, in this embodiment, even if the face detection processing is performed by multithreads, the order of the face-related information for which the personal recognition processing is executed in S1514 is the same for the image data used for face detection. As long as there is, it will always be the same. As a result, the reproducibility of the face dictionary, the personal ID associated with each face-related information, and finally the album creation result is improved. In other words, when face detection of each of a plurality of image data is performed multiple times by multi-thread processing, the execution order of personal recognition processing for the face information detected by the previous detection processing and the face information detected by the detection processing again. The execution order of the personal recognition processing for is the same.

  In this embodiment, the personal recognition process and the face detection process are executed in parallel. As in the present embodiment, there is an advantage that the execution time can be shortened by sequentially performing the personal recognition on the face-related information detected from the image data that has been executed. Further, it is also possible to sort the face-related information and execute the personal recognition process after performing the face detection on all the image data.

  The image analysis unit 204 and the person information analysis unit 205 store the image analysis information acquired in S402 to S404 in a storage area such as the RAM 103 by distinguishing the image analysis information for each ID for identifying each image data as illustrated in FIG. For example, as shown in FIG. 5, the shooting date / time information and the focus determination result acquired in S402, the number of face images detected in S403, and the position information are stored in a table format. The position information of the face image is stored separately for each individual ID acquired in S404.

  In S405, it is determined whether or not the processes of S401 to S404 have been completed for all the image data of the image data group of the HDD 104 designated by the album creating condition designating unit 201. Here, if it is determined that the processing has not ended, the processing from S401 is repeated. If it is determined that the processing has ended, the processing proceeds to S406.

  In S406, the image classification unit 206 executes scene division. The scene division refers to dividing and managing the image data group designated by the condition designation unit 201 into a plurality of image groups. The image classification unit 206 divides the image data group into a plurality of groups based on the time difference between the image data calculated from the shooting date / time information acquired in S402. In the present embodiment, for example, when there is a non-photographed day between the image data, that portion is used as a reference for division. The division may be performed according to other criteria. For example, in the case where the division is performed even when the shooting days are continuous, if the time difference is 16 hours or more, the division is performed using that portion as a reference. In the case of less than 16 hours, if the time difference from the first shooting to the last shooting on each successive day is less than 4 hours, the image data group of each day is divided into one image group. If the time difference from the first shooting to the last shooting on each day is 4 hours or more and the number of consecutive shootings on each day is less than 50, the image data group for each day is regarded as one image group. Will be divided. On the other hand, if the number is 50 or more, the division is not performed. FIG. 6A is a diagram showing an example of the result of scene division of the image data group by the above scene division method. In FIG. 6A, the captured image data group is divided into eight image groups. In FIG. 6A, the arrows indicate the boundaries of the groups. It should be noted that the threshold value for the difference in image capturing date and time and the threshold value for the number of captured image data used in the above description of the scene division method are examples, and the present invention is not limited to these values.

  In S407, the image classification unit 206 executes scene classification. Specifically, the image classification unit 206 scores the image groups divided into scenes in S406 for each scene type, and classifies the determination target group into the scene with the highest score. In the present embodiment, the image group is classified into one of scene types of travel, daily life, and ceremony. Here, before the automatic layout processing unit in FIG. 4 is started, the user collects and designates a plurality of image data which are determined to be scenes of travel, daily life, and ceremony in advance.

  Here, the designation made by the user in advance will be described.

  For example, the album creating condition designating unit 201 receives designation of a plurality of image data that the user has determined to be a travel scene on a user interface screen (not shown). Then, the image analysis unit 204 acquires the feature amount for those image data. Here, the acquired feature amount is, for example, a shooting period, the number of shots, and the number of shots. The shooting period is a time difference from the first shooting to the last shooting of a plurality of image data. The number of shots is the number of pieces of image data. The number of photographed persons is the number of faces photographed in a plurality of image data. As a result, the feature amount of the shooting period, the number of shots, and the number of shots is acquired for one image data group including a plurality of image data that the user has determined to be a travel scene. Note that only one of these may be used as the feature amount, or another feature amount may be used.

  Then, the image analysis unit 204 also obtains the characteristic amounts of the photographing period, the number of photographed persons, and the number of photographed persons as described above for other image data groups further designated by the user. Then, the image analysis unit 204 calculates the average value and standard deviation of the shooting period, the average value and standard deviation of the number of shots, and the average value and standard deviation of the number of shots based on the feature amount acquired from the plurality of image data groups. Ask. FIG. 7 shows the respective average values and standard deviations corresponding to each of the plurality of types of scenes (travel, daily life, ceremony) obtained above, and the image analysis unit 204 stores these values in the ROM 102 or the like. It is stored in advance in the storage area. Alternatively, it may be incorporated in advance in the program of the album creating application.

  The respective average values and standard deviations shown in FIG. 7 are reflected in selection criteria (image data evaluation criteria) when selecting image data corresponding to each of a plurality of types of scenes. The details will be described later.

Returning to FIG. 4 again, the subsequent processing will be described. After the album creating application is started, the image classification unit 206 determines the characteristics of the shooting period, the number of shots, and the number of shots for each group in which the image data group designated by the user in the pass box 302 is divided into scenes in S406. Calculate the amount score. The image classification unit 206 calculates the shooting period of each group, the number of shots, the score of the number of shots, and the average score thereof by using the average value and standard deviation for each scene shown in FIG. 7 and equations (1) and (2). Calculated from That is,
Score = 50− | 10 × (average value−feature amount) / standard deviation | ... (1)
Average score = (score of shooting period + score of number of shots + score of number of shoots) / number of feature amount items
…… (2)
Is.

  As a result of the calculation, an average score for each scene of travel, daily life, and ceremony is calculated for each divided image data. Then, the image classification unit 206 classifies the image data of each group into the scene corresponding to the highest score among the above.

  Each average value and standard deviation corresponding to each of a plurality of types of scenes shown in FIG. 7 are selection criteria (evaluation criteria for image data (equation) that is used when selecting image data corresponding to each of a plurality of types of scenes. This is reflected in (1), so that the image data can be selected according to the selection criterion appropriate for the type of scene.

  Here, when the scenes have the same score, the scenes are classified according to a predetermined priority order of the scenes. For example, in the present embodiment, the priority order of daily life> ceremony> travel is set, and the priority of everyday scene is the highest. For example, in the image data group 5 after the scene division in FIG. 6A, the shooting period is 36 hours, the number of shots is 300, and the number of shots is 1.7. The average score of the travel scene calculated by the above formulas (1) and (2) is 45.32, the average score of the daily scene is 18.38, and the average score of the ceremony scene is -29.92. Therefore, the image data group 5 is classified into travel scenes. The image classification unit 206 manages the classified scenes by assigning scene IDs so that the scenes can be identified.

  In S408, it is determined whether or not the scene classification of S407 is completed for all the groups obtained by the division of S406. Here, if it is determined that the processing has not ended, the processing from S407 is repeated. If it is determined that the processing has ended, the processing proceeds to S409.

  In S409, the image scoring unit 208 executes hero setting. The hero setting is performed on the image data group designated by the user, and is performed by one of two setting methods, automatic and manual. The image scoring unit 208 determines the number of times each individual ID appears in the image data group, the number of times each individual ID appears in each scene, based on the individual recognition result executed in S404 and the result of the scene division executed in S406. The number of scenes in which the personal ID appears can be acquired. The image scoring unit 208 automatically sets the protagonist from such information without receiving the designation from the user. In the present embodiment, the image scoring unit 208 sets the personal ID, which has a large number of appearances in a plurality of scenes, as the main character ID when there are a plurality of scenes, and the appearance number in a single scene when there is only a single scene. A personal ID with a large number is set as the main character ID.

  When the user has designated the hero designation icon 304, the hero information input unit 207 notifies the image scoring unit 208 of the designated personal ID. When the personal ID is designated by the user, the image scoring unit 208 sets the personal ID designated by the user as the hero ID, regardless of the hero ID automatically set. This setting is called manual setting.

  The manual setting can be set by the user on the edit screen after the album creation is completed once. In order to simplify the trouble of the user on the editing screen, it is possible to make a device such as presenting the face image representing the individual ID to the user. At this time, the image serial number or the face serial number may be used to determine the face image representing the individual ID. For example, the face serial number is set from the face detection reliability. If a face image representing a face with high face detection reliability is used, a face image with less blurring can be displayed on the editing screen.

  In S410, the image scoring unit 208 executes scoring (evaluation of image data). The scoring is to give a score evaluated from the viewpoint described later to each image data (scoring), and is referred to when selecting the image data of the layout target used for the layout described later. FIG. 10 is a diagram showing a template group used for layout of image data. Each of the plurality of templates included in the template group corresponds to each spread. Template 1001 is a single template, slot 1002 indicates a main slot, and slots 1003 and 1004 indicate subslots. The main slot 1002 is a main slot (frame for laying out image data) in the template 1001 and has a larger size than the sub-slots 1003 and 1004. The image scoring unit 208 gives both a main slot score and a sub slot score to each image data.

  FIG. 8A is a diagram showing the concept of the scoring axis. FIG. 8A shows a travel scene, a daily scene, and a ceremony scene classified by the scene classification of S407. Generally, the viewpoints of photography that are considered preferable are different in travel scenes, daily scenes, and ceremony scenes. The most preferred composition in the travel scene is, for example, a photograph in which a symbolic scenery at a destination and a person who plays a leading role in the travel are shown together. Also, in the ceremony scene, for example, a photograph of a composition in which the bride and groom are close together (a photograph in which the two are close) is preferred.

  In addition, the number of photographs considered to be preferable in each scene is not limited to one. In addition to the above-mentioned portraits of people and landscapes, it is also preferable to take photographs of people with close-ups or profiles, commemorative photographs of people accompanying or on the trip, foods, buildings, and landscapes at the destination. May be considered.

  As described above, preferable photograph conditions are different for each scene, and one or more preferable photograph conditions exist in each scene. In the present embodiment, in order to calculate the scores of two patterns, that is, the score for the main slot and the score for the sub-slot for the image data, as shown in FIG. Set two.

  Before the execution of the automatic layout processing unit of FIG. 4 is started, the user previously acquires a plurality of pieces of image data determined to match the characteristics of the main slot orientation and the sub slot orientation of each scene shown in FIG. 8A. Collected and specified.

  Here, the designation made by the user in advance will be described.

  For example, the album creating condition designation unit 201 accepts designation of a plurality of image data that the user has determined on the user interface screen (not shown) to be for the main slot (or sub-slot) of the travel scene. The person information analysis unit 205 acquires the number of faces, the position of the face, and the feature amount of the face size of each designated image data. As a result, the number of faces, the position of the face, and the feature amount of the face size are acquired for each of the plurality of image data that the user has determined to be, for example, for the main slot (or for the sub-slot) of the travel scene. Then, the person information analysis unit 205 obtains the average value and standard deviation of the number of faces, the average value and standard deviation of the face positions, and the average value and standard deviation of the face size. The person information analysis unit 205 obtains the average value and the standard deviation as the statistical value of each feature amount as described above for each slot type (main slot and sub slot) of each scene. The person information analysis unit 205 stores these values in a storage area such as the ROM 102 in advance. Alternatively, it may be incorporated in advance in the program of the album creating application.

The image scoring unit 208 can acquire the information to which scene each image data to be evaluated belongs from the result of the scene classification in S407. The image scoring unit 208 uses the above-described average value and standard deviation obtained in advance corresponding to the scene of the image data of interest, and the number of faces, face position, and face size of the hero ID of the image of interest data. , The average score of the image data to be evaluated is calculated by the equations (3) and (4). That is,
Score = 50− | 10 × (average value−feature amount) / standard deviation | ... (3)
Average score = (face count score + face position score + face size score) / number of feature amount items
…… (4)
Is.

  In the expression (3) which is the evaluation criterion of the image data, it is obtained for the type of the scene corresponding to each image data according to each average value and standard deviation corresponding to each of the plurality of types of scenes shown in FIG. Mean value and standard deviation are applied. Therefore, the image data can be selected according to the selection criterion (image data evaluation criterion) appropriate for the type of scene.

  The image scoring unit 208 executes the above scoring for both the main slot and the sub slot. Here, it is preferable that the image data used for the album be in focus, so that a predetermined score may be added to the image data of the image ID having the focus feature amount ◯ shown in FIG. good. FIG. 8B shows an example of the scoring result obtained by the above-mentioned scoring, in which the main slot and the sub-slot are scored for each image ID.

  That is, in the present embodiment, conditions suitable for the main slot and the sub-slot for each scene are defined as shown in FIG. 8A, and the user previously specifies the image data determined to be suitable for them. is doing. The person information analysis unit 205 acquires the number of faces, the position of the face, and the feature amount of the face size for the image data specified by the user, and obtains the average value and the standard deviation for each feature amount. Then, when the automatic layout process of FIG. 4 is started after the album creating application is started, each image data (scene classified) subject to the automatic layout process is determined based on the user determination criteria such as how it is suitable for the main slot. Scores (similarity) indicating whether they are close are performed. For example, in FIG. 8B, the image ID 1 is given 20 points for the main slot, and the image ID 2 is given 45 points for the main slot. This means that the image ID2 is closer to the user's judgment standard for the main slot.

  Here, the detailed configuration of the image scoring unit 208 will be described.

  FIG. 15 is a block diagram showing a detailed configuration of the image scoring unit 208.

  According to FIG. 15, the image group acquisition unit 1101 acquires the image data group input by the user. The image group acquisition unit 1101 also acquires the scene division information obtained by dividing the scene in S406 when acquiring the image data group.

  The color analysis information acquisition unit 1102 acquires the image feature amount acquired in S402. In addition, the color analysis information acquisition unit 1102 also acquires image data and date information of the image data, and analysis information such as the hue, edge, and focus amount of the image data. The person-related analysis information acquisition unit 1103 acquires the processing results of face detection in S403 and individual recognition in S404. In addition, the person-related analysis information acquisition unit 1103 also acquires information on the main character setting in S409.

  The image analysis information storage unit 1104 comprehensively manages various image analysis information acquired by the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103. The image analysis information storage unit 1104 also manages secondary features that can be calculated / estimated by the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103. For example, the image analysis information storage unit 1104 manages not only the position of each face obtained in S403 but also the distance between each face that can be estimated from it.

  The image analysis information managed by the image analysis information storage unit 1104 is not limited to the above information. For example, subsidiary information other than the information acquired by the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103 may be managed. The image analysis information storage unit 1104 may use only one of the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103.

  The scene storage unit 1105 stores each type of shooting scene and a scene classification method for determining each shooting scene. In the present embodiment, three types of travel scenes, daily scenes, and ceremony scenes, and a scene classification method for discriminating each scene are stored. For example, the scene storage unit 1105 stores the information shown in FIGS. 7 and 8A.

  The scene determination unit 1106 determines the shooting scene of the image data from the image analysis information managed by the image analysis information storage unit 1104. The scene determination unit 1106 acquires the processing result corresponding to the scene classification in S407. In the present embodiment, each image data is classified into a travel scene, a daily scene, and a ceremony scene based on the image analysis information. The scene classification process in the scene determination unit 1106 determines the shooting scene of each image data.

  The parameter setting unit 1107 sets parameters used for scoring based on the determined shooting scene. In the present embodiment, the parameter is, for example, the average value and standard deviation for each of the number of faces / face positions / face size in a predetermined scene. As the parameters, predetermined parameters may be used, or the parameters may be set by the user's empirical rule or by the user at the time of system execution.

  The scoring units 1108, 1109, and 1110 are given different parameters from the parameter setting unit 1107. Each scoring unit scores (scoring) image data from the parameters set by the parameter setting unit 1107 and the image analysis information acquired by the image analysis information storage unit 1104. In the present embodiment, any of the scores acquired by the scoring units 1108, 1109, 1110 corresponds to the score for the main slot 1002 and the scores for the sub-slots 1003 and 1004. Although the number of scoring units is three in FIG. 15, the present invention is not limited to this number. For example, if there are three or more scoring axes required for album creation, scoring is performed by three or more scoring units. If less than three scoring axes are sufficient, scoring is performed by less than three scoring units.

By calculating the score of each image data by the image scoring unit 208 having the above-described configuration, it is possible to perform scoring on the scoring axis according to the shooting scene of the image data. Since the scoring axis and the method of determining the parameters are not uniform among the shooting scenes, it is possible to acquire the image data of the optimal composition and the number of people composition for each shooting scene or type of slot.
FIG. 16 is a flowchart showing the scoring process after scene classification performed by the image scoring unit 208.

  In step S1201, the image analysis information storage unit 1104 acquires and sets the image analysis information from the color analysis information acquisition unit 1102 and the person-related analysis information acquisition unit 1103. In step S1202, the scene determination unit 1106 sets a shooting scene of image data classified based on the scene classification method prepared in the scene storage unit 1105.

  In S1203, a loop process for acquiring a plurality of scoring axes in the shooting scene is executed based on the shooting scene of the set image data. In the present embodiment, the scoring axis is, for example, scoring for a main slot or a sub slot. In S1204, the parameter setting unit 1107 sets the parameters required for each scoring axis in the shooting scene of the set image data. In S1205, the scoring units 1108, 1109, and 1110 score points according to the set parameters. In the present embodiment, the scoring is calculated by the above equations (3) and (4).

  A determination threshold value and determination weight information are associated with the n parameters set in S1204. In the present embodiment, the n parameters are, for example, the number of faces / face positions / face size for the main slot described above. As the judgment threshold value and the judgment weight information, n pieces corresponding to each parameter on each scoring axis e (i, j) are secured. Here, i corresponds to the ID of the shooting scene of the image data set in S1202. Further, j corresponds to a plurality of scoring axes in the shooting scene.

  Here, the judgment threshold value and the judgment weight information will be described.

  The image analysis information acquired in S1201 is compared with the determination threshold Th {n, e (i, j)}. Here, if the value of the image analysis information exceeds the judgment threshold value Th {n, e (i, j)}, the value W {n, e (i, j)} of the judgment weight information in Expression (4) is used. It is reflected in the corresponding parameter (for example, the score of the number of faces). The final score is a value in which the determination weight information is reflected in each parameter of Expression (4).

  As described above, not only the score is calculated by the formulas (3) and (4), but also the value of the judgment weight information of each parameter may be reflected in the final score on each score axis. . Depending on the scoring axis and the judgment weight information as described above, a parameter having a weight of 0 depending on the scoring axis (a parameter that is not used, for example, the size of the face is not used), or any shooting scene or scoring axis is common. You can flexibly set the parameters used in.

  In the present embodiment, a score for the main slot and a score for the sub slot as shown in FIG. 8B are given to each image data. By scoring on each scoring axis, it is possible to quantitatively determine how superior or inferior the image data that matches the condition desired by the user is with other image data. Further, the scores may be normalized so that superiority or inferiority can be evaluated even between different scoring axes (for example, between the main slot and the sub-slot).

  The remaining processing will be described with reference to FIG. 4 again.

  In S411, the image scoring unit 208 determines whether or not the image scoring in S410 is completed for all the image data in the image data group designated by the user. If it is determined that the processing has not ended, the processing from S410 is repeated. If it is determined that the processing has ended, the processing proceeds to S412.

  In step S412, the two-page spread allocation unit 210 determines whether the number of scene divisions in step S406 is the same as the number of two-page spreads of the album input from the number-of-two-pages input unit 209. If it is determined that they are not the same, the process proceeds to S413. When it is determined that they are the same, the process proceeds to S416. For example, if the number of scene divisions is 8 and the number of inputs to the spread number input unit 209 is 8 as shown in FIG. 6A, the process proceeds to S416.

  In step S413, the two-page spread allocation unit 210 determines whether the number of scene divisions in step S406 is less than the number of two-page spreads input from the two-page spread input unit 209. When it is determined that the number is not small (large), the process proceeds to S415. When it is determined that the number is small, the process proceeds to S414. As shown in FIG. 6A, when the number of scene divisions is 8 and the number of inputs to the spread number input unit 209 is 10, the process proceeds to S414.

  In step S414, the double-page spread allocation unit 210 executes sub-scene division. The sub-scene division means that when the number of scene divisions is smaller than the number of double-page spreads of the album, the divided scenes are further subdivided. The case where the number of spreads of the designated album is 10 with respect to the number of scene divisions 8 in FIG. 6A will be described. FIG. 6 (B) shows the result of sub-scene division of FIG. 6 (A). The number of divisions is set to 10 by dividing at the portion indicated by the broken line arrow.

  The criteria for division will be described. In the division of FIG. 6A, a division location having a large number of image data is searched. Here, in order to increase the number of divisions from 8 to 10, the number of image data is determined to be two. In FIG. 6A, the image group 5 is first, followed by the image group 1 and the image group 2. Although the image groups 1 and 2 have the same number of images, the image group 2 has a larger time difference from the first image to the last image, and therefore the image group 2 is set as a division target, and the image groups 5 and 2 are divided.

  First, the division of the image group 2 will be described. Image group 2 has two peaks of the number of pieces of image data, and since these two have different photographing dates, they are divided at the portion indicated by the broken arrow in FIG. 6B corresponding to that portion. Next, the division of the image group 5 will be described. The image group 5 has three peaks of the number of pieces of image data, which is three consecutive days. Although there are two locations where the shooting date changes, division is performed at the location of the dashed arrow in FIG. 6B so that the difference in the number of sheets after division is small. As described above, the number of divisions is changed from 8 to 10. Here, the image data is divided at different locations on the shooting date, but when the image data is large on a single day, the image data may be divided at a location having the largest time difference in the single day.

  In step S415, the double-page spread allocation unit 210 executes scene integration. The scene integration means integration of the divided scenes when the number of scene divisions> the number of spreads of the album. FIG. 6C shows the result of scene integration of FIG. 6A. The number of divisions is set to 6 by integrating the broken lines.

  Explain the standard of integration. Among the divisions in FIG. 6A, a division location having a small number of image data is searched. Here, in order to reduce the number of divisions from 8 to 6, the number of image data is determined to be two. In FIG. 6A, the image group 8, the image group 3, and the image group 7 are arranged in order from the smallest one. The image group 3 and the image group 7 have the same number of image data. However, since the image group 8 adjacent to the image group 7 is an integration target, the image group 3 is an integration target. As a result, the image group 8 and the image group 3 are integrated.

  First, the integration of the image group 3 will be described. Comparing the time differences between the image groups 2 and 4 before and after the image group 3, the time difference with the image group 4 is smaller. Therefore, the image group 3 is integrated into the image group 4 as shown by the broken line in FIG. Next, the integration of the image group 8 will be described. Since the image group 8 has no subsequent image group, it is integrated with the previous image group 7 as shown by the broken line in FIG. 6C.

  In S416, the two-page spread allocation unit 210 executes two-page spread allocation. Through S412 to S415, the number of scene divisions and the number of designated spreads are the same. The two-page spread allocation unit 210 sequentially allocates from the first division to the first page of the two-page spread in the order of shooting date and time.

  In S417, the image selection unit 211 executes selection of image data. Hereinafter, an example in which four pieces of image data are selected from the divided image data group assigned to a certain spread will be described with reference to FIG.

  The section from start to end shown in FIG. 9A shows a time difference (divided shooting period) from the shooting date and time of the first image data to the shooting date and time of the last image data for the group assigned to the spread. . A method of selecting the first sheet will be described with reference to FIG. The template has one main slot 1002. Therefore, as the first sheet, the image data for the main slot is selected from the target group. Image data having the highest score for the main slot provided in S410 is selected from the plurality of image data corresponding to the divided shooting period shown in FIG. 9B. The image data for the sub-slot is selected for the second and subsequent sheets.

  In the present embodiment, the image data is selected so as not to concentrate on a part of the shooting period within the image group.

  Hereinafter, a method of subdividing the imaging period for the divided portions in the present embodiment will be described. As shown in FIG. 9C, the imaging period for the division is divided into two. Next, as shown in FIG. 9D, the image data of the second sheet is selected from the solid line shooting period in which the first sheet is not selected. Image data having the highest score for the sub-slot is selected from the plurality of image data corresponding to the imaging period indicated by the solid line in FIG. 9D.

  Next, as shown in FIG. 9E, the imaging period for each division in FIG. 9D is divided into two. Then, as shown in FIG. 9F, the image data with the highest score for the sub-slot among the plurality of image data corresponding to the solid-line shooting periods in which the first and second images are not selected is It is selected as the third sheet.

  A case where there is no image data in the shooting period for which the image data is to be selected and therefore the image data cannot be selected will be described by taking the fourth image as an example. FIG. 9G illustrates an example in which, when the fourth image is selected from the shaded shooting period in which the image data is not yet selected, there is no image data in the shaded shooting period. In this case, as shown in FIG. 9 (H), the imaging period for each division is divided into two. Next, as shown in FIG. 9 (I), the image data of the highest point for the sub-slot is the fourth image data among the plurality of image data corresponding to the solid line photographing periods in which the first to third images are not selected. Is selected as.

  In this way, by selecting image data based on the scores obtained by different scoring axes like the main slot and the sub-slot, it is possible to give variations to the image data selected for the layout. In other words, since image data of different judgment criteria can be selected, it is possible to create a variety of albums.

  The subsequent processing will be described with reference to FIG. 4 again. In step S418, the image layout unit 213 determines the image layout. Hereinafter, an example in which the template input unit 212 inputs (a) to (p) of FIG. 10 for a certain spread according to the specified template information will be described.

  Here, the number of slots of the input template is designated as 3 as an example. Further, FIG. 10 (q) is a result of arranging three pieces of selected image data in order of shooting date and time, and also shows the orientation of the image. That is, the image data 1006 has a vertical image orientation, and has the oldest shooting date and time. In the image data 1007, the orientation of the image is horizontal, and the shooting date and time is the second oldest. In the image data 1005, the orientation of the image is vertical, and the shooting date and time is the latest. Here, the image data 1005 is for the main slot, and the image data 1006 and 1007 is for the sub slot. In the present embodiment, image data older than the shooting date and time is laid out at the upper left of the template, and newer image data as the shooting date and time is laid out at the lower right. In FIG. 10 (q), the image data 1005 for the main slot has the latest shooting date and time, so the templates in FIGS. 10 (i) to (l) are candidates. Further, since the older image data 1006 for the sub-slot is the vertical image and the newer image data 1007 is the horizontal image, the template of FIG. 10 (i) is consequently converted into the selected three image data. It is decided as the most suitable template and the layout is decided.

  At this time, tentatively, the number of slots, the length and width, and the position of each slot are matched with the template of FIG. 10 (i), and the size of the main slot is larger than that of FIG. 10 (i). Suppose a template is searched. That is, based on the order of shooting dates and times and the vertical and horizontal positions of the selected image data, when there are a plurality of template candidates, in the present embodiment, the template is selected from the score difference of each image data. For example, when the score of the image data of the main slot is larger than the average score of the image data of the plurality of sub-slots by a predetermined threshold value or more, the template shown in FIG. 17 having a larger main slot is selected. On the other hand, if the threshold is not exceeded, the template of FIG. 10 (i) is selected. In the above, the case where another template is searched for the main slot has been described, but when another template is similarly searched for the sub slot, the average score of the other slots may be compared.

  In S418, which image data is laid out in which slot of which template is determined.

  In S419, the image correction unit 215 executes image correction. The image correction unit 215 executes the image correction when the image correction ON is input from the image correction condition input unit 214. As the image correction, for example, dodging correction, red-eye correction, and contrast correction are executed. The image correction unit 215 does not execute the image correction when the image correction OFF is input from the image correction condition input unit 214. The image correction can be performed on image data whose size is converted into the color space of sRGB with 1200 pixels on the short side, for example.

  In S420, the layout information output unit 216 creates layout information. The image layout unit 213 lays out, for each slot of the template determined in S418, the image data subjected to the image correction in S419. At this time, the image layout unit 213 scales and lays out the image data to be laid out according to the size information of the slot. Then, the layout information output unit 216 generates bitmap data in which the image data is laid out in the slots of the template.

  In S421, it is determined whether or not the processes of S417 to S420 have been completed for all spreads. If it is determined that the processing has not ended, the processing from S417 is repeated. When it is determined that the layout has been completed, the automatic layout process of FIG. 4 ends.

  Therefore, according to the embodiment described above, the reproducibility of the personal recognition result can be improved as long as the same data group is used even if the face detection processing is executed by the multithread processing. Further, since the image scoring section and the image layout section following this process also use the result of the individual recognition, the reproducibility of the individual recognition result is reflected in the layout result of the album. As a result, even if the face dictionary that is most suitable for the user is created, it is possible to speed up the album creation and improve its reproducibility.

<Embodiment 2>
A common feature of the embodiments of the present invention is that a multi-thread can be executed for each page of an album designated by the user. In the first embodiment, the image data group input by the user for creating the album is assigned to the cores to perform face detection. At that time, the reference of the number of image data assigned to each core is set as the shooting date and time, and the processing of each core is performed. The image data was distributed so that there was no bias.

  In this embodiment, the event division process and the page allocation process are executed after the image data group input by the user to create the album is acquired. The difference of this embodiment from the first embodiment is that the input image data group is divided into the required number of pages before the image analysis and the evaluation value calculation are executed. The image processing apparatus allocates thread processing to each core on a page-by-page basis, and each core executes image analysis processing for an image group belonging to the allocated page.

  Note that the software configuration of the album creating application used in this embodiment is the same as the configuration described in Embodiment 1 with reference to FIG. However, this embodiment is characterized in that the main character information input unit 207 influences the processing of the image selection unit 211 instead of the image score unit 208.

  FIG. 18 is a flowchart showing the processing executed by the automatic layout processing section of the album creating application. Note that, in FIG. 18, the same process reference numerals as those already described in the first embodiment with reference to FIG. 4 are designated by the same step reference numerals, and description thereof will be omitted.

  In step S1801, the image conversion unit 203 acquires Exif information of image data. The date and time when the image data was captured is recorded in the acquired Exif information. Next, in the scene division processing of S406, the image group is divided for each scene based on the shooting date and time of each image data acquired in S1801. With respect to the divided scenes, the division / integration processing according to the required number of pages is executed by the processing of S412 to S416. The scene dividing method and the processing method of dividing / integrating into the number of pages are as described in the first embodiment.

  After that, S401 to S405 are executed for all image data, and scene classification is executed for all scenes in S407 to S408.

  After that, in S1802, the image scoring unit 208 executes scoring (evaluation of image data). This execution method is the same as that described in the first embodiment, but the image scoring unit 208 of this embodiment does not set the protagonist and does not reflect the protagonist information in scoring.

  When the scoring (evaluation of the image data) is completed for all the image data, S417 to S418 and S421 are executed until the full spread is completed, and thereafter, the main character is set in S409.

  Next, in step S1803, the image selection unit 211 replaces the image data in which the layout information acquired in the image layout determination in step S418 is added with the hero setting information in step S409.

  In this embodiment, the hero information is not reflected at the time of image selection in S417, and the image data selection process is executed based on the result of scoring according to the evaluation criteria such as composition information of the image data and person balance. Therefore, there is no guarantee that the hero appears in the image data selected in the image layout determination in S418. Therefore, even if the image data with a good composition is arranged in the main slot, it may not be the image data that is preferable to the user. In step S1803, in response to the result of the main character setting in step S409, the image data that is not preferable to the user and the image data that is preferable to the user in the selection candidate image data are replaced in the selected image data.

  For example, if the selected image data is "image data showing a passerby", "image data showing the protagonist" determined to be interchangeable based on factors such as image data score, shooting date, composition, etc. It is possible to replace the image data. Given that a person is not the protagonist, it can be considered a "passerby".

  As for the replacement standard, the main character information may be prioritized and the next priority may be changed according to the type of slot. For example, when the image data of the main slot is replaced, the priority of the “image data score” may be increased in order to select image data whose score of the target selected image data and image data is as high as possible. . When replacing the image data in the sub-slot, raise the priority of "Image data capture date / time" to select the image data considering the capture date / time of the target selected image data or the adjacent selected image data. May be. In any case, by executing the image data replacement process, it is possible to complete an album layout that is preferable to the user even in a process different from the first embodiment.

  Next, details of the processing in page units will be described.

  FIG. 19 is a flowchart for executing a process in page units by multithread.

  In this process, a parameter l indicating the number of processed pages, which will be described later, is initialized to "0", and a parameter k indicating the number of processed image data of each page is also initialized to "0".

  In S1901, the parameter 1 indicating the number of processed pages is incremented by "+1". In this embodiment, a page allocation process of allocating the image data group input by the user for creating an album to each page is executed in advance. Then, it is assigned to each thread in the processing unit of each page of the album. For example, as shown in FIG. 20, it is assumed that the user creates an album of two-page spreads. In this case, the two-page spread allocation unit 210 executes page allocation according to the shooting date and time of the image data, as shown in FIG. 20, and sequentially prints 1 to 9 pages of 20, 30, 25, 30, 25. The numbers are assigned to 10, 10, 20, 30, and 10. Here, the uneven distribution of the number of assigned image data for each page is a result of the influence of the density of the shooting date / time intervals of the image groups and the cohesion as a scene. Each page is assigned to each core according to the instruction of the OS. The designation of the number of cores may be adaptively changed according to the instruction of the OS. The number of pages assigned to each core may be uniform, or may be biased according to the system configuration and the performance of the CPU 101.

  In the example shown in FIG. 20, pages with A = {1, 4, 7}, B = {2, 5, 8}, and C = {3, 6, 9} are provided in three cores A, B, and C, respectively. Assigned. That is, each core is in charge of processing three pages. Therefore, if the example shown in FIG. 20 is applied, each thread executes the loop of S1901 to S1907a three times. That is, the number L of pages allocated for processing is L = 3.

  In step S1902, the parameter k indicating the number of processed image data among the number of image data assigned to each page is incremented by “+1”. If the example shown in FIG. 20 is applied, in the case of the processing of the core A, 20 pieces of image data are processed for the first page, 30 pieces for the fourth page, and 20 pieces for the seventh page. Therefore, the number K of assigned image data in the first loop is K = 20, K = 30 in the second loop, and K = 20 in the third loop.

  In S1903, image analysis processing is executed. The method of executing the image analysis process is the same as the process in the image analysis unit 204 and the process for acquiring the image feature amount in S402 described in the first embodiment. In step S1904, face detection processing is executed. The content of the face detection process is the same as that described with reference to FIG. 13 in the first embodiment.

  In S1904a, it is checked whether the processing has been completed for all the image data to be allocated. Here, if K = k, it is determined that the process has ended, and the process proceeds to S1905. However, if K ≠ k, the process is incomplete and the process returns to step S1902.

  In S1905, scoring processing is performed. The execution method of the scoring process is the same as the description of the image scoring in S410 and the image selection unit 211 described in the first embodiment, except that the hero information is not reflected in the image evaluation value. In S1906, image selection processing is executed. The execution method of the image selection process is the same as the description of the image selection unit 211 and the image selection in S417 described in the first embodiment. In step S1907, layout selection processing is executed. The execution method of the layout selection process is the same as the description of the image layout determination in the image layout unit 213 and S418 described in the first embodiment.

  In S1907a, it is checked whether or not the processing of all allocated pages L has been completed. Here, if L = 1, it is determined that the processing has ended, and the processing ends, but if L ≠ 1, the processing is incomplete and the processing returns to step S1901.

  As described above, in this embodiment, it is possible to execute a multi-thread for each page and create an album layout for each page in each thread. Therefore, unlike the first embodiment, not only the face detection but also the scoring process, the image selection, and the layout selection are executed by the multithread, so that the processing can be further speeded up.

  Further, in this embodiment, as in the first embodiment, the individual recognition processing is executed in parallel for each core that operates in multithread. The personal recognition process is as described with reference to FIG. 14 in the first embodiment. Further, similar to the first embodiment, the configuration of the face detection in S403 and the individual recognition processing in S404 is as shown in FIG. The reproducibility of the personal recognition result is improved by the processing in the face information numbering unit 1403, the information management unit 1404, and the order control unit 1405.

  Next, the album layout replacement process related to the image selection unit 211, which is executed after the personal recognition process is completed for all face-related information of the face detection results, will be described.

  FIG. 21 is a flowchart showing details of the album replacement process.

  In S1908, it is checked whether or not the individual recognition process shown in FIG. 16 has been completed. Here, if it is determined that the individual recognition process is completed, the process proceeds to S1910, but if it is determined that the individual recognition process is not completed, the process proceeds to S1909 and waits for the predetermined time. In this embodiment, the predetermined time is set to 10 seconds, and after 10 seconds have elapsed, the standby is ended, the processing returns to S1908, and it is checked whether or not the individual recognition processing is ended. In this way, the process waits until the individual recognition process is executed for all the faces appearing in the image group input by the user.

  The waiting end condition may be a designated number of seconds other than 10 seconds, or the waiting may be ended by using the data update information on the ROM 102, the HDD 104, or the web server as a trigger. It is also possible to share the end determination flag among the threads and use the change of the flag content as a trigger when the individual recognition is completed.

  In S1910, the main character is set. The hero setting execution method is the same as the hero information input unit 207 described in the first embodiment and the hero setting process in S409. In the process shown in FIG. 21, when the main character setting is performed in S1910, face detection and individual recognition end determination have already been made for all image data input by the user for creating an album. You can use the to determine the protagonist. In addition, since face detection and personal recognition are executed with the same configuration as that shown in FIG. 11 described in the first embodiment, the execution sequence of personal recognition and the reproducibility of album creation results are improved.

  In step S1911, the image selection unit 211 performs image data replacement processing using the layout information determined in step S1907. The hero information acquired in S1910 is used as the information used for the replacement determination.

  In this process, if the hero does not appear in the selected image data selected in S1906, the image data score, the image data shooting date, and the image data that is approximated by at least one element from the composition of the image data are replaced candidates for replacement. Image data. Then, if there is image data showing the main character in the replacement candidate image data, the replacement process is executed with the selected image data. Regarding each approximation determination condition that is the interchange determination criterion, the weight may be changed between the main slot and the sub slot, or each approximation determination condition may be changed between different scenes.

  For example, on a page on which a travel scene is posted, it is considered that a high score is often given to the scenery image data in which a passerby is mixed. Therefore, in the travel scene, it is necessary to loosely set the approximation condition of each element in the image data replacement. If the approximation condition of each element becomes loose, even if the score is inferior, the image data showing the main character is preferentially replaced.

  On the other hand, in the wedding scene, there are many images mainly taken by the bride and groom in the image group, and thus image data replacement processing depending on the result of individual recognition may not be necessary. It is not necessary to replace the image data of the bride and groom whose faces are facing each other for which individual recognition could not be performed, with other image data. Therefore, in the wedding scene, the approximation condition of each element in the replacement of the image data may be set strictly. Even in image data in which it is determined that the main character is captured if the approximation conditions for each element are severe, the image data is not replaced unless the shooting date and time are close.

  Therefore, according to the embodiment described above, the reproducibility of the personal recognition result can be improved even if the album layout is executed by multithreading from the image analysis of each page. Then, since the reflection of the individual recognition result is finally executed by the image data replacement process, the album layout result also has reproducibility. Further, by increasing the number of processes in the multi-thread, it is possible to realize a higher speed than in the first embodiment.

<Embodiment 3>
In this embodiment, an example in which the reproducibility of the result is improved even when the album is created again by using the album creating application while the album is being interrupted during the album creation or after the application is closed explain.

  FIG. 22 is a row chart showing the face detection processing executed by the person information analysis unit 205. Here, in the embodiment, face detection is executed by multi-thread as in the first embodiment. Note that, in FIG. 22, the same processing step as that described in the first embodiment with reference to FIG. 13 is denoted by the same step reference number, and description thereof will be omitted.

  After S1501, in S1501a, in response to the count-up of the parameter n executed in S1501, it is checked whether or not the image data indicated by the parameter has already been subjected to image analysis processing including face detection. For example, assume that the user inputs 200 image data for creating an album. If the application is terminated when the user analyzes 120 pieces of image data at the time of launching the application last time, the face-related information as the analysis result is stored in the HDD 104, the web server, or the like. Similarly, a data table that manages the analysis results for 120 sheets is also stored in the HDD 104, the web server, or the like. Therefore, if such stored data can be reused at the time of creating the next album, the processing can be shortened accordingly. For this reason, in S1501a, the available data at the time of the previous creation is searched, and it is checked whether or not the image analysis process including the face detection has been already executed for the corresponding image data.

  If it is determined that there is information obtained by such image analysis processing, the processing skips S1502 to S1507a and proceeds to S1507b. On the other hand, if it is determined that there is no information obtained by such image analysis processing, the processing proceeds to S1502, and the processing described in the first embodiment is executed.

  Now, when the processing of S1502 to S1506 is executed, the information management unit 1404 stores the face-related information, which is a combination of the face detection result and the serial number, in the HDD 104 in S1507 '. In the first embodiment, the face-related information is stored in the RAM 103 as much as possible, but in this embodiment, it is unconditionally stored in the HDD 104. By storing in the HDD 104, past analysis results can be reused when the application is restarted. In S1507 ', the data table recording the face detection result of the image data that matches the count value of the parameter n obtained in S1501 can be stored in the HDD 104 as well. In addition, when the management system can realize the correspondence between the user and the image analysis result, the data table can be stored on the web server.

  As described above, in this embodiment, as compared with the first embodiment, it is confirmed whether or not the past analysis result remains. Therefore, with respect to image analysis and face detection, the same result can be obtained when the album is recreated, so that the execution of the process for the same image data can be omitted when the album is recreated. In this way, the processing time is shortened by reusing the past analysis results.

  FIG. 23 is a flowchart showing a personal recognition process executed by the personal information analysis unit 205. Here, in the embodiment, face detection (multithread) and personal recognition processing are executed in parallel as in the first embodiment. Note that, in FIG. 23, the same processing step as that described in the first embodiment with reference to FIG. 14 is denoted by the same step reference number, and description thereof will be omitted.

  In step S1508a, the personal identification information acquisition unit 1406 updates the face dictionary. The face dictionary can also store the previous analysis result in the HDD 104. In this embodiment, the face dictionary is updated to a new state in order to improve the reproducibility of the personal recognition result. It should be noted that a mechanism for rolling back to a stage where the face dictionary and the data table can be matched / consistent may be provided. In this case, in step S1508a, the personal identification information acquisition unit 1406 performs an update process of returning the face dictionary to a state in which the face dictionary is consistent with the time when it was created last time.

  In S1508b, the order control unit 1405 updates the data table. As shown in FIG. 12D, the data table stores the personal recognition execution order at the time of the previous creation. The execution order is not given to the face-related information for which the personal identification execution is completed and the personal ID is determined. In this embodiment, the face dictionary is updated to the new state in S1508a. In accordance with this, the personal recognition execution order is also updated to the new state. When the face dictionary is updated by the rollback function or the like so as to be consistent, an update process is performed to return the personal recognition execution order to the state that the face dictionary is also compatible.

  The subsequent processing is as described in the first embodiment.

  Therefore, according to the embodiment described above, as compared with the first embodiment, the past analysis result is most used, and when the past analysis result remains, each information including the data table is reused when the album is recreated. can do. As a result, even when the writing of the data table ends in the midway due to the forced termination of the application or the like, it is possible to improve the reproducibility of the result by dealing with an error in associating with the face-related information.

  Note that the present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by the processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions. In this case, the program and the storage medium storing the program constitute the present invention.

101 CPU, 102 ROM, 103 RAM, 104 HDD

Claims (15)

An image processing apparatus for processing a plurality of image data,
An acquisition means for acquiring a plurality of image data,
A detection unit that detects information of a face portion of each of the plurality of image data acquired by the acquisition unit by multithread processing,
Execution of recognition processing for information of the face portion of each of the plurality of image data based on a sorting result obtained by sorting information of the face portion of each of the plurality of image data detected by the detection unit A determining means for determining the order,
In accordance with the execution order determined by said determining means, an image processing apparatus characterized by comprising an execution means for executing the recognition processing for the portion of the detected face. The image processing apparatus according to claim 1, further comprising a layout unit that lays out the plurality of image data based on a result of the recognition processing executed by the execution unit. Dividing means for dividing the plurality of image data acquired by the acquiring means into a plurality of image groups,
Further comprising allocation means for allocating a plurality of image groups divided by the dividing means to a plurality of cores,
Said detection means, with respect to the image data included in the plurality of picture groups using the plurality of cores assigned by the assigning means, according to claim 2, characterized in that to detect the information of the portion of the face Image processing device. When laying out the plurality of image data over a plurality of pages, the dividing unit divides the plurality of image data into the plurality of image groups according to a predetermined standard and assigns the plurality of image data to the plurality of pages. ,
The image processing apparatus according to claim 3 , wherein the assigning unit assigns the image data assigned to each of the plurality of pages by the dividing unit to the plurality of cores. The image processing apparatus according to claim 4 , wherein the predetermined criterion includes a shooting date and time of the plurality of image data. A replacement unit that replaces at least a part of the plurality of image data laid out by the layout unit after the information of the face portion of the image data included in the plurality of image groups is detected by the detection unit. The image processing apparatus according to claim 3 , further comprising: Further comprising storage means for storing information on the face portion detected by the detection means,
When the layout is performed again by using the plurality of image data acquired by the acquisition unit, the layout unit executes the layout by using the information of the face portion stored by the storage unit. The image processing apparatus according to claim 5, wherein The detection by the detection means is re-executed based on the information of the face portion stored in the storage means when the detection by the detection means is interrupted and then re-executed. 7. The image processing device according to 7 . The image processing apparatus according to claim 7, wherein the storage unit includes an HDD. Information portion of the face, the image processing apparatus according to any one of claims 1 to 9, characterized in that the information about the face of a person included in the image data. First numbering means for assigning a number to the acquired plurality of image data to identify the plurality of image data,
Second numbering means for assigning a number to the detected face to identify the face detected by the detecting means,
The determining unit may determine the execution order based on the image data number assigned by the first numbering unit and the face number assigned by the second numbering unit. The image processing apparatus according to claim 10 . 12. The recognition process is a process of comparing the similarity between the detected face part information and the representative face image stored in the database for each individual ID. The image processing apparatus according to any one of items . When the detection process of detecting the face part information of each of the plurality of image data by the multithread process is performed again, the execution order of the face part information of each of the plurality of image data detected in the previous detection process When the image processing apparatus according to any one of claims 1 to 12, characterized in that the execution order of the information of the detected portion of the plurality of image data each face detection processing again is the same . Program to function as each unit of the image processing apparatus according to computer in any one of claims 1 to 13. An image processing method in an image processing apparatus for processing a plurality of image data, comprising:
An acquisition process for acquiring a plurality of image data,
A detecting step of detecting information on the face portion of each of the acquired plurality of image data by multithread processing,
Based on the sorting result obtained by sorting the information of the face portion of each of the detected plurality of image data, the execution order of the recognition process for the information of the face portion of each of the plurality of image data is determined. Decision process to
In accordance with the execution order in which the determined, an image processing method characterized by comprising an execution step of executing the recognition processing for the portion of the detected face.

Images