JP4158544B2 - Image search device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image search apparatus for searching for a specific image portion such as a face portion from image data such as a photograph.
[0002]
[Prior art]
In recent years, it has been considered that a specific object included in a photograph or the like, for example, a part such as a human face is specified and predetermined processing is performed based on the specified part. As an example, detect the face part of each person from the photographed picture and burn only the face part, or detect the face part of the person from the image being captured and use it for face authentication processing, Such a thing can be considered.
[0003]
In a device for recognizing a target object such as a conventional face image, in order to cope with the case where it is difficult to recognize the target object depending on the imaging state (tilt, size, illumination state, etc.) of the target object, Some perform processing adapted to the imaging state (reference state).
[0004]
Conventionally, in this process, specifically, a neural network is learned using learning image data taken while changing the imaging state, and a photograph that is a processing target using the learned neural network is used. It has been considered to detect how much the imaging state deviates from the reference state and perform image processing so as to correct the deviation.
[0005]
As an example of processing for detecting a desired pattern from target image data, a method such as a kernel nonlinear subspace method disclosed in Patent Document 1 is known.
[0006]
[Patent Document 1]
JP 2001-90274 A
[0007]
[Problems to be solved by the invention]
However, for example, in the case of a person's face, there are various changes such as lateral adjustment, upward adjustment, neck caulking condition, lighting condition, etc. When trying to detect a deviation from the conventional reference state, a neural network A large amount of learning image data used for learning is required in accordance with the various changes described above. Moreover, as a result of learning with such a large amount of image data, the scale of the neural network has become enormous, and it has been impossible to complete the processing within a realistic time.
[0008]
Further, in a conventional apparatus for recognizing a target object, processing is performed using the entire photograph or the like as a search source for the target object as a search range. For this reason, the amount of data to be processed also increases, and the processing load is great.
[0009]
As a result of these, the current state of the apparatus for recognizing an object is conventionally limited to a laboratory one, and it is necessary to eliminate such essential technical difficulties.
[0010]
The present invention has been made in view of the above circumstances, and provides a practical image search device that can reduce the load of processing for searching for an object to be searched from a photograph and can be applied to various uses. Is one of its purposes.
[0011]
[Means for Solving the Problems]
The invention according to claim 1 is an image search device for searching for a search target image data portion to be searched from within the target image data to be processed, wherein a search region is included in the target image data. A plurality of learning samples obtained by converting different conversion amounts from a predetermined reference state with respect to at least one means for defining at least one conversion type, and converting each learning sample to a reference state. Referring to the conversion database obtained by associating and learning the information on the quantity, for the search partial data included in each of the defined search areas, the conversion to be applied to make the search partial data the reference state A conversion condition including a type and a conversion amount is acquired, and conversion based on the acquired conversion condition is converted into a conversion condition for each search partial data. Refer to the search database acquired by learning using the conversion means for converting the search partial data into the reference state and the search target image data example in the reference state by repeatedly performing the conversion amount shown to “0”. Search means for determining whether or not each of the converted search partial data includes a search target; and on the target image data corresponding to the search partial data determined to include the search target Means for generating and outputting search result information for specifying the region, and the generated search result information is subjected to a predetermined process.
[0012]
According to a second aspect of the present invention, in the image search device according to the first aspect, the conversion database includes a database group for storing conversion conditions obtained by learning for each type of conversion, and the conversion means. Search for each database group included in the conversion database, and for the search partial data included in each of the defined search areas, the type and conversion of conversion to be applied to set the search partial data as a reference state The conversion condition including the amount is acquired, and the conversion based on the acquired conversion condition is repeatedly performed until the conversion amount indicated by the conversion condition for each search partial data becomes “0”, and the search partial data is used as a reference. It is supposed to be converted to a state.
[0013]
The invention according to claim 3 is an image search method for searching a search target image data portion to be searched from target image data to be processed, and defines a search area using a computer. A means for defining at least one search area in the target image data, and a plurality of conversion means converted by a different conversion amount from a predetermined reference state for at least one type of conversion. With respect to the search partial data included in each of the defined search regions with reference to the conversion database obtained by associating and learning the information on the conversion amount for each learning sample to be returned to the reference state. Acquiring a conversion condition including a conversion type and a conversion amount to be applied in order to set the search partial data as a reference state; The conversion based on the conversion condition is repeatedly performed until the conversion amount indicated by the conversion condition for each search partial data becomes “0”, and the search means converts the search partial data into a reference state, and the search means includes the reference state. A step of referring to a search database acquired by learning using an example of image data of a search target in the step of determining whether or not a search target is included in each of the converted search partial data, and output means, Generating and outputting search result information for specifying an area on the target image data corresponding to the search partial data determined to include the search target, and the generated search result information It is intended to be subjected to predetermined processing.
[0014]
The invention according to claim 4 is a program for searching a target image data portion to be searched from target image data to be processed, and a computer is searched in the target image data. A plurality of learning samples obtained by converting at least one region and means for converting at least one type of conversion from a predetermined reference state by different conversion amounts are used to return each learning sample to the reference state. With reference to a conversion database obtained by learning by associating conversion amount information, the conversion to be applied in order to set the search partial data as a reference state for the search partial data included in each of the defined search areas The conversion condition including the type and the conversion amount is acquired, and conversion based on the acquired conversion condition is applied to each search partial data. It is repeatedly acquired until the conversion amount indicated by all the conversion conditions becomes “0”, and learning is acquired using the conversion means for converting the search partial data into the reference state and the image data example of the search target in the reference state. Search means for referring to the search database and determining whether or not each of the converted search partial data includes a search target, and corresponding to the search partial data determined to include the search target It is made to function as a means for generating and outputting search result information for specifying an area on the target image data, and the generated search result information is subjected to a predetermined process.
[0020]
DETAILED DESCRIPTION OF THE INVENTION
[Basic configuration]
Embodiments of the present invention will be described with reference to the drawings. As shown in FIG. 1, the image search apparatus according to the embodiment of the present invention includes a control unit 11, a storage unit 12, a database unit 13, a display unit 14, an operation unit 15, and an external storage unit 16. It is comprised including. These units are connected to each other via a bus, which is realized by using a general computer. This computer may be incorporated in another product, such as a camera.
[0021]
The control unit 11 operates according to a program stored in the storage unit 12, and includes a search area demarcation process that demarcates at least one search area from the target image data to be processed, and a reference state. A conversion process for conversion, a search process for detecting a search area including a search target, and a predetermined process using the search result are executed. Specific processing contents of these control units 11 will be described in detail later.
[0022]
The storage unit 12 stores software executed by the control unit 11. The storage unit 12 also operates as a work memory that holds various data required by the control unit 11 during the process. Specifically, the storage unit 12 can be realized as a storage medium such as a hard disk, a semiconductor memory, or a combination thereof.
[0023]
As will be described later, the database unit 13 is a database including a conversion database 13a used in the conversion process of the control unit 11 and a search database 13b used in the search process. The database unit 13 is specifically a storage medium such as a hard disk, and the storage unit 12 may also serve as the database unit 13, but is illustrated here separately for the sake of explanation.
[0024]
The display unit 14 is, for example, a display device or a printer device, and displays information according to an instruction input from the control unit 11. The operation unit 15 is a keyboard or a mouse, for example, and accepts a user operation and outputs the content of the operation to the control unit 11.
[0025]
The external storage unit 16 reads a program or data from a computer-readable removable medium (a type of storage medium) such as a CD-ROM or a DVD-ROM, and outputs the program or data to the control unit 11. The processing to be stored in is performed. The program according to the present embodiment can be distributed and stored in a portable storage medium such as a CD-ROM, and is copied to the storage unit 12 using the external storage unit 16 and used. Note that the program according to the present embodiment may be copied to the storage unit 12 not only from such a storage medium but also from a server on the network via a communication unit (not shown).
[0026]
[Processing of control unit 11]
Here, the content of the process of the control part 11 is demonstrated concretely. In the present embodiment, image data (target image data) to be processed is input from the outside via the external storage unit 16 or a communication unit (not shown) and stored in the storage unit 12. Here, the target image data may be one or plural. When the user operates the operation unit 15 to instruct the control unit 11 to perform a process of searching for a search target for specific target image data (instruction to start processing), the control unit 11 2 is started.
[0027]
The control unit 11 delimits the target image data sequentially, demarcates a search area for each reduced-converted target image data, performs conversion processing to the reference state and search processing for each search area, and performs each reduction Map data representing which part of the converted target image data contains the search target is generated.
[0028]
Specifically, the control unit 11 first sets the reduction rate S to the minimum reduction rate (for example, 1 time, that is, does not reduce) (S1), and reduces the target image data at the reduction rate S (S2). Then, an area of map data having a size equal to the size of the target image data after reduction is secured on the storage unit 12, the value of the area is set to “false”, and the map data is initialized. (S3). For example, if the target image data after reduction is image data of 1000 × 1000 pixels, an area for 1000 × 1000 bits is secured, and each bit value is initialized to “0”.
[0029]
Next, the control unit 11 performs a process of defining at least one search area for the reduced target image data (S4). The search area defining process will be described in detail later. Then, one of the search areas is selected (S5), and conversion processing is executed for the search area (S6). This conversion process will also be described in detail later.
[0030]
The control unit 11 executes a process (search process) for determining whether or not a search target is included for the image data portion included in the search area after the conversion process (S7), and is determined to be included. If yes (Yes), the value of the area on the map data corresponding to the search area after the conversion process is set to “true” (S8). Then, it is checked whether or not there is an unselected search area (S9). If there is an unselected search area (if Yes), the process returns to step S5, and one of the unselected search areas is selected. Is selected and processing continues (A).
[0031]
On the other hand, when it is determined that the search target is not included as a result of the search process in the process S7 (when No), the process directly proceeds to the process S9. If there is no unselected search area in process S9, that is, if the conversion process and the search process have been completed for all search areas (if No), the currently set reduction rate S is set in advance. (S10), if not (if No), the reduction rate S is adjusted to be increased (S11), and the process returns to step S2 and processed. Continue (B). Here, the process S11 for largely adjusting the reduction rate S may be, for example, a process for increasing the reduction rate S by a predetermined ratio, or by multiplying the magnification which is the reduction rate S by a predetermined multiplier ΔS, S = A new reduction ratio S may be set as S × ΔS.
[0032]
Further, if the currently set reduction rate S in the process S10 exceeds the predetermined maximum reduction rate (if Yes), based on the map data corresponding to the target image data at each reduction rate, the original In the target image data (before reduction), an area including a search target is defined (S12), and the process ends.
[0033]
[Search area definition processing]
Next, a process in which the control unit 11 defines a search area (search area defining process) will be described. The search area defining process can be performed using information on the start point input by the user or by autonomously defining an area that satisfies a predetermined condition.
[0034]
For example, when using information input from the user, the control unit 11 determines a predetermined size with each start point coordinate as an upper left corner based on information of at least one start point coordinate input from the operation unit 15 or the like. Are defined as search areas.
[0035]
When the area satisfying the predetermined condition is autonomously searched and demarcated, the control unit 11 starts the coordinates (for example, X = 0, Y = 0) of the upper left corner of the target image data (after reduction) as the starting point. As a result, it is checked whether or not a predetermined demarcation condition is satisfied for a rectangular area of a predetermined size. If the delimitation condition is satisfied, the process of setting the rectangular area as a search area When the start point deviates from the width of the target image data (X> the width of the target image data), the start point is moved by a predetermined amount in the height direction. (X = 0, Y = Y + ΔY) and repeat the process in the width direction. Thus, an area that satisfies the demarcation condition is defined as a search area in the entire target image data.
[0036]
Here, the moving amounts of the starting point are ΔX and ΔY in the width direction and the height direction, respectively, but these moving amounts are relative to ΔX and ΔY when the image data is 1 time depending on the reduction rate S of the target image data. ΔX / S, ΔY / S may be used.
[0037]
[Conversion processing]
Next, the conversion process will be described. One characteristic of the present embodiment is that this conversion process is performed step by step, and conversion corresponding to one degree of freedom of conversion is performed at each step. The control unit 11 according to the present embodiment calculates predetermined feature vector information based on the image data part included in the search area. Here, the feature amount vector information is a vector amount including a plurality of feature amount elements selected in accordance with the property of the search target.
[0038]
In the present embodiment, the control unit 11 will be described as specifying the transformation by the kernel nonlinear subspace method using the feature vector information and the feature vector information stored in the transformation database 13a.
[0039]
[Contents of conversion database 13a]
Since this kernel nonlinear subspace method is widely known as a method for classifying data into a certain category, a detailed description thereof will be omitted. However, in brief, in the space F stretched on the basis of a feature element, Recognize each of a plurality of subspaces Ω included in the space F as a category to which data is classified, and feature vector information (for example, Φ) in the space F created based on the data to be classified. Projection is performed on each subspace Ω (the projection result is, for example, φ), and the subspace Ω (assuming that the distance E between the feature vector information Φ before projection and the feature vector information φ after projection is minimized) This is a method for determining that the data to be detected and classified is belonging to the category represented by the subspace Ω.
[0040]
Therefore, at the learning stage, the nonlinear mapping (mapping to the space F, that is, mapping to the space F, that is, the closest subspace Ω based on the feature vector information corresponding to the exemplary data for learning (learning sample) that should belong to the same category, is made. Parameters and the like included in the kernel function) and at least one of the hyperplanes separating the subspaces Ω corresponding to the respective categories are adjusted.
[0041]
In the present embodiment, this conversion database 13a is formed in order to determine a method (type and amount of conversion) for converting the search target into the reference state. That is, the conversion database 13a has been learned and acquired so that the type of conversion to be performed and the amount (category) of conversion to be performed can be determined for image data whose unknown state is unknown. In the present embodiment, a conversion database 13a is created for each type of conversion (degree of freedom) to be performed on an image, such as image rotation, parallel movement, and size change. The conversion database 13a corresponding to each degree of freedom of conversion is acquired by learning the conversion amount of the corresponding conversion as a category.
[0042]
In order to acquire this learning, a learning sample is generated as follows in the learning process of the conversion database 13a of the present embodiment. That is, a plurality of examples of image data to be searched in a predetermined reference state are prepared, and for each example of image data, conversion is performed with different conversion amounts for each degree of freedom. A plurality of converted image data is generated. The converted image data generated for each degree of freedom is used as a learning sample for each degree of freedom. Specifically, when a face is a search target, a plurality of face image data in a predetermined reference state (predetermined shooting condition / posture) is prepared as an example, and for each image data, for example, rotation / For each degree of freedom such as parallel movement and size change, if it is a rotation, the converted image data rotated by an angle of 5 degrees in the range from -180 degrees to 180 degrees is used as a learning sample for the degree of freedom of rotation. . In the case of parallel movement, a plurality of converted image data moved by 5 pixels vertically and horizontally are used as learning samples for the degree of freedom of translation. Since these learning samples include conversion degrees of freedom such as movement, the learning samples are generated by moving the area of the reference state from the image data in a region wider than the reference state while moving it by 5 pixels.
[0043]
In this way, for each of a plurality of image data examples, a plurality of image data that has been subjected to a plurality of conversions for each degree of freedom is generated, and information indicating what conversion has been performed on each image data (a large amount of conversion) Etc.).
[0044]
Here, in order to obtain image data having undergone conversions of different conversion amounts, the image data that has been converted while changing the conversion amount by a predetermined step (for example, 5 degrees in terms of rotation) is used as a learning sample. Although they are included, it is also possible to perform conversion while determining the conversion amount by a random number, and include each in the learning sample, instead of changing each predetermined step.
[0045]
Next, the conversion database 13a corresponding to each degree of freedom is learned using the learning sample for each degree of freedom.
[0046]
[Conversion operation]
The control unit 11 performs the conversion process as follows for each of the search areas defined by the search area defining process, using each of the conversion databases 13a learned in this way. That is, the image data part included in the search area to be processed (for example, can be equated with a vector value as a column of pixel values) is converted into feature vector information in the space F (for each conversion database 13a, that is, A set of features defined for each degree of freedom of transformation), and then project the mapping onto each subspace Ω. Then, a conversion amount that minimizes the distance E between the feature vector information before projection and the feature vector information after projection is determined. Further, the control unit 11 calculates a square value L of the distance E, and stores this in the storage unit 12 as an error.
[0047]
Here, the conversion amount is determined for each degree of freedom based on each conversion database 13a, but the control unit 11 selects one of the conversion amounts corresponding to each degree of freedom as a predetermined condition (for example, corresponding to each conversion amount). The conversion corresponding to the selected degree of freedom is converted by an amount corresponding to the selected conversion amount.
[0048]
That is, from the image data portion included in the search area, the reference state is obtained by, for example, 10 degree rotation conversion with respect to the rotation degree of freedom by the information acquired by learning in each conversion database 13a corresponding to each degree of freedom. The error is Lr, and for the degree of freedom of translation, information such as Lp is obtained by approaching the reference state by converting 5 pixels to the left, and from this, the error is the minimum. Select a transformation with degrees of freedom. For example, in the case of the above-described example, if Lr <Lp, a 10-degree rotation transformation is applied to the search area to define a new search area. Then, the image data portion included in the new search area is further mapped to the feature vector information in the space F, and the mapping is further projected to each partial space Ω. And it repeats from the process which determines the conversion amount from which the distance E of the feature-value vector information before projection and the feature-value vector information after projection becomes the minimum.
[0049]
If the conversion amount corresponding to each degree of freedom represents “0” (that is, no conversion), the process ends at that stage, and if there is an unprocessed search area, The conversion process is performed with any of the unprocessed search areas as a processing target.
[0050]
Here, the image data portion included in the defined search area is used as it is in the target image data. However, processing for reducing the resolution of the image data portion is performed to obtain coarse-grained data. You may perform a conversion process using data. In this case, each conversion database 13a is learned and acquired using a learning sample corresponding to the coarse-grained data.
[0051]
In addition, the control unit 11 may perform processes such as calculation of feature vector information, mapping to a partial space, distance evaluation, and error evaluation in parallel instead of sequentially for each degree of freedom.
[0052]
Furthermore, although the case where the kernel nonlinear subspace method is used has been described here as an example, other methods such as an auto encoder may be used as long as data classification and error evaluation at the time of classification are possible.
[0053]
[Search process]
Next, the search process will be described. In this search process, it is determined whether or not a search target is included using the search database 13b for each image data portion included in each search area for which the conversion process has been completed. As a specific example of the search process, there is a method disclosed in Japanese Patent Laid-Open No. 2002-329188. Next, the outline will be described.
[0054]
[Learning course of search database]
This search database 13b is formed by learning a neural network using an example of search target image data in a reference state as a learning sample. That is, the control unit 11 receives a plurality of learning samples, calculates a feature amount set (feature amount vector) selected in advance according to the properties of the search target for each of the learning samples, and generates learning data. To do. Next, using this learning data, a lattice space map is formed on the M × M ′ lattice space stored in the storage unit 12 by SOM (self-organizing map). That is, the control unit 11 calculates the distance between the feature vector that is the input learning data and the weight vector assigned to each grid with a predetermined measure (for example, Euclidean measure), and the distance is the minimum. A lattice (most matching node) c is detected. Then, the weight vectors of the plurality of grids near the most matched node are updated using the input feature vector. By repeating this process, a lattice space map is formed on the storage unit 12, and the best matching nodes for similar feature vectors form a continuous region. That is, in this lattice space, a non-linear projection from a feature vector, which is a multi-dimensional input signal, to a two-dimensional map is formed while maintaining the phase, and by updating the weights, feature portions of the data are organized, As a learning result, there are grids that react to similar data in close proximity.
[0055]
When the learning based on each learning data is completed, the control unit 11 classifies each lattice of the lattice space map into a category. This classification can be performed based on, for example, the distance between the lattices (the distance between the weight vectors associated with each lattice), and the category of the lattice group that reacts to image data similar to the search target (search target category). And other categories of lattice groups (non-search target categories).
[0056]
[Search operation]
The control unit 11 secures an area for storing map data having the same size as the target image data in the storage unit 12 and initializes the value of the area to “false”.
[0057]
The control unit 11 calculates a predetermined feature vector based on the image data portion of the search area for which the conversion process has been completed, using the learned and acquired search database 13b. Then, the distance between the calculated feature vector and the weight vector associated with each grid in the search database 13b is obtained, the grid (the most consistent node) having the minimum distance from the feature vector is specified, and the specified grid If it belongs to the search target category, it is determined that the search area includes the search target. If the specified lattice belongs to the non-search target category, the search area does not include the search target. to decide.
[0058]
[Operation of Control Unit 11]
As described above, the control unit 11 according to the present embodiment sequentially extracts the target image data to be searched, extracts the area to be searched from each of the reduced target image data, After the conversion is performed so that the image data is in the reference state, it is determined whether the search target is included in the image data in the converted search area. That is, since the control unit 11 performs the process of setting the image data to be searched for as a reference state, the position of the search area before conversion may be slightly shifted. Even if the reduction ratio is slightly deviated from the reference state, there is no problem.
[0059]
Thus, conventionally, multistage reduced image data reduced by 0.8 times is generated and the search area is extracted while being shifted one pixel at a time. It is possible to reduce by 5 times, and ΔX and ΔY can be set to 6 pixels or the like even when a region that satisfies a predetermined condition is autonomously extracted when defining a search region. As a result, the number of patterns to be searched can be significantly reduced, and the processing load for searching for a search object from a photograph or the like can be reduced.
[0060]
The control unit 11 generates map data representing an area determined to include the search target as search result information. This map data corresponds to each of the target image data after being reduced at each reduction rate. Are generated. Therefore, an area including the search target is determined by using the plurality of map data (each having the reduced target image data size) in an integrated manner.
[0061]
For example, each map data is enlarged at an enlargement ratio corresponding to the respective reduction ratios, compared with the size of the original target image data, and a region that is “true” in common to all map data ( Based on the target image data of any reduction ratio, it may be determined that the search target is included in an area where the search target is determined to be included. Further, an area that is “true” in any one of the map data may be determined as an area that includes a search target.
[0062]
[Operation and application examples]
Next, an application example of processing for finding an area including a search target from target image data will be described.
[0063]
According to the image search device according to the present embodiment, for example, it is possible to count how many people are captured in a certain photo. In other words, each database 13 is learned by using a human face as a search target by using a photograph read as processing target image data. Then, the process shown in FIG. 2 is performed to count how many areas are “true” on the map data. According to this, it can utilize for uses, such as traffic census, for example.
[0064]
Similarly, using the database 13 in which a human face is learned as a search target, image data digitally picked up by a camera is used as target image data to detect which part of the image data the face is picked up. Personal authentication can also be performed using the face imaged in the detected part.
[0065]
Further, for example, by using image data obtained by reading a plurality of photographs as target image data, the process shown in FIG. 2 detects in which part of the image data corresponding to each photograph the person's face is reflected, and the search condition Let the user specify which part of the photo the person's face is imaged to be searched, search for a detection result that matches the specified search condition, and select a photo corresponding to the matching detection result There are also application examples such as presenting to the user.
[0066]
If this is further performed on image data acquired from a web server on the network, it can be applied to search for image data published on the web where a person is placed in a specific situation. Become.
[0067]
Furthermore, when this example is applied to a camera, the following camera can be obtained. That is, by letting the user specify the arrangement state of the human face in advance, each piece of image data continuously taken digitally is set as the target image data, The arrangement state of the face of the included person is detected using the processing of FIG. 2, and image data when the detection result matches the arrangement state specified by the user is selectively stored in the storage unit 12. Hold. According to this, it is possible to take a photograph in which a human face is in a desired arrangement state. This can also be used when it is difficult to obtain an image when two persons are facing the front, such as when photographing two children. This is because it is detected that the face is arranged at the corresponding portion by the processing of FIG. 2 when facing the front. In addition, when taking a group photo, instead of using a self-timer or remote controller as in the past, it is also used for recording images when the person who set up the camera arrives at the desired position on the photo it can.
[0068]
Furthermore, in this embodiment, the search target is not limited to a human face. For example, in an industrial parts production factory, when the number of parts flowing on a conveyor is counted, or when a natural image is taken or a scientific experiment is recorded, an image where the subject is in a desired arrangement state is recorded. It can also be used for purposes such as.
[0069]
[Modification of processing]
It is also preferable to determine whether or not there is a possibility that a search target is included at the stage of the search area defining process, and to define the search area based on the determination result. This is because it can be assumed that the search areas defined by autonomously searching for areas that satisfy the predetermined condition include areas that overlap each other. In other words, even when the same front-facing face is the search target, the search area is defined as a different search area between the area around the left ear, the area around the nose, and the area around the right ear. This is based on the fact that not all search areas need to be processed in this case.
[0070]
In this case, the control unit 11 performs the process shown in FIG. 3 instead of FIG. In FIG. 3, the same reference numerals are given to the portions that perform the same processing as in FIG. 2, and detailed description thereof is omitted. The control unit 11 first sets the reduction rate S to the minimum reduction rate (S1), and reduces the target image data at the reduction rate S (S2). Then, an area of map data having a size equal to the size of the target image data after reduction is secured on the storage unit 12, the value of the area is set to “false”, and the map data is initialized. (S3). Further, the control unit 11 generates an array of Boolean values having the same size as the target image data (hereinafter referred to as a “gazing map”) (S20). Each Boolean value is initially set to “false”.
[0071]
Next, the control unit 11 performs a process of defining at least one search area for the reduced target image data (S4). In this process S4, a search area is demarcated by the autonomous process already described. Next, the control unit 11 changes and sets the Boolean value of the area corresponding to the search area to “True” among the Boolean values (S21). Then, the control unit 11 selects any one of the search areas (S22), and the center coordinates (for example, (x1, y1) and (x2, y2) of the area on the gage map corresponding to the selected search area. ) Is a rectangular area with upper left and lower right, respectively, the center coordinates are points represented by ((x1 + x2) / 2, (y1 + y2) / 2), and are converted into integers by a predetermined method) Referring to the Boolean value, it is checked whether it is true (S23).
[0072]
If the Boolean value is true in this process S23 (if Yes), the conversion process (S6) and the search process (S7) are performed on the selected search area, and the search target is selected by the search process. When it is determined that it is included in the search area (if Yes), the value of the area on the map data corresponding to the search area after the conversion process is set to “true” ( Along with S8), the Boolean value of the area corresponding to the search area is set to false on the aging map (S24). As a result, the search area in which the center point coordinates are included in the portion is not processed in an overlapping manner.
[0073]
The control unit 11 further checks whether there is a Boolean value that is true on the aging map (S25). If there is a Boolean value that is true (if Yes), the control unit 11 A new search area is demarcated using either one as a central coordinate (S26), and the process returns to process S22 to select the search area and continue the process. If the Boolean value is not true in step S23 (if No), the process proceeds to step S25 to check whether there is a search area that has not been selected (C).
[0074]
On the other hand, if it is determined that the search target is not included as a result of the search process in the process S7 (No), the process proceeds to the process S25 as it is. Also, in process S25, if there is no Boolean value that is true on the aging map (if No), whether or not the currently set reduction rate S exceeds a predetermined maximum reduction rate. (S10), if it does not exceed (if No), the reduction rate S is adjusted to be increased (S11), and the process returns to the process S2 to continue the process. Here, the process S11 for largely adjusting the reduction rate S may be, for example, a process for increasing the reduction rate S by a predetermined ratio, or by multiplying the magnification which is the reduction rate S by a predetermined multiplier ΔS, S = A new reduction ratio S may be set as S × ΔS.
[0075]
Further, if the currently set reduction rate S does not exceed the predetermined maximum reduction rate (if Yes) in the process S10, based on the map data corresponding to the target image data at each reduction rate, the original In the target image data (before reduction), an area including a search target is defined (S12), and the process ends.
[0076]
According to this process, even if a plurality of search areas are defined for one search object in the search area defining process, a plurality of adjacent search areas are not processed in an overlapping manner.
[0077]
[Variation of search area definition processing]
Further, in the search area defining process, the control unit 11 determines the entropy, hierarchical entropy, color, luminance distribution, and two or more of these values of the image data included in the area to be defined as the search area. A combination may be used to determine whether or not the search area is actually defined. For example, when searching for a face part of a person, the entropy is high in the peripheral part (outline part) of the face, so if the entropy is larger than a predetermined threshold, the area to be defined as the search area is actually A search area is defined. Otherwise, the area is not set as a search area, and other processes are continued. According to this, the search area to be subjected to the conversion process / search process can be rationally reduced, and the processing load can be reduced.
[0078]
[Still another variation of processing]
Furthermore, in the description so far, the control unit 11 defines the search area, but the search area itself may be specified by the user. In this case, the control unit 11 causes the size and direction of the target image data to be selected, and displays a frame line corresponding to the size on the display unit 14. Further, the control unit 11 causes the user to draw search area candidates within the frame line. Specifically, a display as shown in FIG. 4 is performed on the display unit 14, and a search area candidate R is drawn in the frame B. When the control unit 11 receives a click operation of the “define” button from the user, the control unit 11 defines a rectangle circumscribing the search region candidate R as a search region based on the drawn search region candidate R.
[0079]
Further, a pattern (template) of a search area defined in advance is stored in the storage unit 12, and when a search area specification is accepted, a list of the stored templates is presented to the user for selection. A search area may be defined based on the selected template.
[0080]
[Modification of conversion process]
When letting the user specify these search areas themselves, information on conversion processing to be performed for each of the specified search areas (information indicating degree of freedom of conversion and conversion amount, hereinafter referred to as user-specified conversion information) is associated. Alternatively, this may be used during the conversion process. This user-specified conversion information corresponds to an example of “information related to a search target state” in the present invention. The conversion process in this case is as follows.
[0081]
That is, the control unit 11 first selects one of the defined search areas, and converts the selected search area based on information on conversion processing associated with the search area (preliminary conversion). Then, the image data portion included in the preliminary-converted search area is mapped to the feature vector information in the space F, and the mapping is projected onto each partial space Ω. Then, a conversion amount that minimizes the distance E between the feature vector information before projection and the feature vector information after projection is determined for each degree of freedom of conversion using the conversion database 13a.
[0082]
That is, from the image data portion included in the search region, information on the amount of conversion for approaching the reference state for each degree of freedom, based on the information learned and acquired in each conversion database 13a corresponding to each degree of freedom, Since the error (square value of the distance E) at that time is obtained, the conversion of the degree of freedom in which the error is minimized is specified from among the conversions of these degrees of freedom. Here, if the specified degree of freedom is the same as the degree of freedom of conversion determined by the user-specified conversion information associated with the selected search region, the control unit 11 has the next degree of freedom with the smallest error. The process of re-specifying the conversion is repeated to specify a conversion with a degree of freedom other than the conversion degree of freedom determined by the user-specified conversion information. Then, the control unit 11 performs a conversion of the specified degree of freedom on the selected search area by the conversion amount determined for the conversion of the degree of freedom, thereby defining a new search area. Then, the image data portion included in the new search area is further mapped to the feature vector information in the space F, and the mapping is further projected to each partial space Ω. And it repeats from the process which determines the conversion amount from which the distance E of the feature-value vector information before projection and the feature-value vector information after projection becomes the minimum.
[0083]
In addition, when the conversion amount corresponding to the degree of freedom other than the degree of freedom determined by the user-specified conversion information represents “0” (that is, no conversion), the degree of freedom determined by the user-specified conversion information If there is no conversion with a degree of freedom other than, the process ends at that stage, and if there is an unprocessed search area, the conversion process is performed with any of the unprocessed search areas as a processing target.
[0084]
In this case as well, a process for reducing the resolution of the image data portion may be performed to obtain coarse-grained data, and the conversion process may be executed using the coarse-grained data. In this case as well, each conversion database 13a is learned and acquired using a learning sample corresponding to the coarse-grained data.
[0085]
Using this conversion process, for example, it is possible to search for image data including the state of the search target in each search region. In other words, when there are upright people and lying people in the photo, and each face is upright and rotated 90 degrees from upright, information on each state is displayed along with the search area. Specify as a search condition. For example, if the search database 13b is learned and acquired as a learning sample in the search database 13b as the reference state, the search region associated with the erect state is designated as performing 0 degree rotation conversion. In this case, since rotation conversion is not performed in the conversion process, only the face of the person in the reference state, i.e., the upright state, is detected in the search process, and the person who is lying down (rotated 90 degrees from the upright position) is also detected. If it is specified that -90 degree rotation conversion is to be performed for (in this state), after the -90 degree conversion is performed in the preliminary conversion, further rotation conversion is performed by the conversion process. In the search processing, only the face of a person whose state after the preliminary conversion is the reference state (that is, the face of a person tilted by 90 degrees) is detected with respect to the degree of freedom of rotation in the search process.
[0086]
[Another configuration example for obtaining the same effect]
Further, the control unit 11 executes the conversion process as shown in FIG. 2 instead of the above process using the preliminary conversion, and when the conversion process is completed, the content of the conversion actually performed in the conversion process And a process of comparing the content of the conversion determined by the user-specified conversion information associated with the search area. In this case, if a conversion equivalent to the conversion indicated in the user-specified conversion information is performed by this comparison process, the search process is executed for the search area where the conversion process is performed, and the equivalent conversion is performed. If not, the search process for the conversion area where the conversion has been performed is skipped.
[0087]
[Other variations]
In the description so far, the conversion performed in the conversion process is two-dimensional rotation, translation, enlargement / reduction (enlargement / reduction of the search area, and the original image data area before the enlargement / reduction) However, other than this, for example, a human face may include a three-dimensional rotation that is influenced by the posture (depression or swing direction). . In this case, assuming an average three-dimensional model to be searched, conversion of three-dimensional rotation can be realized by using an image data portion projected onto the average three-dimensional model.
[0088]
Further, in the search process, for example, even a human face may be further classified into categories and may include conditions such as age, sex, and whether or not the mouth is open.
[0089]
Further, in the flowcharts of FIG. 2 and FIG. 3, the control unit 11 sequentially performs the processing at each reduction rate. However, the processing at each reduction rate is independent of each other. You may perform the process in a reduction ratio in parallel.
[Brief description of the drawings]
FIG. 1 is a configuration block diagram of an image search apparatus according to an embodiment of the present invention.
FIG. 2 is a flowchart illustrating an example of processing of a control unit 11.
FIG. 3 is a flowchart showing another example of processing by the control unit 11;
FIG. 4 is an explanatory diagram illustrating an example of an interface of an apparatus to which the image search apparatus according to the embodiment of the present invention is applied.
[Explanation of symbols]
11 control unit, 12 storage unit, 13 database unit, 14 display unit, 15 operation unit, 16 external storage unit.

Claims (4)

An image search device for searching a search target image data portion to be searched from target image data to be processed,
Means for defining at least one search area in the target image data;
For at least one type of conversion, a plurality of learning samples converted from different predetermined conversion amounts from a predetermined reference state are used, and learning information associated with conversion amounts for returning each of the learning samples to the reference state is learned. With reference to the obtained conversion database, for the search partial data included in each of the defined search areas, the conversion including the type of conversion and the conversion amount to be applied in order to set the search partial data as the reference state Conversion means for acquiring a condition, repeatedly performing conversion based on the acquired conversion condition until the conversion amount indicated by the conversion condition for each search partial data is “0”, and converting the search partial data into a reference state When,
Search means for referring to a search database acquired by learning using an example of image data of a search target in the reference state, and determining whether a search target is included in each of the converted search partial data;
Means for generating and outputting search result information for specifying a region on the target image data corresponding to the search partial data determined to include a search target;
Including
An image search device, wherein the generated search result information is subjected to a predetermined process. The image search device according to claim 1,
The conversion database includes a database group for storing conversion conditions obtained by learning for each type of conversion,
The conversion means searches each database group included in a conversion database, and for the search partial data included in each of the defined search areas, conversion to be applied to set the search partial data as a reference state. The conversion condition including the type and the conversion amount is acquired, and the conversion based on the acquired conversion condition is repeatedly performed until the conversion amount indicated by the conversion condition for each search portion data becomes “0”, and the search portion An image search apparatus characterized by converting data into a reference state .   An image search method for searching a search target image data portion to be searched from target image data to be processed, using a computer,
  Means for defining a search area defining at least one search area in the target image data;
  The conversion means uses a plurality of learning samples converted by a different conversion amount from a predetermined reference state for at least one type of conversion, and associates information of the conversion amount for returning each of the learning samples to the reference state. With reference to the conversion database obtained by learning, the type of conversion and the conversion amount to be applied in order to set the search partial data as the reference state for the search partial data included in each of the defined search areas. The conversion condition is acquired, and the conversion based on the acquired conversion condition is repeatedly performed until the conversion amount indicated by the conversion condition for each search partial data becomes “0”, and the search partial data is set to the reference state. Converting, and
  The search means refers to the search database acquired by learning using the image data example of the search target in the reference state, and determines whether or not the search target is included in each of the converted search partial data Process,
  An output means for generating and outputting search result information for specifying an area on the target image data corresponding to the search partial data determined to include a search target; and
  Including
  An image search method, wherein the generated search result information is subjected to a predetermined process.   A program for searching a search target image data portion to be searched from target image data to be processed,
  Computer
  Means for defining at least one search area in the target image data;
  For at least one type of conversion, a plurality of learning samples converted from different predetermined conversion amounts from a predetermined reference state are used, and learning information associated with conversion amounts for returning each of the learning samples to the reference state is learned. With reference to the obtained conversion database, for the search partial data included in each of the defined search areas, the conversion including the type of conversion and the conversion amount to be applied in order to set the search partial data as the reference state Conversion means for acquiring a condition, repeatedly performing conversion based on the acquired conversion condition until the conversion amount indicated by the conversion condition for each search partial data is “0”, and converting the search partial data into a reference state When,
  Search means for referring to a search database acquired by learning using an example of image data of a search target in the reference state, and determining whether a search target is included in each of the converted search partial data;
  Means for generating and outputting search result information for specifying a region on the target image data corresponding to the search partial data determined to include a search target;
  Function as
  A program characterized in that the generated search result information is subjected to a predetermined process.

Images