JP7329933B2 - Similar image retrieval device, similar image retrieval method and similar image retrieval program - Google Patents

Description

When an image is given as a search query, the present invention selects similar images from among a plurality of images to which classification information has been assigned, considering the similarity based on the classification information in addition to the similarity based on the shape of the image. Apparatus, method and program for finding

In recent years, a technique for retrieving an image using an image as a retrieval query has attracted attention. It is expected to be applied to many fields such as autonomous driving, security systems, and robotics. Many of them perform searches based only on the graphical and shape features of the image.
However, there are fields in which a classification system has been established for images, and classification information based on the classification system has been manually assigned, sorted, and accumulated. One specific example is the examination of trademarks, especially figure trademarks.

Trademarks are signs attached to products and services that businesses and companies provide, and play an important role in showing brand images. In order to obtain exclusive trademark rights, it is necessary to apply to the Patent Office, undergo examination, and be approved for registration. Trademark examination is the process of confirming whether or not there are already registered trademarks (registered trademarks) that are similar to the registered trademark in terms of appearance, concept, and pronunciation. If a similar one is found among registered trademarks, it will be rejected. This leads to lost business opportunities. Therefore, if an applicant can search for similar trademarks using a system similar to that used by trademark examiners before filing an application with the Patent Office, it will be possible to reduce the risk of rejection after the trademark application and prevent loss of business opportunities. it is conceivable that.
Consider the trademark examination process. It is unrealistic to manually evaluate the similarity of all registered trademarks in large quantities. In particular, it is not easy to search for similar trademarks that include graphics or design characters. Therefore, the JPO uses its own trademark search system to narrow down the registered trademarks and then evaluates the similarity.

In this specification, explanations will be directed to the most basic of the appearances, concepts, and appellations, and the analogy of appearances, which has a large number of instances. To narrow down trademarks that are similar in appearance, the Wiener Classification, which classifies the graphic elements of trademarks, is used. Specifically, the Vienna classification code given by the expert to the applied trademark is entered into the system and compared with the Vienna classification code of the registered trademark. Registered trademarks that have undergone examination have already been assigned a Vienna Classification Code. Then, the trademarks on which the figures of the same classification are drawn are output. The trademark examiner determines the similarity of the trademarks narrowed down in this way to the applied trademark.

Conventionally, several methods have been proposed for retrieving appearance similarities of trademarks. However, it mainly searches for similar trademarks based on shape similarity. However, there are cases where, for example, even if the shapes are different, the trademarks are considered to be similar if the figures of the same classification are drawn. In other words, the method of retrieving similar-looking trademarks must consider not only the shape of figures but also their classification.

Retrieval that also considers the classification of figures requires similarity evaluation based on the classification of figures. Regarding registered trademarks, the Vienna Classification Code is assigned to registered trademarks in many countries, including Japan. On the other hand, since the applied trademark is a new trademark, it has not been given a Vienna classification code. Expert knowledge and rules of thumb are indispensable for assigning a Vienna Classification Code to an applied trademark. However, as far as we know, there is no case study that automatically assigns the Vienna classification code.
Deep learning is a technology that has the potential to solve this problem. Image classification using deep learning extracts the optimal features for classification without human intervention, dramatically improving classification accuracy. Therefore, we describe related research on image classification using deep learning.

Research by Saito et al. includes Illustration2vec, which assigns classification to illustrations (see, for example, Non-Patent Document 1). In this research, the elements drawn in the illustration are represented by 1,539 classifications, and a network is created that automatically assigns multiple classifications to the input illustration. For example, if an illustration of a blue-eyed, blonde girl is input into the network, it can be classified as "blue-eyed," "single," "blonde," and "girl." In addition, Pan et al. are conducting research on DeepFood, which classifies images of food into one type out of 41 types (see, for example, Non-Patent Document 2). Using a data set of 100 images for each food, three-fold cross validation was used to classify the foods with an accuracy of about 85% or more. In this way, in recent years, research on image classification using deep learning has produced remarkable results. In addition, He et al. are conducting research on ResNet that recognizes objects in natural images from among 1,000 classes of objects (see, for example, Non-Patent Document 3). ResNet achieved the highest accuracy in the Large Scale Visual Recognition Challenge 2015 image recognition competition, and its recognition error rate is very low at about 4%, making it a representative method among image classification methods. there is

M Saito, and M Yusuke. “Illustration2vec: a semantic vector representation of illustrations.” SIGGRAPH Asia 2015 Technical Briefs. ACM, (2015), pp. 5:1-5:4 C. Liu, Y. Cao, Y. Luo, G. Chen, V. Vokkarane, and Y. Ma, “Deepfood: Deep learning-based food image recognition for computer-aided dietary assessment,” International Conference on Smart Homes and Health Telematics. Springer, Cham, (2016), pp. 37-48 K. He, X. Zhang, S. Ren and J. Sun “Deep residual learning for image recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. (2016), pp.770-778

As described above, in order to search for registered trademarks similar in appearance to an applied trademark to which no Vienna classification code has been assigned, it is preferable to perform the search after assigning a suitable Vienna classification code to the applied trademark. However, the task of assigning Vienna Classification Codes to unassigned trademarks is troublesome and lacks stability for applicants who are ordinary companies or businesses. If the Vienna classification code can be assigned automatically, anyone can use it easily, and a more efficient search method for similar trademarks can be realized.
The present invention has been made in view of the circumstances described above, and is capable of determining similarity based on classification information in addition to similarity based on image shape from among a plurality of images to which classification information has been assigned in advance. is considered to provide a method for finding images similar to the query image. For example, as a method for retrieving trademarks that are similar in appearance, a search method that considers similarity based on the classification of figures in addition to similarity based on the shape of figures is implemented.

This invention
(1) A query feature extraction unit that extracts one or more shape features from a search query image, and a plurality of reference images to which one or more classification information is assigned in advance and from which one or more shape features are extracted, A shape matching unit that compares a search query image with shape features, a shape similarity calculation unit that calculates the shape similarity of each reference image to the search query image based on the matching, and a classification given to each reference image A classification assigning unit for estimating and assigning one or more pieces of classification information to a search question image based on information, and based on the degree of matching between the classification information assigned to the search question image and the classification information assigned to each reference image. a classification similarity calculation unit that calculates the classification similarity of each reference image to a search question image; and an output unit that outputs one or more reference images similar to the search question image based on the shape similarity and the classification similarity. To provide a similar image search device comprising:

(2) A process executed by a computer, which includes a query feature extraction process for extracting one or more shape features from a search question image, and one or more classification information given in advance and one or more shape features extracted. A shape matching process of referring to a plurality of reference images and matching the search question image and shape features, and a shape similarity calculation process of calculating the shape similarity of each reference image to the search question image based on the matching. , a classification assignment process for estimating and assigning one or more classification information to a search question image based on the classification information assigned to each reference image; and the classification information assigned to the search question image and the classification assigned to each reference image. a classification similarity calculation process for calculating the classification similarity of each reference image to a search question image based on the degree of matching with information; and one or more similar to the search question image based on the shape similarity and the classification similarity and an output process for outputting the reference image of the similar image retrieval method.

(3) A program for causing a computer to execute processing, which is a program for causing a computer to execute processing, which is a query feature extraction process for extracting one or more shape features from a search question image, and one or more classification information in advance and a plurality of reference images from which one or more shape features are extracted, shape matching processing for matching the search question image and shape features, and each reference image for the search question image based on the matching A shape similarity calculation process for calculating the shape similarity of each reference image, a classification assignment process for estimating and assigning one or more classification information to the search question image based on the classification information assigned to each reference image, and a classification assignment process for assigning to the search question image a classification similarity calculation process for calculating a classification similarity of each reference image to a search query image based on the degree of matching between the classification information assigned to each reference image and the classification information assigned to each reference image, and the shape similarity and the classification and output processing for outputting one or more reference images similar to a query image based on similarity.

A similar image retrieval apparatus according to the present invention includes a shape similarity calculation unit that calculates the shape similarity of each reference image to a search question image based on matching, a classification imparting unit that assigns classification information to the search question image, and a classification information a classification similarity calculation unit that calculates the classification similarity of each reference image to the search question image based on the degree of matching of the reference image, and one or more references similar to the search question image based on the shape similarity and the classification similarity and an output unit that outputs an image, so that an image similar to the search query image can be found by considering the similarity based on the classification information in addition to the similarity based on the shape of the image.
The same applies to the similar image retrieval method and similar image retrieval program according to the present invention.

FIG. 4 is an explanatory diagram showing the classification system of the Wien classification code according to this embodiment; FIG. 4 is an explanatory diagram showing an example of a Vienna classification code assigned to a registered trademark according to this embodiment; FIG. 10 is an explanatory diagram showing a different example of the Vienna classification code assigned to the registered trademark according to this embodiment; FIG. 11 is an explanatory diagram showing a further different example of the Vienna classification code assigned to the registered trademark according to this embodiment; FIG. 4 is an explanatory diagram showing an overview of the flow of processing for searching for similar-appearance trademarks according to this embodiment; FIG. 6 is an explanatory diagram showing details of giving a Wien classification code among the processes shown in FIG. 5; It is explanatory drawing which shows the calculation example of the classification similarity a in this embodiment. 10 is a graph showing the number of data used for learning the large classification of the Wien classification code for each large classification in an experimental example in this embodiment. It is a graph which shows the result using the threshold parameter N and the weight parameter m with the highest prediction accuracy in this embodiment, and its peripheral value. FIG. 10 is a diagram showing a specific example of a Wien classification code given by machine in this embodiment; FIG. 4 shows different examples of machine-assigned Wien Classification codes in this embodiment; FIG. 11 shows yet another example of a machine-assigned Wien Classification code in this embodiment; It is a graph which shows the 11-point complement average precision of the method by this embodiment, the search method of the assignment|providing by an expert, and the existing method. FIG. 10 is an explanatory diagram showing an example of search results of appearance-similar trademarks according to this embodiment; FIG. 10 is an explanatory diagram showing a different example of appearance-similar trademark search according to this embodiment;

The present invention will be described in further detail below with reference to the drawings. It should be noted that the following description is illustrative in all respects and should not be construed as limiting the present invention.
As an example of images that have been assigned classification information and have been sorted and accumulated, a registered trademark assigned with a Vienna classification code will be given below, and a specific image search will be described.
≪Vienna Classification and Coding System≫
First, the Wien classification related to this invention will be described.
The Vienna Classification is an international index created in 1973 in Vienna, Austria, that classifies figurative elements contained in trademarks according to features such as shape. In Japan, the Vienna Classification has been used for trademark examination since 2004.

As shown in Fig. 1, the Wien Classification has a tree structure of large, medium, and small classifications, each of which consists of 29/144/1887 classifications. For example, major categories are divided into 29 categories as shown in Table 1.
There are classifications such as major category 1 (celestial bodies, natural phenomena, maps) representing objects and phenomena, and major category 26 (geometric figures and three-dimensional geometric figures) representing shapes of figures. Focusing on the major categories belonging to major category 1 (celestial bodies, natural phenomena, and maps), major category 1.1 (stars and comets), major category 1.3 (sun), major category 1.5 (earth, globes and planets), and major category 1.7 (Moon), major category 1.1 (stars, comets), minor category 1.1.1 (stars), minor category 1.1.2 (one star), minor category 1.1.3 (two stars) finely divided. In other words, in the Wien classification, the lower the classification, the more complicated elements of the figure can be indicated.
Basically, the Vienna Classification Code is expressed as a combination of large, medium, and small classifications, but Japan has its own fine classifications, which are one level finer than the small classifications.

Examples of Vienna classification codes actually assigned to registered trademarks are shown in FIGS. 2 to 4. FIG. Each figure shows the trademark in question on the top, and the Vienna classification code given to the trademark on each row of the table on the bottom. The first column of the table shows the assigned Wien classification code, the second column shows the major classification, the third column shows the middle classification, the fourth column shows the minor classification, and the fifth column shows the fine classification.

The trademark (Registered Trademark No. 1406589) dealt with in FIG. 2 is a registered trademark written as "TAMIYA", and two figures in which a star is drawn in a square are arranged above the alphabet. Since the part representing the figure of this registered trademark is the part where a star is drawn in a square, ) have been given 7 Viennese classification codes. Among them, the Vienna classification code that expresses "stars" is major category 1 (celestial bodies, natural phenomena, maps), major category 1 (stars, comets), minor category 1 (stars), and is listed as 1.1.1. be done. This star has also been assigned the Vienna Classification Code, which overlaps with 1.1.1 (star) and 1.1.3 (two stars). The assignment of such duplicate Vienna classification codes is considered to be basically done to prevent search omissions, but such judgments are based on expert knowledge and empirical rules.

Next, FIG. 3 shows the registered trademark (registered trademark No. 3220448) and the Vienna classification code, which is obtained by transforming characters into figures. In this example, three horizontal lines are drawn in the "H" part written as "Hisamitsu", which is a special typeface. For these reasons, the Vienna classification code 27.5.1.8 (H,h) has been assigned. In addition, as the Vienna classification code 27.5.21 (one letter) is given, the part where the Vienna classification code is given is only the "H" part, and the remaining "isamitsu" has the Vienna classification code. Not granted. In this way, the criteria for judging which character represents a graphic element also depend on expert knowledge and empirical rules.

Finally, FIG. 4 shows the registered trademark (registered trademark No. 4624372) using characters and the given Vienna classification code. The registered trademark depicts a circular character with limbs and has been given seven Vienna Classification codes. Focusing on the Vienna Classification Code 4.5.5, it states "personification of other unidentifiable things and other imaginary things in human form." "Others" here refers to other Vienna classification codes under the middle classification 4.5 (e.g. 4.5.1 (anthropomorphic plants, combinations of anthropomorphic plants) and 4.5.4 (human-looking robots) etc.). Such a Wien Classification code that includes "Others" requires expert knowledge because other Wien Classification codes of the target layer must also be considered.

≪Overview of Similar Appearance Trademark Search≫
FIG. 5 is an explanatory diagram showing an overview of the flow of processing for searching for similar-appearance trademarks according to this embodiment. As shown in FIG. 5, the appearance-similar trademark search process can be roughly divided into a registration process 11 and a search process 21 .
In this embodiment, the registration process 11 and the search process 21 are realized by a computer having hardware resources such as a CPU (Central Processing Unit) and memory executing a processing program stored in advance in the memory.
Registration processing 11 is processing for the computer to register the trademark image in the database. The computer extracts local features from the trademark image to be registered (registered trademark 13 shown in FIG. 5, hereinafter also referred to as database image) (feature extraction processing 15 shown in FIG. 5). Then, the extracted local features are registered in the database 17 in association with the Vienna classification code assigned to the registered trademark 13 .
The search process 21 is a process in which the computer measures the similarity based on the shape as a graphic of the search question 23 and the Wien classification, and searches for trademarks with similar appearances based on the similarity.
Here, the computer that executes the search process may be the same as the computer that executes the 21 registration process 11, or may be different.

Further, the search processing 21 is divided into processing for measuring similarity based on the shapes of figures and processing for measuring similarities based on the classification of figures.
First, the process of measuring the similarity based on the shape of figures will be described.
The computer first extracts local features from a trademark image (hereinafter, query image) that serves as a query (search question 23 shown in FIG. 5) (query feature extraction processing 25 shown in FIG. 5). Next, for each database image registered in the database 17, the local feature extracted from the database image and each local feature obtained from the query image are matched (shape matching process 27 shown in FIG. 5). Then, verification processing is performed to evaluate whether each collated local feature is a reliable feature (verification processing 29 shown in FIG. 5).
In the verification process 29, the computer considers the matching of local features in locations that are locally similar but not globally similar to be unreliable, and does not include the matched local features in the calculation of the shape similarity. to dispose of. In other words, the search for dissimilar trademarks is suppressed by considering the positional relationship of each local feature.
More specifically, the reliability of the matched local features is determined by evaluating the relative positional relationship with other matched local features, and those with low reliability are discarded.
Reliability here refers to whether or not the matched local features contribute to the determination of shape similarity. Since each local feature is a feature focused on a local region of an image, it may correspond to a local feature of an image that is globally dissimilar. Such local features do not contribute to the determination of shape similarity and are merely noise. In verification process 29 the computer removes such noise.
Finally, the computer measures the similarity s (hereinafter referred to as shape similarity s) based on the shape of the figure in the database image based on the number of matches evaluated as reliable local features (shape similarity s shown in FIG. 5). Similarity calculation processing 31).
Details of the verification process 29 and the shape similarity calculation process 31 can be found in the research by Kudo et al. of the Imaging Society of Japan, 57, pp.403-412 (2018)).

Techniques using local features can search for globally and partially similar trademarks. However, since partially similar trademarks are not necessarily similar trademarks, this method also causes searches for dissimilar trademarks.
In order to solve this problem, in this embodiment, a process of suppressing retrieval of dissimilar trademarks and a process of promoting retrieval of similar trademarks are introduced into the retrieval method based on local features. The former suppression of dissimilar trademarks is handled by considering the positional relationship of local features. Pseudo relevance feedback, see R. Kudo, M. Iwata, M. Iwamura, and K. Kise, “Improvement of Trademark Retrieval Using Pseudo Relevance Feedback,” 11th Indian Conference on Vision, Graphics and Image Processing, (2018)).
However, even with these methods, it is still not possible to retrieve all similar trademarks. It is necessary not only to evaluate the appearance similarity of trademarks based on the degree of similarity in shape of figures, but also to consider the similarity based on the classification of figures.

Next, processing for measuring similarity based on the classification of graphics will be described. The computer first assigns a Wien classification code to the query image (machine assignment by the Wien classification code assignment process 33 shown in FIG. 5). One or more Vienna classification codes are assigned. Next, the similarity a (hereinafter referred to as classification similarity a) based on the classification of the figure is measured from the number of matches between the Wien classification code of the query image and the Wien classification code of the database image (classification similarity calculation shown in FIG. 5). process 35). Based on the obtained shape similarity s and classification similarity a through these processes, an image to be output is determined in descending order of overall similarity.
As mentioned above, in order to improve the search accuracy of similar trademarks, this embodiment employs pseudo-match feedback. The details of each process will be described below.

[Vienna classification code assigning process]
The computer applies a deep learning technique to assign a Wien classification code to the trademark as the query image. The Wien classification code assigning process is divided into two stages, a learning process and an assigning process.
Among them, in the learning process, registered trademarks that have already been assigned a Vienna classification code are used, the figure of each registered trademark is input, and 1,644 small classifications of the Vienna classification code assigned to the registered trademark are output. It learns the input-output relationship when In this embodiment, fine classification, which is an index unique to Japan, is not subject to learning. That is, for a registered trademark assigned a Vienna classification code that includes a detailed classification, the major, middle, and minor classifications excluding the detailed classification are used for learning. For example, the Vienna classification code 2.9.1.2 (Trump's Heart) learns as 2.9.1 (Heart).

The type of deep learning network used for the learning process of the Wien classification code assigning process 33 is ResNet (Residual Network), which is a typical network for image recognition. ResNet is generally a model that uses three types of layers, 50 layers, 101 layers, and 152 layers, but this embodiment uses the simplest ResNet50.
The loss function handled by ResNet50 usually uses softmax cross entropy that handles single labels. However, one registered trademark may be assigned not only one Wien classification code, ie, labeled, but also multiple labels, so this embodiment uses sigmoid cross entropy to handle multi-labels. By using sigmoid cross entropy, we perform binary multi-label classification where each Wien classification code of interest exists or does not exist. Also, by passing the sigmoid function, the elements of the output vector are normalized from [-∞, ∞] to [0, 1]. The closer the normalized value is to 1, the more likely it is that the corresponding Vienna Classification code has been assigned to the trademark. In addition, normalization of data used for learning and data augmentation are used as techniques for improving the accuracy of assigning the Wien classification code.

Next, the assignment processing of the Vienna classification code assignment processing 33 will be described. FIG. 6 shows the flow of provision processing. The computer first inputs a new trademark (search query 23) to which a Vienna classification code is to be assigned into the trained model described above. Then, the output of the Vienna classification code for the input trademark is obtained from the trained model. The trained model outputs a 1,644-dimensional vector as output. Each dimensional element of the output vector corresponds to each small classification of the Wien classification code that has been trained. The search question shown in FIG. 6 shows an example in which registered trademark No. 4772234 is entered instead of a new trademark. For each dimension, the computer assigns the Wien classification code corresponding to the element to the trademark of the search query 23 if the element is equal to or greater than the threshold N. For example, if the element of the vector corresponding to the Wien classification code 2.5.6 (infant) is 0.78, then if N=0.5, the query 23 is given the Wien classification code 2.5.6. However, if N=0.8, the Wien classification code is not assigned (see FIG. 6). The threshold parameter N used here is determined by experiments described later.

[Classification similarity calculation process]
In the classification similarity calculation process 35 shown in FIG. 5, the computer calculates the classification similarity a of the database image using the Wien classification code. That is, the classification similarity a is calculated from the number of the same Wien classification codes given to the query image and the database image. In this case, if the number of corresponding Wien classification codes is defined as the classification similarity a as it is, the more the number of Wien classification codes given to the database image, the higher the classification similarity a. Therefore, the classification similarity a used in this embodiment is normalized by the number of Wien classification codes assigned to database images.
In other words, if the set of Wien classification codes assigned to the trademark x in the query image is X and the set of Wien classification codes assigned to the database image y is Y, then the classification similarity a is defined as follows: . where |Y| represents the number of Wien classification codes assigned to the database image y.

It should be noted that dividing by the value of the number of Wien classification codes of the database image whose square root is taken in the normalization is obtained from knowledge obtained from prior experiments.
FIG. 7 shows a calculation example of the classification similarity a. Three Wien classification codes are assigned to this query image, and five Wien classification codes are assigned to the database image (Registered Trademark No. 4488647). Also, there are two numbers with the same Vienna classification code. That is, in the example of FIG. 7, the classification similarity a of the database image is 2/√5.

[Output processing]
After going through the above process, the computer outputs all the database images matched with the query image (output process 37 shown in FIG. 5). In the output process 37, the computer calculates the overall similarity s' based on the shape similarity s and the classification similarity a of the database image, and outputs the values in descending order of the overall similarity s'. The following formula is used for calculation.
As shown in the formula (2), the shape similarity s and the classification similarity a have a complementary relationship with respect to the constant parameter m indicating the weight of the shape similarity s and the classification similarity a of the database image. Note that the value of the weight parameter m is determined by an experiment to be described later.

[Pseudo relevance feedback]
Pseudo-relevant feedback (see Pseudo-relevant feedback process 39 in FIG. 5) is a technique for extending the search query 23 to reduce search omissions. First, using the original query image (hereinafter referred to as initial query), output processing is performed to obtain the top N images (initial output) in descending order of total similarity s′. After that, the top N images are used as a new search question 23 and the search is performed again. The output obtained by the second search is used to update the initial output and obtain the final output 41 . By using this technique, similar trademarks missing in the initial search can be included in the final output 41 . The following formula is used to obtain the final overall similarity score s'' for each trademark.
In equation (3), _s'0 indicates the total similarity s' when the initial query is used. s' _k indicates the overall similarity s' when the k-ranked trademark in the search results is used in the search query.

≪Experiment≫
Two experiments were performed to investigate the performance of the approach according to this embodiment.
In a first experiment, we examine the appropriate search parameters, specifically the threshold parameter N and the weight parameter m, to use in searching for this approach. Due to the nature of trademark searches, it is most important to avoid missing similar trademarks. Therefore, we investigated the trend by changing the search parameters, and obtained the search parameters (hereinafter referred to as the optimum search parameters) for the case where the search omission of this method is minimized.

In the second experiment, in order to demonstrate the effectiveness of this method, we compared the search results of a method of searching for trademarks similar in appearance only by the shape similarity s (hereinafter referred to as the existing method) and the method according to this embodiment. made a comparison. We also investigated how much the search accuracy of the method according to this embodiment improves when the Wien classification code is correctly assigned to the search query 23 . As a specific method, the Wien classification code assigning process 33 shown in FIG. The search results of the given search method) and the method according to this embodiment were compared.

[First Experiment: Experiment to Investigate Appropriate Parameters Used for Search]
(Dataset used in the experiment)
623,101 images of Japanese registered trademarks were used as database images, and 4,222 images were used as query images. First, 1,000 out of 4,222 query images were used to find the optimal search parameters, and then the remaining 3,222 images were used to confirm the accuracy of machine assignment of Wien classification.
The number of correct images for the 1,000 query images was 14,226, and the number of correct images for the remaining 3,222 query images was 45,422. This correct image was determined by visual evaluation one by one with the cooperation of a trademark searcher and 28 graduate students. In this case, "an image in which it can be judged that the same shape clearly exists in part or all of the image without considering the difference in resolution and color with respect to the query image". made me choose. In addition, the data set used in this experiment does not have completely identical trademarks, and trademarks with differences in resolution, color, etc. were used.

(Experimental conditions)
The CPU used in the experiment was an AMD Opteron (registered trademark) Processor 6238 and the memory was a computer with 512 GB. Also, 9 NVIDIA Corporation GP102 GPUs were used. The following parameters are used when obtaining the shape similarity s. For details of various parameters, refer to the research by Kudo et al. (R. Kudo, M Iwata, M. Iwamura, and K Koichi, “Proposal of Similar Image Retrieval System Using Large Scale Trademark Image Database,” Journal of the Imaging Society of Japan, 57, pp.403-412 (2018)).
The parameters used to calculate the shape similarity s are as follows.
Number of local features to search by estimated distance C = 1000
Parameter k = 10 in k-neighbor search
The parameters used in this experiment were changed from the parameter k=50 used in the study by Kudo et al. to k=10. This is because the parameters have been changed to enable the fastest and most accurate search by adding correct images and correcting the implementation of the program.

ResNet50 was trained using 694,428 Japanese registered trademarks. A maximum of 40 Vienna classification codes and a minimum of 1 were assigned to the registered trademarks used for training. Furthermore, the average number assigned to the Vienna classification code was 8.1 with a standard deviation of 5.75. In order to analyze the learning data in more detail, FIG. 8 shows the number of data used for learning each major classification. In FIG. 8, the horizontal axis indicates the major classification, and the vertical axis indicates the number of Wien classification codes used for learning. From this graph, the number of learning data for major category 26 (geometric figures and three-dimensional geometric figures) is large at 2,332,188, and the smallest number of learning data for major category 12 (furniture, sanitary facilities) is 5,617. I understand. In other words, there was a bias in the Vienna Classification Code for registered trademarks, resulting in learning imbalanced data.

The learning parameters used in ResNet50 are as follows.
Batch size: 512
Number of epochs: 100
optimizer : Adam(alpha=0.0001, beta1=0.9, beta2=0.999, eps=1.0 x 10-8, eta=1.0)
Decrease alpha: 1/10 alpha value at 60epoch and 90epoch
Data normalization: 1/255 scaling by subtracting the average value of the learning data set for each image channel used for learning
Data augmentation: Random Crop, Horizontal Flip
Parameter initialization of ResNet50: Following the work of He et al. (K. He, X. Zhang, S. Ren and J. Sun “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” arXiv preprint arXiv, (2015 ) reference).

(Details of the experiment to find the optimal search parameters)
In this experiment, the optimal search parameters for this method were obtained by grid search using 1,000 query images, and the tendency when each search parameter was changed was investigated.
Furthermore, to examine the performance of the machine assignment when using the optimal search parameter N, we checked the difference with the Wien classification code assigned by experts for 3,222 queries.

[Experimental conditions]
As the index with the least search omissions, the precision when the recall is 100% was used. In this experiment, the recall rate can be said to be the ratio of the trademarks to which the Vienna classification code was assigned by machine, out of all the trademarks that should have been given the Vienna classification code, among the trademarks that were used as search queries. A recall rate of 100% means that, for each class, the Vienna Classification Code was automatically assigned to all trademarks that should have been given the Vienna Classification Code of that class. On the other hand, the precision rate in this experiment can be said to be the ratio of trademarks to which a certain Vienna classification code should not have been assigned, out of all the trademarks to which a certain Vienna classification code should have been assigned. 11-point imputed mean precision (Edited by K. Kita, K. Tuda and M. Shishibori, “Jyouhoukensaku arugorizumu,” Kyoritsu Shuppan (2002), pp.20-21) was used to calculate recall and precision. adopted. The search parameters targeted in this experiment are the following two points, and in addition, the values to be investigated for each search parameter are shown below.
Threshold Parameter N for Machine Assignment of Wien Classification Codes
Value to be investigated: Weight parameter m of shape similarity s and classification similarity a of database images set in 0.1 intervals from 0.1 to 1.0
Value to investigate: set from 0.01 to 0.2 in 0.01 intervals

[Experimental results and discussion]
First, the threshold parameter N and the weight parameter m, which indicate the highest precision when the recall is 100%, were examined using the above values. FIG. 9 shows the result of using the threshold parameter N that indicates the highest matching rate and its surrounding values.
In this figure, the vertical axis plots the precision when the recall is 100%, and the horizontal axis shows the weight parameter m. The values of N and m showing the highest precision were 0.3 and 0.1, respectively. Using these parameters in the search, the technique in this embodiment was able to obtain a precision of about 67%.
Next, the tendency of the precision when each parameter N and m is changed is examined.
First, when comparing the peak matching ratios showing the highest values when m is changed for each set value of the threshold parameter N, the maximum peak matching ratio is 66.5% when N = 0.3 and m = 0.1. be. On the other hand, the minimum peak matching ratio is 66.75% when N=0.5 and m=0.09, and the difference is within 0.2%. From this, no significant difference was found in the change of the peak matching rate with respect to the threshold parameter N.
On the other hand, when the weight parameter m is increased, the matching rate tends to decrease near m=0.15 for any threshold parameter N, but the degree of decrease is gentler as the value of N increases. That is, the larger the value of N, the smaller the influence of increasing the value of the weight parameter m.

Next, 3,222 queries were examined to see if the Wien classification was correctly assigned when the threshold parameter N=0.3. The rate of machine assignment was 50.2% for recall and 50.6% for precision. Machine assignment of the Wien Classification code may not achieve sufficiently high performance.
A specific example of a machine-assigned Wien Classification code is shown and discussed in FIGS. 10-12. These examples show the query image of interest and the Wien classification code assigned to it. The expert-assigned Wien classification code is shown enclosed by a solid line, and the machine-assigned Vienna classification code is shown enclosed by a dashed line.
FIG. 10 shows an example in which all expert-given Wien Classification codes could be correctly machine-given. In the search query image (Registered Trademark No. 4170472) shown in FIG. 10, a crescent moon with a human face is depicted, and all eight Wien classification codes assigned by experts can be accurately machine-assigned. In this way, there were 519 examples in which all Vienna classification codes could be correctly assigned.

FIG. 11 shows an example in which none of the expert-given Wien Classification codes could be machine-given. In the search question image (registered trademark No. 4764206) shown in FIG. 11, an animal character is drawn. None of the 7 expertly assigned Wien Classification codes were assigned by machine assignment. Thus, there were 54 cases in which no 11 Vienna classification codes were machine assigned.
FIG. 12 shows an example in which omissions or erroneous Wien classification codes are assigned to the Wien classification codes assigned by an expert in machine assignment. In the search question image (Registered Trademark No. 4341421) shown in FIG. 12, a man with a horse is depicted. Of the 16 Wien classification codes assigned by experts, 9 have been machine assigned, but the remaining 7 have not been assigned and omissions have occurred. In addition, 6 Wien classification codes are erroneously machine assigned. In this way, there were 2,649 cases of omission or incorrect application, excluding cases in which none were applied mechanically.

Next, the specific example of FIG. 12 is scrutinized. There are 7 Vienna Classification codes that could not be assigned by machine, but there are 03.03.01 (horses, mules) and 02.01.20 (men riding or accompanied by horses, mules or donkeys) that could be assigned in common. A lot of code and duplicate content. In addition, 4 out of 6 Vienna Classification Codes that were erroneously assigned by machine are related to Major Classification 29 (colors), such as 29.01.01 (red, pink, orange), which seems to represent the trademark correctly. There was also, but the experts did not grant it.
From the above points, it is considered that experts do not sufficiently learn the tendency to assign duplicate Wien classification codes and the tendency to assign color Wien classification codes. Also, 26.01.16 (circle or ellipse containing other graphic elements) is machine assigned, but it contains "other" and other Vienna classification codes of the target layer must also be considered. , is considered to be difficult to assign accurately. For reference, 288 were given by experts with the Vienna classification code 26.01.16, and 260 were given by machine, but none of them could be given correctly.

[Second experiment: Experiment showing the effectiveness of this method]
In this experiment, 3,222 query images were used to demonstrate the effectiveness of this method with optimal search parameters, and a comparison with existing methods was performed.
(Experimental conditions)
In this comparison, as in the first experiment, the 11-point imputed average precision (see Edited by K. Kita, K. Tuda and M. Shishibori, “Jyouhoukensaku arugorizumu,” Kyoritsu Shuppan (2002), pp.20-21) It was adopted. In addition, in order to investigate whether there is a significant difference between the search of this method and the existing method, we calculated the average precision (Average Precision) and performed a statistical test. The average matching rate is an index that measures how many matching images appear at the top of the ranking. The test used is the Wilcoxon signed rank sum test (see Frank Wilcoxon, "Individual comparisons by ranking methods," Biometrics bulletin, 1, pp. 80-83 (1945)). Wilcoxon's signed rank sum test is used when there is correspondence between two data, and is a method that recognizes a significant difference if there is a difference in the median values of the data.

In addition, in order to investigate how much the retrieval accuracy improves when the Vienna classification code is assigned correctly, we compared it with the retrieval method assigned by experts. The optimal weight parameter m for the retrieval method given by experts was 0.13 as a result of examining by grid search in the same manner as in the first experiment. Therefore, the parameter m is set to 0.13 in the search method for granting by experts. Finally, a specific search example will be shown and discussed.

(Experimental results and discussion)
FIG. 13 shows the results of the 11-point complement average precision of the method according to this embodiment, the search method given by experts, and the existing method. A recall-precision curve (also referred to as a Precision-Recall curve or a PR curve) as shown in FIG. 13 is often used as an indicator of prediction accuracy in the field of machine learning. When registered trademarks similar to the new trademark being tested were ranked in the order of predicted similarity, it shows the accuracy of the search, as to whether similar trademarks with a high degree of similarity were actually grouped at the top. I can say.
From this result, it can be seen that the precision rate is high in all recall rates in order of the retrieval method given by the expert, the method according to this embodiment, and the existing method.
Also, the precision of the method according to this embodiment and the existing method when the recall is 100% will be compared. When the recall rate is 100%, the precision rate of the method according to this embodiment is 66.5%, which is about 3.0% higher than the 63.5% of the existing method. As a result of applying the test method to the average precision of the existing method and this method, the P value was 7.71×10 ^(-85) . Since the P-value is below the significance level of 1%, it can be said that there is a significant difference between the average precision of this method and the existing method.
From the above, it can be said that it is effective to use the Wien classification code machine-assigned by deep learning for retrieval.

Furthermore, when compared with the search method given by experts, the precision rate of the search method given by experts is 68.5% when the recall rate is 100%, which is about 2.0% higher than the method according to this embodiment. it was high. From the above, if the accuracy of assigning the Wien classification code can be further improved, it is thought that the precision of the method according to this embodiment can be raised to the precision of the search method of assigning by experts.

14 and 15 show specific examples of the results of the search processing 21. FIG. 14 and 15 are divided vertically and horizontally into four parts. The upper left shows the query image, the upper right shows the existing method, the lower right shows the method according to this embodiment, and the lower left shows the top 15 search results of the search methods assigned by experts. The correct image is surrounded by a red frame.
FIG. 14 shows an example in which this method is superior to the existing method. In the existing method of FIG. 14, 6 correct images are retrieved up to the 15th place in the retrieval ranking. On the other hand, 11 correct images were retrieved up to the 15th place in the retrieval rankings by the retrieval method given by the expert and by this method. Focusing on the query image, what is drawn is a crest with a crown at the top, lions on both sides, and an arrow and a sword crossed in the center. On the other hand, the method of searching for grants by experts and the correct images retrieved by this method include emblems with crossed swords and arrows, crowns, shields, and the like. The reason why these correct images could be output is that for the trademark of the query image, there are Vienna A classification code was given.

On the other hand, FIG. 15 shows an example in which the existing method is superior to this method. Here, an example in which the search accuracy of the existing method was the highest is shown. According to the existing method, the correct image was retrieved at 1st and 2nd place, and in this method and the retrieval method given by the expert, the second correct image was retrieved at 5th and 11th place, respectively. As described above, in this method and the search method with the assignment by the expert, the search ranking of the second correct image was lowered. The reason is that the Wien classification code assigned to the query image and the Wien classification code of the second correct image do not match even one in both methods. Table 2 shows the number of query images for which the existing method has good retrieval accuracy and the number of query images for which this method has good retrieval accuracy. From Table 2, it can be seen that adding the machine-assigned Wien classification code to the search resulted in more query images with improved search accuracy.
≪Summary≫

This specification proposes a search method that evaluates similarity based on the classification of figures in addition to similarity based on the shape of figures in order to search for trademarks that are similar in appearance. A feature of the present invention is the use of deep learning machine-assigned Wien classification codes in evaluating the classification-based similarity of figures. Therefore, we first examined whether the Wien classification code could be accurately machine-assigned. As a result, the precision was 50.4% when the recall was 50.2%. From this, it may be said that the machine assignment of the Wien classification code does not achieve sufficiently high accuracy.

However, as a result of comparing the search accuracy of the method according to this embodiment and the existing method, the precision rate of the method according to this embodiment is improved by about 3.0% when the recall rate is 100%. Also, by improving the accuracy of machine assignment of the technique according to this embodiment, the search accuracy can be further improved by about 2.0%. As a result of applying the test method to the average precision of the method according to this embodiment and the existing method, it was demonstrated that there is a significant difference. From the above, the effectiveness of the search method that evaluates the similarity based on the classification of figures in addition to the similarity based on the shape of figures was demonstrated.

As mentioned above,
(i) A similar image retrieval device according to the present invention comprises a query feature extraction unit for extracting one or more shape features from a query image, and one or more classification information assigned in advance and one or more shape features extracted. A shape matching unit that refers to a plurality of reference images and compares the search question image and shape features, and a shape similarity calculation unit that calculates the shape similarity of each reference image to the search question image based on the matching. , a classification assigning unit that estimates and assigns one or more classification information per search question image based on the classification information assigned to each reference image; and the classification information assigned to the search question image and the classification assigned to each reference image. a classification similarity calculation unit that calculates the classification similarity of each reference image to a search question image based on the degree of matching with information; and one or more similar to the search question image based on the shape similarity and the classification similarity and an output unit for outputting the reference image.
In the present invention, the shape feature represents the shape feature of an image and is a feature amount applicable to image retrieval. Specific examples thereof include SIFT and local features obtained by improving it. Shape features are not necessarily limited to local features representing local features. However, recent advances in image retrieval are largely due to the contribution of local features.
Classification information is information labeled with an image, and the category to which the image belongs is determined by the given classification information. As a specific example of the classification information, the Vienna classification code given to the graphic trademark was described in the above-described embodiment, but it is not limited to this. However, in the classification information in this specification, categories are determined from a viewpoint different from the shape similarity of images. For example, the category is determined by including the idea, impression, meaning or symbol that the image gives to the viewer. Otherwise, similar image retrieval based only on shape features will suffice.
Furthermore, the reference image is an image to be searched for images similar to the query image. The reference image is provided with classification information in advance. As a specific example, the registered trademark was mentioned in the above-described embodiment, but it is not limited to this. In this specification, it is assumed that the reference image is given classification information in advance, but the search question image is used for retrieval without being given classification information.
The above embodiment and the query feature extraction processing shown in FIG. 5 correspond to the function of the query feature extraction unit according to the present invention. Similarly, the shape matching process corresponds to the function of the shape matching section according to the present invention, and the shape similarity calculation process corresponds to the function of the shape matching section according to the present invention. Furthermore, the Vienna classification code assigning process 33 shown in the above embodiment and FIG. 5 corresponds to the function of the classification assigning unit according to the invention, and the classification similarity calculation process corresponds to the classification similarity calculation part according to the invention. Similarly, output processing corresponds to the function of the output unit according to the invention.
The above-described embodiment assumes that each reference image is given a Wien classification code in advance. Applicable only to searches for trademarks that meet that premise. However, it is also conceivable to apply the Wien classification code assigning process 33 shown in FIG. 5 to the reference image. It is also conceivable to create a database for images to which the Wien classification code has been assigned. By doing so, the method described in this embodiment can be extended and applied to searches for unregistered trademarks and images other than trademarks that have not been given classification information in advance, such as illustrations and images of general objects. It is considered possible.

Furthermore, preferred embodiments of the present invention will be described.
(ii) the reference image is a registered trademark image, the classification information assigned to each reference image is a Vienna classification code, and a plurality of shape features related to the search question image and each reference image are extracted from each image; The shape matching unit compares each local feature related to the query image with each local feature related to each reference image, and determines the similarity of the matched shape features. may
In this way, as a method of searching for trademarks similar in appearance, it is possible to realize a search that considers similarity based on the classification of figures in addition to similarity based on the shape of figures.

(iii) The output unit may determine the similarity of each reference image to the search question image based on a comprehensive similarity obtained by complementary weighting shape similarity and classification similarity for each reference image.
In this way, using the total similarity, it is possible to realize a search that takes into account the similarity based on the classification of the figure in addition to the similarity based on the shape of the figure.

(iv) at least one of a verification unit that verifies whether the local features used for matching by the shape matching unit are reliable features, and one or more registered trademark images output by the output unit in response to the search query image; as a new search question, a pseudo-match feedback unit that obtains a similar image for the new search question, and the output unit performs pseudo-match feedback processing on the registered trademark image obtained as an output for the search question. The final output may be obtained by adding the obtained registered trademark image.
In this way, the verification unit eliminates unreliable features, and the pseudo-match feedback reduces retrieval omissions for similar images, thereby further enhancing retrieval accuracy.
The above embodiment and the verification process shown in FIG. 5 correspond to the function of the verification unit. Similarly, the pseudo-adaptive feedback process corresponds to the function of the pseudo-adaptive feedback section described above.

(v) The classification imparting unit includes a machine learning model constructed to output classification information for an image input by learning each reference image and the classification information assigned thereto as teacher data. may be
In this way, classification information can be mechanically assigned to any image to which no classification information has yet been assigned by learning reference images to which classification information has been assigned. Then, using the machine-given classification information, similar images can be found by considering the similarity based on the classification information in addition to the similarity based on the shape of the image.

(vi) A preferred aspect of the present invention is a process executed by a computer, comprising query feature extraction processing for extracting one or more shape features from a search question image, and referring to a plurality of reference images from which one or more shape features have been extracted, shape matching processing for matching the search question image with the shape features, and shape similarity of each reference image to the search question image based on the matching. A shape similarity calculation process for calculating the degree of similarity, a classification assignment process for estimating and assigning one or more classification information to the search question image based on the classification information assigned to each reference image, and a classification assigned to the search question image Classification similarity calculation processing for calculating the classification similarity of each reference image to a search query image based on the degree of matching between information and classification information assigned to each reference image; output processing for outputting one or more reference images similar to the query image based on the similar image retrieval method.

(vii) Alternatively, a preferred aspect of the present invention is a program for causing a computer to execute processing, which is a program for causing a computer to execute processing, wherein query features for extracting one or more shape features from a query image an extraction process; a shape matching process for matching a search question image with a shape feature by referring to a plurality of reference images to which one or more classification information has been assigned in advance and from which one or more shape features have been extracted; A shape similarity calculation process for calculating the shape similarity of each reference image with respect to the search question image based on and estimating and assigning one or more classification information to the search question image based on the classification information assigned to each reference image Classification assignment processing and classification similarity calculation for calculating the classification similarity of each reference image to the query image based on the degree of matching between the classification information assigned to the query image and the classification information assigned to each reference image and an output process of outputting one or more reference images similar to the query image based on the shape similarity and the classification similarity.
Preferred aspects of the present invention include combinations of any of the aspects described above.
Various modifications of the present invention are possible in addition to the above-described embodiments. Those modifications should not be construed as not belonging to the scope of the present invention. The present invention should include all modifications within the scope of the claims, the meaning of equivalents, and the scope.

11: Registration process 13: Registered trademark 15: Feature extraction process 17: Database 21: Search process 23: Search question 25: Query feature extraction process 27: Shape matching process 29: Verification process 31: Shape similarity calculation process 33: Vienna classification code assignment process 35: Classification similarity calculation process 37: Output process 39: Pseudo-match feedback process 41: Final output

Claims (6)

a query feature extraction unit that extracts one or more local features from a search question image that is a trademark image;
Search by referring to registered trademarks as reference images stored in a database with one or more Vienna classification codes assigned in advance as classification information relating to the shape features of the graphic elements of the image and one or more local features extracted. a shape matching unit that matches the query image and local features;
A shape similarity calculation unit that calculates the shape similarity of each reference image with respect to the search query image based on the collation;
a classification assigning unit that estimates and assigns one or more Wien classification codes as classification information for each search query image based on the classification information assigned to each reference image;
a classification similarity calculation unit that calculates the classification similarity of each reference image to the search question image based on the degree of matching between the classification information assigned to the search question image and the classification information assigned to each reference image;
an output unit that outputs one or more reference images similar to the search query image based on the shape similarity and the classification similarity;
with
The classification similarity is calculated by the following formula
However, a: Classification similarity, X: a set of Vienna classification codes assigned to trademark images by the classification assigning unit, Y: a set of Vienna classification codes assigned to each registered trademark stored in the database

calculated by
The output unit uses the shape similarity s calculated by the shape similarity calculation unit and the classification similarity a, the following formula
where m is a constant indicating the weight of shape similarity s and classification similarity a

A similar image search device that calculates a total similarity s' with each reference image by and outputs a predetermined number of reference images in descending order of the calculated total similarity s'. The shape matching unit compares each local feature related to the query image with each local feature related to each reference image, and determines the similarity of each reference image to the query image with respect to the matched shape features. Item 2. The similar image search device according to Item 1. a verification unit that verifies whether the local feature used for matching by the shape matching unit is a reliable feature;
Among the predetermined number of reference images output by the output unit for the search question image, a predetermined number of reference images in descending order of overall similarity s' are used as a new search question, and a similar image for the new search question is obtained. and a pseudo adaptive feedback unit for obtaining
The output unit adds the registered trademark image obtained by the pseudo-matching feedback process to the reference image as an initial output obtained as an output for the initial search question, which is the search question, as a final output, and a predetermined number of images that are the final output The following formula is used for the total similarity when outputting the reference image of
However, s' ₀ is the total similarity s' when using the initial search question, s' _k is the total similarity s' calculated using the initial output of rank k as the search question, and N is the pseudo-matching feedback process. number of initial outputs applied to

3. The similar image retrieval device according to claim 2, wherein The classification imparting unit includes a machine learning model configured to output classification information for an image input by learning each reference image and the classification information imparted thereto as teacher data. Item 4. The similar image search device according to any one of items 1 to 3. A process executed by a computer,
a query feature extraction process for extracting one or more local features from a search question image that is a trademark image;
Search by referring to registered trademarks as reference images stored in a database with one or more Vienna classification codes assigned in advance as classification information relating to the shape features of the graphic elements of the image and one or more local features extracted. Shape matching processing for matching the query image and local features,
A shape similarity calculation process for calculating the shape similarity of each reference image with respect to the search question image based on the collation;
a classification assignment process for estimating and assigning one or more Wien classification codes as classification information for each search query image based on the classification information assigned to each reference image;
Classification similarity calculation processing for calculating the classification similarity of each reference image to a search question image based on the degree of matching between the classification information given to the search question image and the classification information given to each reference image;
an output process of outputting one or more reference images similar to the search query image based on the shape similarity and the classification similarity;
with
The classification similarity is calculated by the following formula
However, a: Classification similarity, X: a set of Vienna classification codes assigned to trademark images by the classification process, and Y: a set of Vienna classification codes assigned to each registered trademark stored in the database.

calculated by
The output process uses the shape similarity s calculated by the shape similarity calculation process and the classification similarity a, the following formula
where m is a constant indicating the weight of shape similarity s and classification similarity a

A similar image retrieval method for calculating a total similarity s' with each reference image by and outputting a predetermined number of reference images in descending order of the calculated total similarity s'. A program that causes a computer to perform processing,
a query feature extraction process for extracting one or more local features from a search question image that is a trademark image;
Search by referring to registered trademarks as reference images stored in a database with one or more Vienna classification codes assigned in advance as classification information relating to the shape features of the graphic elements of the image and one or more local features extracted. Shape matching processing for matching the query image and local features,
A shape similarity calculation process for calculating the shape similarity of each reference image with respect to the search question image based on the collation;
a classification assignment process for estimating and assigning one or more Wien classification codes as classification information for each search query image based on the classification information assigned to each reference image;
Classification similarity calculation processing for calculating the classification similarity of each reference image to a search question image based on the degree of matching between the classification information given to the search question image and the classification information given to each reference image;
an output process of outputting one or more reference images similar to the search query image based on the shape similarity and the classification similarity;
with
The classification similarity is calculated by the following formula
However, a: Classification similarity, X: a set of Vienna classification codes assigned to trademark images by the classification process, Y: a set of Vienna classification codes assigned to each registered trademark stored in the database

calculated by
The output process uses the shape similarity s calculated by the shape similarity calculation process and the classification similarity a, the following formula
where m is a constant indicating the weight of shape similarity s and classification similarity a

A similar image search program for calculating a total similarity s' with each reference image, and outputting a predetermined number of reference images in descending order of the calculated total similarity s'.