JP7824601B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device related to machine learning (in particular, an information processing device for improving explainability related to machine learning) and related technologies.

In recent years, inference processing technologies using machine learning such as deep learning have evolved rapidly.

However, due to factors such as the extremely complex nature of learning models in machine learning, there is a problem in that it is not always clear (it is not easy to explain) what basis the inference results obtained by the learning model are based on.

In particular, in situations where the inference results have a significant impact, there is a demand for improved explainability of the basis for judgments. For example, the technology described in Patent Document 1 is one technology that meets this demand.

Japanese Patent Application Laid-Open No. 2001-33376

In the aforementioned Patent Document 1, the region of interest within the image area (the region of interest in the inference process using the learning model) is visualized, and this region of interest is recognized as the region used in the inference. In other words, it is possible to understand which region within the image was focused on to determine similarity.

However, with the technology of Patent Document 1, even if it is possible to identify the region of interest (position of interest) within an image during inference processing, it is difficult to understand what concept similarities are influencing the inference results. In other words, it is difficult to explain the basis for judgments made during inference processing on a concept basis.

The objective of this invention is to provide technology that can explain the basis for image similarity on a concept-based basis.

In order to solve the above problem, the information processing device of the present invention is characterized by comprising a control unit that acquires multiple feature vectors output from a machine-learned learning model corresponding to each input of multiple input images to the learning model, generates multiple hierarchical clusters by performing a hierarchical clustering process on the multiple feature vectors, and extracts a subspace or vector corresponding to a specific cluster from the multiple clusters as a concept of the specific cluster.

The control unit may extract a representative vector related to the specific cluster as a concept of the specific cluster.

The control unit may generate a concept visualization image, which is a virtual input image corresponding to the representative vector and which visualizes the concept of the specific cluster, based on the representative vector and the learning model, and display the concept visualization image on the display unit.

The information processing device may further include a reception unit that receives input of attribute information for the specific cluster.

When the control unit determines the similarity between the first image and the second image based on a first feature vector output from the learning model in response to a first image input to the learning model and a second feature vector output from the learning model in response to a second image input to the learning model, the control unit may calculate the contribution of at least one concept among a plurality of concepts corresponding to the plurality of clusters to the determination of the similarity between the first image and the second image.

When determining the similarity between the first image and the second image based on a first feature vector output from the learning model in response to a first image being input to the learning model and a second feature vector output from the learning model in response to a second image being input to the learning model, the control unit may extract, from among a plurality of subspaces corresponding to the plurality of clusters, a subspace that relatively reduces the distance between the projection vectors of the first and second feature vectors onto the orthogonal complementary space, or a vector spanning the subspace, as a concept that serves as the basis for determining that the first image and the second image are not similar to each other.

The information processing device according to the present invention is also characterized by comprising a control unit that acquires a plurality of feature vectors output from a machine-learned learning model corresponding to the input of a plurality of input images to the learning model, generates a plurality of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors, and determines two or more input images corresponding to a specific cluster among the plurality of clusters as a group of images representing the concept of the specific cluster.

In addition, the information processing method of the present invention is characterized by comprising: a) a step of acquiring a plurality of feature vectors output from a machine-learned learning model corresponding to each input of a plurality of input images to the learning model; b) a step of generating a plurality of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors; and c) a step of extracting a subspace or vector corresponding to a specific cluster from the plurality of clusters as a concept of the specific cluster.

This invention makes it possible to explain the basis of image similarity on a concept-based basis.

FIG. 1 is a schematic diagram illustrating an image processing system. FIG. 1 is a conceptual diagram illustrating an overview of processing in an image processing device. 10 is a flowchart showing processing in the image processing device. 10 is a flowchart illustrating a concept analysis process. 10 is a flowchart illustrating a process for explaining the basis for similarity determination. 10 is a flowchart illustrating a process for explaining the basis for determining dissimilarity. FIG. 10 is a conceptual diagram illustrating a learning process in a first phase. FIG. 10 is a diagram showing a feature space etc. in a state where learning has progressed. FIG. 10 is a diagram illustrating the inference process in the second phase. FIG. 10 is a diagram illustrating an example of an inference processing result. FIG. 10 is a diagram showing a dendroid (tree diagram) and the like related to the results of the hierarchical clustering process. FIG. 10 is a diagram showing the hierarchical relationship around a specific cluster. FIG. 2 is a diagram showing input images and the like that constitute a cluster. The feature vectors are shown mapped onto the hypersphere. FIG. 10 is a diagram showing the correspondence between two-dimensional and three-dimensional representations of feature spaces. FIG. 10 is a diagram showing detailed learning results (distribution on a hypersphere). FIG. 10 is a diagram showing a separating plane etc. generated by a linear separator. FIG. 10 is a diagram showing the concept vectors of each cluster. FIG. 10 is a diagram showing the concept vectors of each cluster. FIG. 10 is a diagram showing the concept vectors of each cluster. FIG. 10 is a conceptual diagram for explaining the similarity between two input images. FIG. 10 is a diagram illustrating how a feature vector is projected onto a specific plane (subspace). FIG. 10 is a diagram illustrating how a feature vector is projected onto a specific line (subspace). FIG. 10 is a diagram showing how a feature vector is projected onto a specific line (when the concept vector faces in the same direction as the x-axis). 10A to 10C are diagrams showing analysis processing results in the third phase, etc.; FIG. 10 is a diagram showing a detailed explanation screen for a certain concept. FIG. 10 is a diagram showing a detailed explanation screen for another concept. FIG. 10 is a diagram illustrating an outline of a process for generating a concept visualization image. FIG. 10 is a conceptual diagram showing a process of obtaining a basis for determining that two images are not similar to each other (basis for determining dissimilarity).

Embodiments of the present invention will be described below with reference to the drawings.

1. First embodiment
<1-1. System Overview>
Fig. 1 is a schematic diagram showing an image processing system 1. As shown in Fig. 1, the image processing system 1 includes a plurality (a large number) of image capturing devices (such as surveillance cameras) 20 that capture images, and an image processing device 30 that processes the captured images. The image processing device 30 is a device that executes various processes for identifying or classifying subjects (target people in this case) in the captured images. The image processing device 30 can also be expressed as an information processing device that processes various types of information.

Images captured by each image capture device 20 are input to the image processing device 30 via a communications network (such as a LAN and/or the Internet). The image processing device 30 then performs image processing to identify or classify people and other subjects in the captured images. In more detail, processing is performed to identify (recognize) a specific person from among multiple people captured in multiple captured images.

For example, a process is performed to search for images capturing a specific person (images containing a specific person as a subject) from among multiple images captured by multiple image capture devices 20 located within a specified area. The multiple image capture devices 20 are distributed across multiple different locations (such as different locations along a road, multiple different stores (locations within each store), and/or multiple different locations within the same store (especially a large store)). The image processing device 30 then searches for the specific person to be searched for among the multiple captured images and identifies the image capture devices corresponding to one or more of the retrieved images, thereby determining the specific person's behavior (path of travel, etc.) within the specified area. In short, the image processing device 30 is capable of tracking a specific person. Examples of specific persons include a lost child in a lost child tracking process, or a criminal (suspect) in a criminal tracking process. For example, consider a case where the specific person (search target person) is included (photographed) in a total of three images, including images captured by a certain image capture device 20A, another image capture device 20B, and yet another image capture device 20C. In this case, the image processing device 30 can determine that the specific person was present in the three locations corresponding to the image capture devices 20A, 20B, and 20C. The image processing device 30 can also determine the order in which the specific person moved between the three locations based on the capture time of each image (more specifically, the order in which the images were captured at the three locations).

This type of inference processing, or the processing of identifying the same person based on multiple images captured by multiple image capture devices 20, is also called person re-identification processing. Note that the tracking processing of a specific person related to person re-identification is not limited to criminal tracking processing for tracking criminals or lost child tracking processing for searching for (tracking) a lost child, but may also be, for example, tracking processing for tracking the behavior of individuals for use in marketing, etc.

Figure 2 is a conceptual diagram showing an overview of the processing in the image processing device 30, and Figure 3 is a flowchart showing the processing in the image processing device 30.

In this embodiment, as shown in Figures 2 and 3, the image processing device 30 first executes machine learning processing (processing for machine learning the learning model 400) in the first phase PH1 (Figure 2) to perform the above-mentioned inference processing (step S11 (Figure 3)). Specifically, metric learning (also known as distance learning) is executed as this machine learning processing. More specifically, deep metric learning using a deep neural network (particularly a convolutional neural network) is used. This metric learning uses a learning model 400 that outputs a feature vector 250 (251) (see Figure 7) in a feature space (feature space) in response to an input image 210 (211). This learning model 400 can also be expressed as a model that shows the transformation (mapping) from the input image (input) to the feature vector (output). The processing in the first phase PH1 generates a machine-learned learning model 400 (trained learning model) (420).

Next, the image processing device 30 executes inference processing as processing of the second phase PH2 (FIG. 2) (step S12). Specifically, the inference processing is performed by utilizing the learning model (trained model) 400 (420) learned in the first phase PH1. Specifically, the inference processing includes processing such as searching for images containing a specific person from among multiple captured images 213 (also referred to as gallery images) captured within a specific area. More specifically, the inference processing includes processing such as searching for images that have a predetermined degree of similarity or higher with a search source image 215 (also referred to as a query image), which is an image of the specific person (in other words, the distance between the feature vectors in feature space is a predetermined distance or less), as images of the same person. Alternatively, the inference processing (person re-identification processing) may include processing such as searching for images similar to the query image 215 in order of similarity. The multiple gallery images 213 are also referred to as a group of images constituting a search range (search range image group).

Furthermore, the image processing device 30 executes a process (such as a process for generating explanatory information) to explain, on a concept basis, the basis for determining the similarity between two images (input images), the query image 215 and the image 213 found in the above-mentioned inference process, as a process in the third phase PH3 (Figure 2) (step S13).

Specifically, before deriving the basis for determining the similarity between the two images, the image processing device 30 first analyzes what concepts have been acquired (learned) in the machine-learned learning model 400 (also referred to as the trained model 420) (see Figure 4). Note that Figure 4 is a flowchart showing this process (concept analysis process).

Next, the image processing device 30 derives the basis for determining the similarity between the two images. In detail, it executes a process to generate information (explanatory information) for explaining the basis for determining the similarity between the two images (the basis for determining that they are similar to each other) on a concept basis (see Figure 5). Figure 5 is a flowchart showing this process (processing for explaining the basis for determining similarity).

More specifically, of the various concepts acquired by the learning model 400, the concept that has a particularly large impact on the similarity between the two images (the concept with the greatest contribution) is extracted as the main concept. For example, of multiple concepts, the top few concepts (in order of highest contribution, etc.) are extracted as the main concepts. This concept is then determined as the basis for judging the similarity between the two images, and various images representing this concept are displayed on the display unit 35b (presented to the user).

This type of processing (such as that shown in Figures 4 and 5) will be described in more detail later.

<1-2. Image processing device 30>
Referring again to Fig. 1, the image processing device 30 includes a controller 31 (also referred to as a control unit), a storage unit 32, a communication unit 34, and an operation unit 35.

The controller 31 is a control device built into the image processing device 30 that controls the operation of the image processing device 30.

The controller 31 is configured as a computer system equipped with one or more hardware processors (e.g., a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit)). The controller 31 performs various processes by executing, in the CPU or the like, a predetermined software program (hereinafter simply referred to as a program) stored in a storage unit 32 (a non-volatile storage unit such as a ROM and/or a hard disk). The program (more specifically, a group of program modules) may be recorded on a portable storage medium such as a USB memory, read from the storage medium, and installed on the image processing device 30. Alternatively, the program may be downloaded via a communication network or the like and installed on the image processing device 30.

Specifically, the controller 31 executes the learning process in the first phase PH1, the inference process in the second phase PH2, and the explanation process (such as the explanation information generation process) in the third phase PH3.

The memory unit 32 is composed of a storage device such as a hard disk drive (HDD) and/or a solid state drive (SSD). The memory unit 32 stores the learning model 400 (including learning parameters and programs related to the learning model) (and thus the trained model 420), etc.

The communication unit 34 is capable of performing network communication via a network. This network communication uses various protocols, such as TCP/IP (Transmission Control Protocol/Internet Protocol). By using this network communication, the image processing device 30 can exchange various data (captured image data, correct answer data, etc.) with a desired destination (for example, the imaging device 20 or an information storage device (not shown)).

The operation unit 35 includes an operation input unit 35a that accepts operation inputs to the image processing device 30, and a display unit 35b that displays and outputs various information. A mouse and keyboard are used as the operation input unit 35a, and a display (such as a liquid crystal display) is used as the display unit 35b. A touch panel that functions as both part of the operation input unit 35a and part of the display unit 35b may also be provided.

The image processing device 30 is also referred to as a learning model generation device because it has the function of machine learning the learning model 400 using training data. The image processing device 30 is also referred to as an inference device because it is a device that performs inference regarding object identification and/or classification using the learning model 400 (420). The image processing device 30 is also referred to as an explanatory information generation device because it is a device that generates explanatory information regarding similarity. The image processing device 30 is also referred to as a concept extraction device because it is a device that extracts concepts acquired by the learning model 400 (420), and as a device that explains (analyzes) the basis for the similarity between two images based on those concepts, so it is also referred to as a (similarity) analysis device.

Furthermore, while various processes (functions) are implemented here by a single image processing device 30, this is not limiting. For example, various processes may be shared and implemented by multiple devices. For example, the learning process in the first phase PH1, the inference process in the second phase PH2, and the explanation process (explanation information generation process, etc.) in the third phase PH3 described above may each be executed by separate devices.

<1-3. Processing in the learning stage (first phase PH1)>
As shown in FIG. 3, in this embodiment, a learning process (step S11) in the first phase PH1, an inference process (step S12) in the second phase PH2, and an explanation process (step S13) in the third phase PH3 are executed in this order.

Below, we will first explain the learning process (step S11) in the first phase PH1 (see Figures 2 and 3).

Figure 7 is a conceptual diagram showing the learning process in the first phase PH1.

As shown in FIG. 7, in the first phase PH1 (step S11), machine learning processing is performed on the learning model 400 (more specifically, the learning model 410 before learning (FIG. 2)) using metric learning (distance learning). Specifically, multiple input images 210 (211) in multiple teacher data sets (teacher data group) with correct answer labels are sequentially input to the learning model 400, and an output group (multiple feature vectors 250 (251)) from the learning model 400 is obtained (see FIG. 7). Then, a mapping relationship between the input image 210 (input) and the feature vector 250 (output) in the feature space is learned. More specifically, the learning model 400 (mapping relationship) is learned so that the distance in the feature space (the distance between feature vectors) reflects the similarity of the input images in the input space. For example, a learning process is performed to minimize (optimize) an evaluation function such as triplet loss. Through this learning process, the pre-learning learning model 400 (410) is trained and a trained model 420 is generated (step S11).

More specifically, the image processing device 30 first generates a person image 210 (also referred to as 211) for machine learning. For example, the image processing device 30 performs a person extraction process and a size adjustment process (resizing process) on each of a plurality of captured images acquired from the image capture device 20 to generate a plurality of person images 210 (211). The plurality of person images 210 are prepared as a group of input images for the learning model 400. In other words, each person image 210 (211) is prepared as an input image 210 (211) for the learning model 400. For example, a color image (3 channels) having a pixel array (rectangular pixel array) with a width (horizontal) of W0 pixels and a height (vertical) of H0 pixels is prepared as each input image 210. In other words, the input image 210 is generated as voxel data of W0 x H0 x CH0 (where CH0 = 3).

In addition, correct answer information (correct answer label) regarding whether the people in the multiple input images 210 are the same person or different people is assigned to each of the multiple input images 210. For example, a person ID (an identifier that identifies a person) is assigned to each input image 210 (211). In detail, the same person ID is assigned to images of the same person, and different person IDs are assigned to images of different people. In this way, the combination of the correct answer label and the input image 210 (211) is assigned as training data with a correct answer label.

Next, the multiple input images 210 (input image group) are sequentially input to the learning model 400, and multiple outputs from the learning model 400, i.e., multiple feature vectors 250 (feature vector group) in feature space, are sequentially output (see Figure 7).

Here, the learning model 400 has a hierarchical structure in which multiple layers (hierarchies) are hierarchically connected. Specifically, the learning model 400 includes an input layer, multiple intermediate layers, and an output layer. The multiple intermediate layers include a feature extraction layer, etc. The feature extraction layer is configured by repeatedly arranging one or more convolutional layers and one pooling layer, etc. In each convolutional layer, features within the image are extracted using a filter that performs convolution processing. In addition, in each pooling layer, a pooling process (such as average pooling or max pooling) is performed to extract the average pixel value or maximum pixel value for each small pixel range (e.g., a 2x2 pixel range), thereby reducing the pixel size (e.g., by half in both the vertical and horizontal directions) (condensing the amount of information). A feature map 230 (not shown) is generated by performing multiple feature extraction processes on the input image 210. Furthermore, by performing a pooling process (for example, a max pooling process) on each channel image of the feature map 230, a feature vector 250 having a predetermined number of channels (number of dimensions) CH1 is generated, and the feature vector 250 is output from the learning model 400.

Note that such a learning model 400 (neural network) may be, for example, VGG16 or ResNet (Residual Network). VGG16 is a convolutional neural network model with three convolutional layers, five pooling layers, and three fully connected layers. ResNet (Residual Network) is a convolutional neural network that includes summing residuals between layers. The feature extraction layer in ResNet is composed of multiple residual blocks, which are made up of combinations of convolutional layers, activation functions, and skip connections (shortcut connections).

Various image features in the input image 210 are extracted for each channel in the feature map 230 (in other words, for each channel (element) of the feature vector 250). Note that the image features in the input image 210 are extracted while maintaining their approximate positions within the two-dimensional image of each channel in the feature map 230.

For example, feature map 230 is three-dimensional array data (voxel data of W1 x H1 x CH1) with one channel CH, each channel consisting of two-dimensional array data of pixels with a width of W1 pixels and a height of H1 pixels (rectangular pixel array). The size (W1 x H1) of each channel of feature map 230 is, for example, 14 x 14. Each element (value) of feature vector 250 represents the feature extracted in each channel. The number of dimensions CH1 of feature vector 250 is the number of channels CH1 of feature map 230, and is, for example, 1024. However, this is not limited to this, and the size (W1 x H1) of each channel and the number of channels CH1 may be other values. For example, the number of channels CH1 (the number of dimensions of feature vector 250) may be 512 or 2048.

Ideally, in the feature space (the output space of the learning model 400), multiple feature vectors 250 (251) corresponding to multiple input images 210 (211) featuring the same person as a subject would be located close to each other, and multiple feature vectors 250 corresponding to multiple input images of different people would be located far from each other. However, the distribution of feature vectors based on the output from the learning model 400 before learning (see the rightmost column in Figure 7) deviates from this ideal distribution.

Next, in metric learning, the learning model 400 is trained to optimize (minimize) an evaluation function such as triplet loss. This trains the learning model 400 (mapping relationship) so that the similarity of input images in the input space corresponds to the distance in the feature space (the distance between feature vectors). In other words, the distribution of feature vectors in the feature space gradually changes as the learning progresses. If very good machine learning is performed, the distribution of feature vectors in the feature space gradually approaches the ideal distribution described above (see the rightmost column in Figure 8). Specifically, in the final feature space, corresponding feature vectors of images of the same person (or people wearing similar clothing) will be relatively close to each other, while corresponding feature vectors of images of different people (or people wearing significantly different clothing) will be relatively far apart. As a result of this machine learning, the pre-learning learning model 400 (also referred to as 410) transforms into the trained learning model 400 (also referred to as 420). The trained model 420 can also be described as a feature extractor that extracts features (feature vectors) according to the input image.

Note that the rightmost columns in Figures 7 and 8 show how multiple feature vectors 250 (251) are mapped onto the feature space. In the rightmost columns, each feature vector 251 is represented as a single point (more specifically, a point-like figure). More specifically, some of the multiple feature vectors 251 corresponding to the multiple input images 211 are represented as point-like figures (white circles, black circles, hatched white circles, hatched black circles, white squares, black squares, etc.). In Figures 7 and 8, the rightmost column (the large rectangular portion representing the feature space) and the left portion of the rightmost column (the portion where multiple feature vectors 251 (shown as elongated strip-like rectangles) are arranged) show the same situation. For convenience, multiple points that actually correspond to the same person (and people wearing very similar clothing) are represented by the same figure (point-like figure). However, the image processing device 30 does not know which points (feature vectors (in other words, input images)) actually correspond to the same person (correct labels).

<1-4. Processing in the inference stage (second phase PH2)>
Next, the inference processing in the second phase PH2 (step S12) (see FIGS. 2 and 3) will be described with reference to FIGS. 9 and 10. FIG. 9 is a diagram illustrating the inference processing using the feature vector 250 (253). FIG. 10 is a diagram showing an example of the inference processing result.

In the second phase PH2 (step S12), the image processing device 30 performs an inference process to identify (or classify) objects (here, target persons) in multiple person images in the search range (specifically, multiple new input images 210 (213)). Specifically, a person identical to a person in the search target (search source) input image 215 (new input image) is searched for among multiple new input images 210 (213) captured in the target area (by a camera device 20 placed in the target area). In other words, the image processing device 30 identifies (recognizes) from among the people in the multiple input images 213, a person identical to a person in the search target input image 215 (query image).

To this end, the image processing device 30 first inputs multiple person images within the search range (specifically, multiple new input images 210 (gallery images 213)) into the learning model 420 and obtains output from the learning model 420. Specifically, as shown in FIG. 9, a feature vector 250 (253) is obtained as output for each input image 213. Furthermore, each feature vector 250 (253) is generated as a 1024-dimensional vector, for example. Such a feature vector 253 is obtained for each of the multiple input images 213 as a vector representing the features of each input image 213 (see the left side of FIG. 9).

Similarly, the image processing device 30 inputs the input image (query image) 215 to be searched into the learning model 420 and obtains the feature vector 250 (255) output from the learning model 420 (see the right side of Figure 9). Note that the query image 215 is, for example, an image (such as an image newly assigned for search purposes) that is different from the multiple input images 213 (gallery images). However, without being limited to this, the query image 215 may also be an image related to the person to be searched that has been discovered (identified) from the multiple input images 213 (gallery images) due to some trigger or the like.

Next, the image processing device 30 calculates the degree of similarity (for example, Euclidean distance or the dot product between vectors (cosine similarity)) between the feature vector 255 of the query image 215 and each of the multiple feature vectors 253 for the multiple input images 213. The multiple feature vectors 253 are then sorted in descending order of similarity (descending order of similarity). More specifically, the multiple feature vectors 253 are sorted in ascending order of Euclidean distance (or descending order of cosine similarity).

For example, the image processing device 30 identifies one or more feature vectors 253 whose distance to the feature vector 255 in the feature space is less than a predetermined distance (i.e., the degree of similarity is greater than or equal to a predetermined level) as the feature vector 255 of the same person as the person in the query image 215. In other words, the image processing device 30 recognizes the person in one or more input images 213 corresponding to the identified one or more feature vectors 255 as the same person as the person in the query image 215.

Figure 10 shows how multiple feature vectors 253 (represented by open circles with sand-hatched lines in Figure 10) corresponding to multiple input images 213 are distributed in feature space. In Figure 10, three feature vectors 253 (V301, V302, V303) exist within a predetermined distance range from the feature vector 255 (see the open star) of the query image 215. In this case, for example, the three images 213 corresponding to the three feature vectors 253 (V301, V302, V303) are extracted as images of the same person. The three feature vectors 253 are also sorted in descending order of similarity to the feature vector 255 (ascending order of distance). Here, the three person images 213 corresponding to the top three feature vectors 253 are recognized as images of the same person (or a person very similar) as the person in the query image 215.

However, without being limited to this, the person in the input image 213 corresponding to a predetermined number of feature vectors 250 (253) sorted in ascending order of distance may be recognized as the same person as the person in the query image 215. Alternatively, multiple input images 213 may simply be sorted in ascending order of distance (related to feature vector 255) from the query image 215 (descending order of similarity). Even in this case, the image processing device 30 essentially executes a process of searching for people who are likely to be the same person as the person in the query image in order of likelihood (same person recognition process), and this process is one type of inference process for recognizing the target person in the query image.

Here, each of the multiple feature vectors 250 (251, 253, 255) is assumed to be normalized (specifically, L2 normalized). The dot product (in other words, cosine similarity) between two feature vectors 250 (also denoted as F) is used as an index indicating the similarity between any two of the multiple feature vectors 250. Specifically, the dot product (= Fq·Fg = q·g) of two feature vectors F ( ^Fq , ^Fg ) corresponding to two input images X (query image ^Xq and gallery image Xg) is used as the similarity between the two input images. The feature vector ^{Fq is} an output vector (feature vector in feature space (output space)) F from the learning model 400 for the input image (query image) ^Xq , and the feature vector ^Fg is an output vector F from the learning model 400 for the input image (a certain gallery image) ^Xg . The symbol "·" represents the dot product. Furthermore, the feature vector ^Fq is also simply expressed as the feature vector q, and the feature vector ^Fg is also simply expressed as the feature vector g. Such a similarity St is expressed as in the following equation (1).

Note that because each feature vector F (250) is normalized (the magnitude of each vector F is 1), the dot product between two feature vectors is equal to the cosine similarity between the two feature vectors. Furthermore, a large cosine similarity (and dot product) (close to 1) means that the angle θ between the two feature vectors F is small (the two feature vectors F are similar), and therefore the two input images corresponding to the two feature vectors F are similar. In other words, the greater the similarity St (the dot product of the two feature vectors F) between the two feature vectors F, the more similar the two input images corresponding to the two feature vectors F are.

<1-5. Distribution of feature vector F>
In FIGS. 7 and 8 (in the rightmost columns), the distribution of feature vectors F is expressed two-dimensionally as a group of points on a plane (hyperplane).

On the other hand, as shown in Figure 14, etc., the distribution of feature vectors F can also be expressed three-dimensionally as a group of points on a sphere (hypersphere). In the following, we will mainly use the latter expression (three-dimensional expression using a hypersphere) for explanation.

Figure 14 shows the state in which multiple (here, three) feature vectors F (250) output from the trained model 420 are mapped onto the hypersphere. Here, each feature vector F (250) is normalized (the norm (magnitude) of each vector F is 1). Therefore, as shown in Figure 14, the feature vector F can be represented as a vector starting from the origin and ending at a point on the hypersphere.

Also, as mentioned above, a small angle θ between two feature vectors F (a large dot product of the two feature vectors F) means that the two input images corresponding to the two feature vectors F are similar.

Figure 15 shows the correspondence between two-dimensional and three-dimensional representations of feature space. Of the four groups of figures in Figure 15, roughly divided into left, right, top, and bottom, the two groups of figures at the top of Figure 15 show how each feature vector F(250) is distributed on a hyperplane. In contrast, the two groups of figures at the bottom of Figure 15 show how each feature vector F(250) is distributed on a hypersphere.

In detail, the group of figures in the lower left of Figure 15 (spheres and point figures on their surfaces, etc.) show the distribution of output vectors (feature vectors F) from the "untrained learning model 400" in feature space, similar to the right-most column of Figure 7. The group of figures in the lower left illustrates the feature space in three dimensions, and shows a similar situation to the group of figures in the upper left of Figure 15 (large rectangles and figures inside them) (although in three dimensions rather than two dimensions).

Furthermore, the group of figures (spheres and point-like figures on their surfaces, etc.) in the lower right of Fig. 15 show the distribution of output vectors (feature vectors F) in the feature space by the "trained learning model 400 (420)," similar to the rightmost column of Fig. 8. The group of figures in the lower right illustrates the feature space three-dimensionally, and shows a situation similar to that of the group of figures (large rectangles and figures inside them) in the upper right of Fig. 15 (however, three-dimensionally, not two-dimensionally).

Note that, for convenience, the upper part of Figure 15 and Figure 8 etc. are conceptual diagrams that represent the distribution of feature vectors F in two dimensions, while the lower part of Figure 15 and Figure 14 etc. are conceptual diagrams that represent the distribution of feature vectors F in three dimensions. Actual feature vectors F are not usually two- or three-dimensional vectors, but rather very high-dimensional (even higher) multidimensional vectors (for example, 1024-dimensional vectors). Vectors with four or more dimensions are difficult to visually illustrate in three-dimensional space, and these graphical representations of feature vectors F (see Figures 8, 14, 15 etc.) are all simplified virtual representations.

<1-6. Details of the first and second phase processes>
In this embodiment (particularly the first phase PH1 and the second phase PH2 described above), more specifically, the following person re-identification (re-identification) process is executed.

First, in the first phase PH1 (step S11), the learning model 400 is machine-learned based on a large number of images (e.g., thousands to hundreds of thousands) each containing a large number of people (e.g., hundreds to tens of thousands). It is assumed that the people in the images are wearing different clothing, and that no two people are wearing exactly the same clothing. Learning is performed so that feature vectors corresponding to images of people wearing similar clothing are positioned relatively closer to each other in the feature space (the output space of the learning model 400) (than feature vectors corresponding to images of people wearing different clothing).

In more detail, feature vectors corresponding to images of the same person (the same person wearing the same clothes) are trained to be positioned very close together in feature space. In other words, multiple images of the same person with certain clothing characteristics are trained to be positioned very close together in feature space. That is, images of the same person (more specifically, their feature vectors) are positioned very close together in feature space. Furthermore, images (more specifically, their feature vectors) of different people wearing similar clothing (different people with similar clothing characteristics) are also positioned close together in feature space. On the other hand, images (more specifically, their feature vectors) of different people wearing very different clothing are positioned relatively far apart in feature space.

An example of such a learning result is shown in the rightmost column of Figure 8. In detail, each output vector (feature vector F) from the trained model 420 is represented by a dot-like figure, and multiple dot-like figures representing multiple feature vectors F are plotted (arranged) within (a large rectangle representing) the feature space.

Furthermore, Figures 11 and 16 show an example of a more detailed learning result (than the rightmost column of Figure 8). Within the large rectangle spanning from the center to the lower half of Figure 11, the distribution of output vectors (feature vectors F) from the trained learning model 400 (trained model 420) is shown in two dimensions. Furthermore, Figure 16 is a diagram that depicts a similar situation in three dimensions.

As described above, in feature space, multiple (e.g., two) feature vectors F corresponding to multiple (e.g., two) images of the same person are located very close (relatively close). For example, in Figures 11 and 16, two point-like figures (e.g., a pair of cross-hatched white circle figures) are located very close to each other. This indicates that two feature vectors F corresponding to two images of the same person (both images of the person wearing the same clothing) are located very close to each other. In other words, the pair of point-like figures corresponds to a pair of images of the same person wearing the same clothing.

Furthermore, images of different people wearing similar clothing (more specifically, their feature vectors F) are also positioned close to each other in feature space (although not as close as images of the same person). For example, in Figures 11 and 16, nine pairs of white circles (a total of 18 white circles) are positioned relatively close to each other. Of these nine pairs of white circles (both with and without hatching), six pairs of white circles (with hatching) are particularly positioned relatively close to each other.

On the other hand, images of people that are significantly different (specifically, their feature vectors F) are positioned relatively far apart in feature space. For example, a pair of black circles and a pair of white circles (no hatching) are positioned far apart.

In this way, feature vectors F relating to the same person are clustered in a relatively narrow range, while feature vectors F relating to different people wearing similar clothing are clustered in a slightly wider range. On the other hand, feature vectors F relating to very different people (different people wearing very different clothing) are clustered relatively far apart (with a relatively large dispersion).

Note that, for convenience of illustration, each figure shows only a portion of the multiple feature vectors F. In particular, in Figures 8, 15, and 17 onwards, even fewer feature vectors F are shown than in Figures 11 and 16, etc.

Next, in the second phase PH2 (step S12), the image processing device 30 uses the trained learning model 400 (420) to determine whether two people (the person in the query image 215 and the person in the gallery image 213) are the same person (the degree of similarity between the two people). It is assumed that the same person wears the same clothes, and people wearing similar clothing are searched for as the same person (more specifically, people who are likely to be the same person). The similarity between pairs of images of people wearing similar clothing is relatively high.

Specifically, for example, the combination of feature vectors F (and thus the corresponding image pair) that maximizes the similarity St shown in equation (1) is found. More specifically, the gallery image corresponding to the feature vector F that maximizes the similarity St with the feature vector F of the query image is extracted. In other words, two images whose feature vectors are similar to each other are extracted as images of the same person. For example, as shown in Figure 21 (or Figure 14), people wearing the same (or similar) clothing are presumed to be the same person and extracted.

Furthermore, in the next third phase PH3 (step S13), the image processing device 30 executes processes such as explaining the basis for the image processing device 30's determination that the individuals are the same (or similar). For example, it analyzes whether the basis for the determination that the individuals are similar is that they are wearing white shorts and/or that they are wearing patterned (checked, etc.) shirts. In addition, as a preliminary step, the trained model 420 executes analytical processes, etc., regarding what concepts it has learned based on the training data. These are described below.

<1-7. Explanation Stage (Third Phase PH3) Processing (Overview)>
In this embodiment, the image processing device 30 further executes the processing of the third phase PH3 (FIG. 2). The processing of the third phase PH3 is a processing (e.g., a processing for generating explanatory information) for explaining the basis for determining the similarity between two images (input images) on a concept basis. Here, the case where the similarity between the query image 215 and the image 213 found in the above inference processing is determined as the similarity between two images will be mainly described.

The processing of this third phase PH3 is broadly divided into processing of subphase PH3a (step S20 (Figure 4)) and processing of subphase PH3b (step S30 (Figure 5)).

In the former subphase PH3a (step S20 (Figure 4)), an analysis process is performed to analyze what concepts have been acquired (learned) in the machine-learned learning model 420. The former subphase PH3a is a preprocessing step for the latter subphase PH3b.

Specifically, in subphase PH3a, multiple feature vectors F are obtained in the feature space output from the learning model 400 in response to multiple input images (e.g., multiple input images 211 used in machine learning) input to the learning model 400. A hierarchical clustering process is then performed on the multiple feature vectors F to generate multiple hierarchical clusters G. Furthermore, a vector corresponding to a specific cluster among the multiple clusters (a representative vector for the specific cluster) is extracted as the concept of the specific cluster (more specifically, a vector representing the concept, etc.). For example, the concept activation vector CAV (described below) for the specific cluster is used as the representative vector for the specific cluster. As described below, a subspace corresponding to the specific cluster (e.g., a subspace spanned by the representative vectors) may also be extracted as the concept of the specific cluster.

On the other hand, in the latter sub-phase PH3b (step S30 (Figure 5)), processing is performed to derive the basis for determining the similarity between the two images. Specifically, the influence of various concepts (concepts obtained in sub-phase PH3a (Figure 4)) on the similarity between the two images is evaluated. More specifically, the contribution of each of the various concepts acquired by the learning model 400 through machine learning (multiple concepts extracted from the learning model 400) to the similarity between the two images is calculated. Then, based on the contribution, etc., a main concept from among these concepts is identified. Furthermore, a screen display explaining the main concept is performed.

Subphases PH3a and PH3b will be explained below in that order.

<1-8. Processing of sub-phase PH3a (step S20)>
First, the processing of sub-phase PH3a will be described with reference to FIG.

<Step S21>
As shown in Fig. 4, first, in step S21, the controller 31 (image processing device 30) acquires a plurality of feature vectors 251 (in feature space) output from the learning model 420 in response to a plurality of input images 210 (211) input to the machine-learned learning model 420. Here, the training data used in the machine learning of the trained model 420 (more specifically, the input images 211 constituting the training data) is used as the input images 210 to the trained model 420. The plurality of feature vectors 251 can also be expressed as feature vectors that are finally output in the final stage of the learning process for the learning model 400 (after the learning process involves multiple iterations).

Each feature vector 251 is a vector (a vector with dimension CH1, for example, a 1024-dimensional vector) output from the trained model 420 in response to the input of each input image 211 to the trained model 420. As described above, the multiple feature vectors 251 obtained in this manner are distributed to appropriate positions in feature space by the trained training model 420. As a result of distance learning (metric learning) for the training model 400, the distance between the multiple feature vectors 251 in feature space reflects the similarity of the corresponding input images in input space (see the bottom of Figure 11 and Figure 16, etc.). The distribution of the multiple feature vectors 251 in feature space can also be considered to be a distribution based on concepts acquired by training. Specifically, feature vectors corresponding to the same concept and feature vectors corresponding to similar concepts are considered to be distributed relatively close to each other in feature space.

<Step S22: Hierarchical clustering process>
Next, in step S22, the controller 31 generates a plurality of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors 251 based on the positional relationships in the feature space, etc. The hierarchical clustering process is also called a hierarchical clustering process or a hierarchical clustering process.

Hierarchical clustering is a process in which multiple elements (elements of a set) (here, feature vector F(251)) are sequentially grouped to form hierarchical clusters (groups).

Specifically, in the hierarchical clustering process, the most similar (highest mutual similarity) tentative clusters (described below) are sequentially combined one by one to generate clusters (until the entire cluster is reduced to one cluster). This results in the formation of hierarchical clusters. A tentative cluster (provisional cluster) is initially composed of a single feature vector F, and thereafter is composed of a single feature vector F or two or more feature vectors F. A tentative cluster composed of two or more feature vectors F represents a (new) cluster generated in the hierarchical clustering process.

In hierarchical clustering, whether two provisional clusters are similar to each other is determined based on the distance between the two provisional clusters (e.g., Euclidean distance or cosine similarity). Hierarchical clustering can be performed using a variety of methods, including the centroid method, shortest distance method, longest distance method, group average method, or Ward's method. These methods are classified according to the specific quantity used to calculate the distance (similarity) between the two provisional clusters. For example, the centroid method uses the distance between the centroids of the two provisional clusters as the distance (similarity) between the two provisional clusters. The shortest distance method uses the shortest inter-element distance between any element in one provisional cluster and any element in the other provisional cluster as the distance between the two provisional clusters. Note that the index value representing similarity in each method is not limited to distance; cosine similarity, etc., may also be used. The following examples primarily use cosine similarity (see Equation (1)).

More specifically, in the hierarchical clustering process, an evaluation value (specifically, similarity St (see equation (1))) is first calculated for all combinations of provisional clusters (initially, individual feature vectors F). Then, a new cluster is formed by combining the provisional clusters (element pairs) with the highest evaluation value (similarity St). By repeating the same process, new clusters (same-level clusters or higher-level clusters) are sequentially formed, until a single large cluster is formed. The inclusion relationships (in other words, the hierarchical relationships) of the multiple clusters that are formed are represented by a single tree diagram (also called a dendroid) (see the top row of Figure 11).

The lower part of Figure 11 (and Figure 16), similar to Figure 8, shows how multiple feature vectors F (251) are distributed within the feature space (the output space of the trained model 420). The multiple feature vectors 251 having such a distribution form multiple clusters hierarchically organized in multiple layers. The upper part of Figure 11 also shows a tree diagram (dendroid) for the multiple feature vectors F.

For example, during the intermediate stage of the hierarchical clustering process described above, cluster G310 (see the area near the center left of Figure 11) is formed. Cluster G310 is composed of multiple (six in Figure 11) feature vectors 251 (dotted black circles) that are close to each other. Then, at a subsequent stage, cluster G300 is formed, which encompasses cluster G310. Cluster G300 is composed of feature vectors 251 within cluster G310 and other feature vectors 251 (hatched dotted black circles) that are close to cluster G310. Cluster G300 is also referred to as the higher-level cluster (also referred to as the parent cluster) of cluster G310.

Similarly, at a certain intermediate stage of the hierarchical clustering process, clusters G111, G112, and G120 are formed (see the lower center of Figure 11). Cluster G111 is composed of multiple (six in Figure 11) feature vectors 251 (dotted white circles with vertical and horizontal cross (checkerboard) hatching) that are close to each other. Cluster G112 is composed of multiple (six in Figure 11) feature vectors 251 (dotted white circles with diagonal cross hatching) that are close to each other. Cluster G120 is composed of multiple (six in Figure 11) feature vectors 251 (dotted white circles without hatching) that are close to each other. At a subsequent stage, cluster G110 (a super-cluster of clusters G111 and G112) is formed, encompassing both clusters G111 and G112. At a later stage, a cluster G100 (a super-cluster of the two clusters G110 and G120) is constructed that includes both clusters G110 and G120. At an even later stage, a cluster G10 (a super-cluster of the two clusters G100 and G200) is constructed that includes both clusters G100 and G200.

In addition, other clusters G400, G510, G500, etc. will be formed as the hierarchical clustering process progresses.

<Results of hierarchical clustering processing>
By such hierarchical clustering processing, a plurality of clusters (G111, G112, G110, G120, G100, G200, G10, G310, G300, G400, G510, G500, etc.) are formed, for example, as shown in Fig. 11. Note that, for convenience of illustration, Fig. 11 shows only some of the many feature vectors 251, and also shows only some of the many clusters.

Furthermore, each feature vector F (251) corresponds to a respective input image X (211). Therefore, the hierarchical clustering process is a process of clustering multiple feature vectors F (251) as well as a process of clustering multiple input images X (211) (see Figure 13). Figure 13 is a diagram showing some of the multiple clusters shown in Figure 11. Figure 13 shows input images 210 (211) corresponding to these some of the clusters. Note that while input image 210 is actually a photographed image, for convenience of illustration, in Figure 13 (and the subsequent figures (Figures 14, 21, 25, 26, etc.)), input images 210 (211, 213, 215) are represented as CG (computer graphics) images.

For example, cluster G111 can also be expressed as including a plurality of input images corresponding to a plurality of dotted white circle shapes with vertical and horizontal cross (checkerboard) hatching (see Figure 11) (see also Figure 13). Similarly, cluster G112 includes a plurality of input images corresponding to a plurality of dotted white circle shapes with diagonal cross hatching (see Figure 11). Furthermore, upper cluster G110 includes both a plurality of input images included in lower cluster G111 and a plurality of input images included in lower cluster G112.

More specifically, as shown in FIG. 13, cluster G111 is composed of multiple images 211 of at least one person (here, three or more people) wearing a "heavily patterned white (white background) shirt (where the pattern is linear)." Cluster G112 is composed of multiple images 211 of at least one person wearing a "heavily patterned white shirt (where the pattern is curved)." Upper cluster G110 is composed of both person images 211 included in lower cluster G111 and person images 211 included in lower cluster G112. More specifically, upper cluster G110 is composed of multiple images 211 of at least one person wearing a "heavily patterned white shirt" (where the pattern may be linear or curved).

Furthermore, cluster G120, which has a peer relationship with cluster G110 (the cluster corresponding to images of people wearing "white shirts with a lot of pattern"), is composed of multiple images 211 of at least one person wearing a "white shirt with a little pattern."

Furthermore, the upper cluster G100 is composed of both the person images 211 included in the lower cluster G110 and the person images 211 included in the lower cluster G120. More specifically, the upper cluster G100 is composed of multiple images 211 of at least one person wearing a "white shirt with a pattern" (which may have a lot of pattern or not).

Furthermore, cluster G200, which has a peer relationship with cluster G100 (the cluster corresponding to images of people wearing "patterned white shirts"), is composed of multiple images 211 of at least one person wearing a "light pink shirt."

Other clusters similarly consist of images of people dressed similarly to each other for a particular concept.

In this way, clustering multiple feature vectors 250 (251) corresponding to each input image 210 (211) is equivalent to clustering multiple input images 210.

In other words, multiple input images belonging to a specific cluster (generated by the hierarchical clustering process) share common features, and the specific cluster is interpreted as having a unique concept.

Furthermore, the concepts of a higher-level cluster are concepts (overarching concepts) that encompass (subsume) the concepts of its lower-level clusters. Conversely, the concepts of lower-level clusters are concepts that subdivide the concepts of their higher-level clusters. In short, higher-level concepts are coarse-grained concepts, while lower-level concepts are fine-grained concepts. It can also be said that higher-level concepts are concepts shared by a relatively large number of people, while lower-level concepts are concepts shared by a relatively small number of people.

Note that the tree diagram (dendroid) in Figure 11 is just an example, and different tree diagrams will be generated depending on the training data, etc.

Furthermore, such hierarchical clustering processing generates a very large number of clusters. All clusters generated by the hierarchical clustering processing may be used in subsequent processing (particularly processing after step S31 (described below)), but this is not limited to this. For example, of these many clusters, it is preferable to use clusters containing a predetermined number of people (e.g., three people) or more as clusters that provide explanatory grounds for similarity (clusters that form the concept of said explanatory grounds). This makes it possible to improve the robustness of the concept (by reducing dependence on characteristics unique to people in the training data). In other words, clusters consisting of only a single person may be excluded from clusters that provide explanatory grounds for similarity.

In particular, it is preferable to use only clusters whose concept vectors U (described below) are linearly independent of each other out of all the clusters generated by the hierarchical clustering process.

<Step S23: Extraction process of concept vector U>
In the next step S23, the controller 31 extracts a subspace or vector corresponding to each cluster as a concept of the cluster (step S23). Here, a vector ("concept vector" (described next)) corresponding to each cluster is extracted as a concept of the cluster.

More specifically, the controller 31 first selects, from all clusters, clusters that can be considered for subsequent processing. Specifically, from all clusters generated by the hierarchical clustering process, a portion of clusters that satisfy predetermined criteria are selected. The selected clusters are also referred to as candidate clusters for consideration. The predetermined criteria may be, for example, that the clusters contain a predetermined number of people (e.g., three people) or more (as described above). Alternatively, all clusters may be selected as candidate clusters for consideration.

The controller 31 then extracts each vector U corresponding to each of the selected clusters (multiple specific clusters) as a concept for that cluster (each specific cluster) (step S23). Specifically, the vector U corresponding to a specific cluster is a representative vector for that specific cluster. Note that the "representative vector" does not have to be one of the multiple feature vectors belonging to the specific cluster; it may be a vector that representatively represents a group made up of the multiple feature vectors (a representative vector that symbolizes the multiple feature vectors belonging to the specific cluster). Furthermore, since vector U is a vector that expresses the concept of the specific cluster (a vector representation of the concept), it is also referred to as a "concept vector." It is preferable that the concept vector U is also normalized, similar to feature vector F.

The concept vector U of a specific cluster is calculated, for example, as the average vector of multiple feature vectors F (251) that belong to (constitute) that specific cluster. This average vector is also called the centroid vector, because it indicates the centroid position (average position) of the multiple feature vectors F on the hyperplane (hypersphere) of the feature space.

Alternatively, the concept vector U of a specific cluster may be the "concept activation vector" (CAV) of the specific cluster. The concept activation vector of a specific cluster is the normal vector of a separation plane 501 that separates elements belonging to the specific cluster from other elements (see Figure 17). Figure 17 shows the separation plane 501 in feature space (here, a hypersphere). The separation plane 501 is a plane that separates elements belonging to the specific cluster Ga (see the dotted white circle (feature vector Fa) represented by) from elements that do not belong to the specific cluster Ga (see the other dotted circle (feature vector Fb) represented by). Such a separation plane 501 can be obtained using a two-class linear separator (linear classifier) (such as a support vector machine). The vector perpendicular (and pointing outward) to this separation plane 501 is the concept activation vector (CAV) of the specific cluster Ga.

Figures 18 to 20 show the concept vectors U (CAV, etc.) for each cluster. In each figure, a certain cluster G is shown on the left, and the concept vector U for that certain cluster G (a concept vector expressed as a vector pointing to a point on the hypersphere) is shown on the right.

For example, the upper part of Figure 18 shows the concept vector U (U110) of cluster G110, and the middle part of Figure 18 shows the concept vector U (U120) of cluster G120. The lower part of Figure 18 shows the concept vector U (U100) of cluster G100.

The upper part of Figure 19 shows the concept vector U (U200) of cluster G200, and the middle part of Figure 19 shows the concept vector U (U310) of cluster G310. The lower part of Figure 19 shows the concept vector U (U300) of cluster G300.

Similarly, the upper part of Figure 20 shows the concept vector U (U400) of cluster G400, and the middle part of Figure 20 shows the concept vector U (U510) of cluster G510. The lower part of Figure 20 shows the concept vector U (U500) of cluster G500.

Although not shown here, other concept vectors U, such as concept vectors U111, U112, U10, etc., can also be calculated in a similar manner.

As shown in these figures, the concept vectors U of clusters that are close to each other in the dendroid (see top row of Figure 11) are relatively similar. For example, the concept vector U110 of cluster G110 (see top row of Figure 18) and the concept vector U120 of cluster G120 (see middle row of Figure 18) are relatively close to each other (have relatively similar orientations). Conversely, the concept vectors U of clusters that are far apart from each other in the dendroid (see top row of Figure 11) are significantly different from each other. For example, the concept vector U100 of cluster G100 (see bottom row of Figure 18) and the concept vector U500 of cluster G500 (see bottom row of Figure 20) are significantly different from each other (have significantly different orientations).

In this way, the similarity of the concept vector U reflects the similarity of the features of each cluster (in other words, the concepts of each cluster).

Furthermore, as mentioned above, multiple input images belonging to a specific cluster (generated by the hierarchical clustering process) share common characteristics, and the specific cluster is interpreted as having a unique concept.

The controller 31 therefore extracts the concept vector U of each cluster as the concept (or concept expression) of that cluster. The concept vector U of each cluster is a vector that expresses the concept of that cluster. In other words, the concept vector U of each cluster is also a "vector-based concept expression" of that cluster.

As described above, the trained model 420 is interpreted as a model that has learned the concepts of each cluster (generated by the hierarchical clustering process). The concept vector U of each cluster is then extracted as the concept of that cluster.

As mentioned above, each concept vector U is represented three-dimensionally in Figure 16 and Figures 18 to 20. However, in reality, each concept vector U is a vector (multidimensional vector) of very high dimension (β dimension (e.g., 1024 dimension)). Therefore, a very large number (γmax) (e.g., 300) (where γmax < β)) of concept vectors U corresponding to different clusters (concepts) exist as linearly independent vectors.

Furthermore, there are limitations to illustrating (three-dimensionally expressing) such multidimensional (e.g., 1024-dimensional) concept vectors U. Each concept vector U often has substantial feature components in dimensions other than the three dimensions that can be illustrated (such as the i-th dimension from the fourth dimension onward). For example, in Figures 16, 18 to 20, etc., multiple concept vectors U that are significantly different from one another have different orientations within the three dimensions. However, in reality, multiple concept vectors U that are significantly different from one another often have different orientations in the i-th dimension from the fourth dimension onward.

<1-9. Similarity of two images (similarity Sc for each concept group, etc.)>
Before describing the processing of sub-phase PH3b (FIG. 5), the similarity between two images (an image pair) will be described. The two images (first and second images) are, for example, a query image and a gallery image extracted in step S12. However, the present invention is not limited to this, and the similarity between any two images can be determined in the same manner.

First, as described above, the similarity St between two input images (the overall similarity St between an image pair) is expressed by the above formula (1). That is, the similarity St between two input images X is calculated (expressed) as the dot product (q·g) (here, cos θ) of the output vectors q and g (feature vector F) from the learning model 400 for the two input images X. For example, the similarity St between two input images (query image and gallery image) shown in Figure 21 is calculated as the dot product of their respective feature vectors q and g (St = 0.778 in Figure 21).

Here, we introduce another index (specifically, similarity Sc) different from the similarity St (overall similarity). This similarity Sc is an index value indicating the degree of similarity (between two images) explained by a concept (also referred to as a concept under consideration or a selected concept) selected from multiple concepts for consideration of its contribution to similarity. Simply put, similarity Sc does not represent the overall similarity between two input images, but rather the partial similarity resulting from some of the selected concepts. In other words, similarity Sc represents the component contributed by some of the selected concepts (concepts under consideration) to the overall similarity St between two input images. In other words, similarity Sc is an index indicating the degree of contribution (contribution) of the selected concept to the similarity between the two images. Similarity Sc can also be expressed as the similarity (contribution) for each concept (each concept vector or each cluster). Note that similarity Sc, like similarity St, is a scalar (value).

This similarity Sc for the selected concept is expressed as the degree of similarity between image features (feature vectors Fq (= q) and Fg (= g)) in a subspace (also called a specific subspace) spanned by the concept vector U corresponding to the selected concept. Specifically, this similarity Sc is expressed as the dot product of a vector (Pq) obtained by projecting (orthogonally projecting) the first feature vector q onto the specific subspace, and a vector (Pg) obtained by projecting (orthogonally projecting) the second feature vector g onto the specific subspace. Vector (Pq) is a vector (orthogonal projection vector) obtained by applying (from the left) the projection matrix P (described below) to feature vector q, and vector Pg is a vector (orthogonal projection vector) obtained by applying (from the left) the projection matrix P to feature vector g.

That is, the similarity Sc is expressed by the following equation (2):

However, matrix P is a specific projection matrix (more specifically, an orthogonal projection matrix) that projects each feature vector F onto the specific subspace (the subspace spanned by concept vector U corresponding to the selected concept). Specifically, the projection matrix (orthogonal projection matrix) P is calculated using matrix B according to the following equation (3). Matrix P has a size of β×β. The value β is the number of dimensions of vector F (q or g) (for example, 1024).

Here, matrix B is a matrix in which a predetermined number (the number of concepts to be selected (γ)) of concept vectors U (column vectors) are arranged horizontally. Matrix B has a size of β x γ (for example, 1024 x 2). γ is the number of concepts to be selected (for example, 2 (or 1 or 3, etc.)). The subscript "T" in the upper right corner of the matrix indicates that it is a transpose (matrix).

The γ concept vectors U are selected so that they are linearly independent of each other. Furthermore, the maximum value of the number γ of selected concepts is β.

Note that equation (3) is also a general equation for finding a matrix (orthogonal projection matrix) that represents an orthogonal projection onto a subspace spanned by one or more vectors b (a subspace based on two or more vectors b). However, the concept vector U is used as vector b.

Matrix B (and therefore matrix P) will differ depending on the selected concepts (more specifically, the concept vectors corresponding to those selected concepts). More specifically, matrix B (and matrix P) will vary depending on which concepts (and how many concepts) are selected.

For convenience in calculating the inverse matrix, the projection matrix P may be calculated using the following equation (4) instead of equation (3). The term +εE is an adjustment term to prevent divergence to infinity. Matrix E is a unit matrix of dimension β (unit matrix of β × β (size)), and the value ε is a tiny constant.

Referring again to equation (2), we will explain equation (2).

In equation (2), as described above, two feature vectors F (specifically, vectors q and g) corresponding to two input images X (Xq, Xg) are transformed into two projection vectors (PF) using a specific projection matrix P. The projection vectors (PF) are vectors obtained by applying the projection matrix P to the feature vectors F (from the left). As described above, the specific projection matrix P is a projection matrix (orthogonal projection matrix) that projects (orthogonally projects) each feature vector in the feature space onto a specific subspace (a subspace corresponding to a specific concept).

This specific projection matrix P (see equation (3), etc.) is defined for each concept group consisting of n selected concepts (one or more concepts) to be evaluated.

For example, if the concept to be evaluated is a single concept, the specific projection matrix P is a projection matrix that projects each feature vector F in the feature space onto a specific line (a specific subspace). More specifically, the specific projection matrix P for a single concept defined by a single concept vector U is a projection matrix that projects each feature vector in the feature space onto a specific line (a line that includes the single concept vector U).

Figure 23 shows how each feature vector q, g is projected (orthogonally projected) onto a specific line (a line containing a single concept vector U1) using such a projection matrix P (which is an orthogonal projection matrix onto a subspace (i.e., a line) spanned by a single concept vector U1).

As shown in Figure 23, this orthogonal projection matrix P transforms vector q into vector q1 and vector g into vector g1. In Figure 23, the transformed vectors q1 and g1 are indicated by thick dashed lines. In this way, vectors q1 and g1 are projection vectors obtained by projecting two feature vectors q and g onto a subspace (more specifically, a line) corresponding to a specific cluster (specific concept vector).

Therefore, equation (2) is transformed into the following equation (5). Specifically, the inner product between the two projection vectors q1 and g1 (q1 * g1 = qs * gs) is calculated as the similarity Sc for each cluster (concept).

Here, the value qs is the magnitude of vector q1 (the projected component of vector q projected onto a line in the direction of concept vector U), and the value gs is the magnitude of vector g1 (the projected component of vector g projected onto a line in the direction of concept vector U). However, if the angle between vector q1(, g1) and concept vector U is between 90 and 270 degrees, the value qs(, gs) is a negative value. The similarity Sc shown in equation (2) is calculated as the product of the values qs and gs, as shown in equation (5).

Note that Figure 24 is similar to Figure 23. However, Figure 24 illustrates the case where the concept vector U is oriented in the same direction as the x-axis. In this case, as shown in Figure 24, the value qs is equal to the x-direction component (first component) of vector q1, and the value gs is equal to the x-direction component (first component) of vector g1. Figure 24 is shown for the sake of simplicity, but a projective transformation like that shown in Figure 23 is generally assumed.

According to equation (2) (particularly equation (5)), the contribution of a single concept vector U (in other words, the contribution of a single selected concept to the similarity between both images) is calculated as the similarity Sc. In other words, the similarity Sc for each concept is calculated.

Furthermore, when there are two concepts to be evaluated, the specific projection matrix P is a projection matrix that projects each feature vector F in the feature space onto a specific plane (a specific subspace). More specifically, the specific projection matrix P for the two concepts is a projection matrix that projects each feature vector F in the feature space onto a specific plane (a plane spanned by the two concept vectors U1 and U2 corresponding to the two concepts). Then, according to equation (2), the inner product (Pq) x (Pg) of the two projection vectors obtained after projecting the two feature vectors F using the projection matrix P is calculated as the contribution Sc of the concept to be evaluated to the similarity between the two images.

Figure 22 shows how each feature vector q, g is projected onto the plane (the plane spanned by the two concept vectors U1, U2) using this projection matrix P (an orthogonal projection matrix onto the subspace (plane) spanned by the two concept vectors U1, U2). For simplicity, Figure 22 shows the case where the concept vectors U1, U2 are parallel to the plane z=0 (xy plane). In other words, the case where the plane spanned by the two concept vectors U1, U2 is the plane represented by z=0 is shown.

As shown in Figure 22, the orthogonal projection matrix P calculated using the two concept vectors U1 and U2 transforms vector q into vector q12 and vector g into vector g12. Note that the transformed vectors q12 and g12 are shown by thick dashed lines.

In this case, the dot product of both vectors q12 and g12 after transformation using the orthogonal projection matrix P (in other words, the similarity between both vectors q12 and g12 in the specific subspace (plane z = 0)) is calculated as the similarity Sc. In other words, when two concepts are being considered (evaluated), the dot product of the two vectors (q12 · g12) after projecting two feature vectors F onto the plane spanned by the two corresponding concept vectors U1 and U2 is calculated as the similarity Sc.

In this way, the contribution of the two concept vectors U1 and U2 (in other words, the contribution of the two selected concepts to the similarity between the two images) can be calculated as the similarity Sc based on equation (2).

Similarly, the contribution (similarity) Sc for any number (predetermined number) of concept vectors U (in other words, the contribution of the predetermined number of selected concepts to the similarity between both images) can be calculated based on equation (2). The contribution Sc for three or more concept vectors U can be thought of as the inner product of two vectors projected from a pre-transformation space of four or more dimensions to a post-transformation space of lower dimensions (dimensions equal to the number of concept vectors) (however, this is difficult to illustrate).

<1-10. Processing of subphase PH3b (step S30)>
<Outline of Sub-Phase PH3b>
Next, the processing of sub-phase PH3b (step S30) will be described with reference to Fig. 5. The processing of sub-phase PH3b includes deriving a basis for determining the similarity between two images.

In subphase PH3b, the image processing device 30 executes processes such as explaining the basis for the image processing device 30's determination that the images are of the same person (or similar people). Here, the process of deriving the basis for determining the similarity between two images (query image Xq and gallery image Xg) determined in step S12 to be images of the same person (images that are similar to a predetermined degree or more) will be mainly described. However, the present invention is not limited to this, and processes such as determining the similarity between any two images and deriving the basis for that determination may be executed.

The processing of this subphase PH3b can analyze whether the basis for determining similarity is, for example, whether the person is wearing a shirt with specific characteristics, or whether the person is wearing bottoms with specific characteristics, etc. It can also analyze whether characteristics were found in the pattern of the shirt, etc.

In this sub-phase PH3b, the basis for determining similarity is analyzed using the concept vector U described above (see step S23 (Figure 4)).

<Step S31: Extraction of main concepts>
Specifically, in step S31 ( FIG. 5 ), among the various concepts acquired by the learning model 400, concepts that have a particularly large influence on the similarity between two images (concepts with a large contribution) are extracted as main concepts. More specifically, among the multiple concepts, the top several concepts according to a predetermined criterion are extracted as main concepts. For example, the main concepts are extracted by executing one of the following two methods, specifically, the first method or the second method.

Below, two methods (method 1 and method 2) are illustrated.

First, we will explain the first method.

The first method calculates the contribution Sc (see formula (5)) of each concept to the similarity between image pairs for all concepts, and extracts the top few concepts with the highest contribution Sc as main concepts.

Specifically, in the first technique, the controller 31 first calculates the contribution Sc of a single concept vector U (the similarity Sc based on the above-described formula (5)) for each of multiple candidate concept vectors U (described below) (e.g., all concept vectors U). That is, the controller 31 performs a process for calculating the degree to which a concept represented by a single concept vector U corresponding to a certain cluster contributes to the overall similarity (a process for calculating the contribution Sc for each concept) for all concepts. Next, the controller 31 sorts all concepts in descending order of their contribution. The controller 31 then determines the top few concepts as main concepts (main concepts) that contribute significantly to the similarity between the two images.

Note that in the first method (and the second method as well), all of the multiple concepts (multiple concept vectors U) extracted in step S20 above may be identified as candidate concepts (also referred to as candidate concepts) that serve as the basis for similarity. However, this is not limited to this, and the candidate concepts may be some of all of the concepts extracted in step S20 above. In other words, the candidate concept vectors (also referred to as candidate concept vectors U) that serve as the basis for similarity may be some of all of the concept vectors extracted in step S20 above. In this way, a main concept (main concept vector) may be determined from some of the candidate concepts (candidate concept vectors).

Next, we will explain the second method.

The second method is similar to the first method in that it considers contribution (similarity).

However, in the second method, a search process is repeatedly performed to search for a next-ranked concept from among concepts (unselected concepts) other than concepts (selected concepts) that have already been selected (considered) at a certain point in time (at the time of the i-th iteration). Here, the next-ranked concept is the concept, among the unselected concepts, that most significantly increases the considered similarity component (the similarity component that is additionally explained by that concept). Specifically, a search process is repeatedly performed to search for a concept vector U that maximizes the considered similarity component from among the concept vectors U (unselected concept vectors U) of unselected (unconsidered) concepts at a certain point in time. In other words, an unselected concept (next-ranked concept) that maximizes the newly (additionally) explainable component from among the similarity components not yet explained by the selected concepts is sequentially searched for.

Specifically, the concept vector U that contributes most to similarity is searched for in the "orthogonal complement" of the subspace spanned by the concept vector U already selected at a certain point in time (see equation (6) below, etc.). This search process is then repeated to determine the main concept. In this respect, the second method differs from the first method, which determines the main concept primarily using the contribution of each concept.

The second method makes it possible to extract concepts with high independence as main concepts. In other words, it is possible to extract main concepts with minimal overlap between concepts.

The second method is explained in detail below.

In the second method, each feature vector F (specifically, feature vectors q and g) is transformed into a projection vector using a specific projection matrix R (described next).

A specific projection matrix R is a matrix (R = 1-P) expressed as (1-P) using a specific projection matrix P.

A specific projection matrix P (more specifically, also referred to as Pn) is a projection matrix (more specifically, an orthogonal projection matrix) based on n (γ) concept vectors U that have already been considered at a certain point in time. The projection matrix P (more specifically, Pn) is a matrix calculated using equation (3) or (4) based on matrix B, in which n concept vectors U are arranged horizontally. Note that the value n increases by one each time the above search process is repeated.

On the other hand, the projection matrix R (more specifically, also referred to as Rn) is a matrix that projects each feature vector F onto a subspace (orthogonal complement) that is orthogonal to the projection space (subspace) created by the projection matrix Pn, and is equivalent to (1-Pn). In other words, the orthogonal complement of the projection space (subspace) created by the projection matrix Pn is the projection space (subspace) created by the projection matrix Rn. Note that the projection matrix Rn is also an orthogonal projection matrix.

Note that feature vector F is separated into vector (PF) obtained by applying projection matrix P (= Pn), which projects onto a specific subspace, to feature vector F, and vector (RF) obtained by applying projection matrix R (= Rn = 1 - Pn), which projects onto the orthogonal complement of that specific subspace, to feature vector F. Vector (PF) can also be expressed as a component considered in the n concept vectors U that make up projection matrix P (more specifically, matrix B). Vector (RF) can also be expressed as a component not yet considered in the n concept vectors U that make up projection matrix P (matrix B).

Furthermore, if the projection matrix P is, for example, a projection matrix onto a subspace (specific straight line) spanned by a single concept vector U, the projection matrix R is a projection matrix onto a subspace (the remaining subspace) orthogonal to that subspace (specific straight line). For simplicity's sake, if the concept vector U is considered a three-dimensional vector, the projection matrix R is a projection matrix onto a subspace ("plane (plane perpendicular to the specific straight line)") orthogonal to that subspace (specific straight line).

In this second method, a new concept (more specifically, a new concept vector Ur corresponding to the new concept) is searched for that maximizes the index value Qn expressed by the following equation (6). Specifically, a new concept vector U of the (n+1)th rank is searched for. In other words, the next-ranked concept vector U (and therefore the next-ranked concept) is searched for.

Specifically, for each of the (remaining) concept vectors Ur (candidates for new concepts to be selected), an index value Qn is calculated based on equation (6). Here, "concept vectors already selected (considered)" refers to the n (one or more) concept vectors U used to construct the matrix Pn (the matrix corresponding to the matrix Rn). Furthermore, "concept vectors not yet selected (considered)" refers to the concept vectors U (remaining concept vectors U) other than the n concept vectors U used to construct the matrix Pn.

The vector Ur in equation (6) is an unselected concept vector U (a candidate concept vector U) corresponding to a new candidate concept to be selected.

Vector (Rq) is the vector obtained by applying the projection matrix R to the feature vector q (from the left), and vector Rg is the vector obtained by applying the projection matrix R to the feature vector g (from the left).

In other words, vector (Rq) is the vector obtained by projecting feature vector q onto the orthogonal complement of the projected space defined by projection matrix P, and vector (Rg) is the vector obtained by projecting feature vector g onto the orthogonal complement of the projected space defined by projection matrix P.

These vectors (Rq) and (Rg) are each projection vectors onto a subspace that has not yet been considered. Furthermore, the dot product (Rq・Ur) of vector (Rq) and vector Ur (a new concept vector that has not yet been considered) represents the contribution component (projection component onto vector Ur) of the projection vector onto a subspace that has not yet been considered, made by vector Ur. The same is true for the dot product (Rg・b) of vector (Rq) and vector Ur (a new concept vector that has not yet been considered).

In equation (6), the product (scalar (value)) of these two dot products is calculated as the index value Qn. Specifically, the dot product ((Rq) · Ur) of vector (Rq) and vector Ur is calculated, and the dot product ((Rg) · Ur) of vector (Rg) and vector Ur is calculated, and the product (scalar product) of these dot products (the right-hand side of equation (6)) is calculated.

This index value Qn is the product of the inner product ((RF) * Ur) of the projection vector (vector (RF) obtained by projecting feature vector F onto the orthogonal complement) of a subspace not yet considered in the concepts already considered (the "orthogonal complement" of the subspace spanned by the concept vector U already considered) and the new concept vector Ur.

Each projection vector (RF) obtained by projecting each feature vector F onto a subspace (orthogonal complement) that has not yet been considered by the concepts considered so far (one or more concept vectors U) represents an unconsidered component of each feature vector F (a component that has not yet been considered by the concepts previously considered). Furthermore, the inner product of the unconsidered component and the new concept vector U represents the similarity between the unconsidered component and the new concept (corresponding to the new concept vector U). Therefore, the index value Qn represents the similarity between the unconsidered component of the two feature vectors q and g and the new concept. In other words, the index value Qn indicates the extent to which the new concept can explain the basis for determining similarity among the unconsidered components. In other words, the index value Qn represents the degree to which the new concept (concept vector U) contributes to the unconsidered component.

Therefore, the concept vector U that maximizes the index value Qn is the concept that maximizes the similarity with the unconsidered components for the two feature vectors F, and is the concept that can best explain the unconsidered components.

As a result of the search process, the concept vector Ur that maximizes the index value Qn from among the unselected concept vectors Ur is determined as the new concept vector U (next-ranked concept vector). In other words, the concept corresponding to the concept vector U that maximizes the index value Qn is selected as the new main concept (next-ranked ((n+1)th) main concept). In other words, the cluster that maximizes the index value Qn is determined as the next-ranked cluster.

Furthermore, new matrices P and R are calculated using this new concept vector (next-ranked concept vector). Specifically, with the addition of the next-ranked concept vector, the number of selected concept vectors U increases (increments) by one, and the value n also increases by one. Furthermore, in response to the increment in the number of selected concept vectors U, the rank (rank) of projection matrix P increases by one, and the rank (rank) of projection matrix R decreases by one. In other words, the number of dimensions of the projection space created by projection matrix P increases by one, and the number of dimensions of the projection space created by projection matrix R decreases by one.

The search process is then repeated in a similar manner until a predetermined number (e.g., five) of major concepts are selected. This allows the second method to determine the top few (top predetermined number) major concepts (more specifically, the concept vectors U corresponding to those major concepts).

More specifically, when determining the first-ranked concept vector U, it is sufficient to start with zero concept vectors U selected. In this case, in equation (6), the vector (Rq) is the feature vector q itself, and the vector (Rg) is the feature vector g itself, and the dot product between the vector (Rq) and the candidate concept vector U can be calculated.

Alternatively, similar to the first method, similarities Sc for multiple concept vectors U may be calculated based on formula (5), and the concept vector U corresponding to the maximum value of the calculated similarities Sc may be determined as the first-ranked concept vector U. Note that when determining the first-ranked concept, formulas (6) and (5) are equivalent (Rq Ur = q Ur = qs = Pq).

For example, after the first-ranked concept vector U is determined as concept vector U110, when the second-ranked concept vector U is determined, matrix B becomes a single concept vector U110 (column vector). Then, matrix P (rank = 1) is calculated based on matrix B (the column vector) (see equation (3) or (4)), and matrix R is calculated based on matrix P (R = 1 - P). Then, the next-ranked (second-ranked) concept vector U (for example, concept vector U400) that maximizes the index value Qn of equation (6) is calculated.

Furthermore, when the third-ranked concept vector U is subsequently determined, matrix B is a matrix in which two concept vectors U110 and U400 are arranged horizontally. Then, matrix P (rank = 2) is calculated based on matrix B (see equation (3) or (4)), and matrix R is calculated based on matrix P (R = 1 - P). Then, the next-ranked (third-ranked) concept vector U (for example, concept vector U510) that maximizes the index value Qn of equation (6) is calculated.

The same process is then repeated to determine the top few (e.g., five) concept vectors U.

<Another termination condition>
Here, the search process is repeated until a predetermined number of main concepts are obtained (in other words, the termination condition is set to be that a predetermined number of main concepts are obtained), but this is not limited to this, and the search process may be repeated under a different termination condition (termination determination condition).

For example, the termination condition may be set to be that the condition in equation (7) is satisfied. In particular, the termination condition in equation (7) may be determined after the value n is incremented.

The left-hand side of equation (7) ((Pq)·(Pg)) is the same as the similarity Sc described above (see equation (2)). The matrix P is a projection matrix onto a subspace spanned by n selected concept vectors U representing selected concepts (already selected concepts). In other words, the left-hand side of equation (7) represents the inner product of the projection vectors obtained by projecting vectors q and g onto the subspace (i.e., the contribution of the n selected concept vectors U).

On the other hand, the right-hand side of equation (7) is the value obtained by multiplying the inner product of the two vectors q and g before transformation by a constant ratio (1-δ). Here, the value δ is a constant indicating the ratio (0<δ<1), and the value (1-δ) also indicates the ratio to the whole.

For example, if the value δ is 0.4, the value (1 - δ) is 0.6. In this case, the termination condition of equation (7) means that the inner product of the transformed vectors (Pq) and (Pg) is greater than 60% of the inner product of the untransformed vectors q and g. In other words, equation (7) means that the search process ends when the value Sc (the similarity Sc calculated using the projection matrix P corresponding to n concept vectors U), which gradually increases as the value n increases, becomes greater than a certain percentage (1 - δ) of the overall similarity St.

For example, after the first five concept vectors U have been determined, when searching for a new concept vector Ur of the (n+1)th order (specifically, the sixth order), if the termination condition of equation (7) is met, the repeated search process ends. Then, the first five concept vectors U are determined as the main concept vectors.

In this way, instead of predetermining the number n of concept vectors U, the value δ (and thus the proportion of contributions to be considered among the n concept vectors U) may be predetermined.

<Example of concept decision>
According to the second technique described above, for example, the top (first) concept C1, the second concept C2, the third concept C3, the fourth concept C4, and the fifth concept C5 are searched for and found in this order.

More specifically, concept vector U110 (see the top row of Figure 18) is identified as the highest-level concept C1 (see also Figures 11 and 13, etc.). In this case, it can be seen that the greatest basis for determining the similarity between the two images (the basis for determination by the trained model 420) is the feature expressed by concept vector U110. In other words, it can be seen that the feature of cluster G110 corresponding to concept vector U110 is the greatest basis for determining (the similarity). Since cluster G110 is composed of a group of images of a person wearing a "white shirt with many patterns" (see Figures 11 and 13), concept C1 can also be expressed as "white shirt with many patterns."

Furthermore, concept vector U400 (see the top row of Figure 20) is identified as the second-ranked concept C2. In other words, it can be seen that the characteristics of cluster G400 corresponding to concept vector U400 provide a relatively large basis for judgment from a different perspective than the first-ranked concept C1. Since cluster G400 is composed of a group of images of people wearing "short white bottoms" (see Figure 11, etc.), concept C2 can also be expressed as "short white bottoms."

Similarly, concept vector U510 (see middle row of Figure 20) is identified as the third-ranked concept C3 ("light blue bottoms"). Furthermore, concept vector U200 (see top row of Figure 19) is identified as the fourth-ranked concept C4 ("light pink shirt"), and concept vector U310 (see middle row of Figure 19) is identified as the fifth-ranked concept C5 ("plain wine-red shirt").

<Step S32: Explanation of main concepts, etc.>
Next, in step S32 (FIG. 5), a process is executed to display the analysis results, etc., from step S31. In other words, a process is executed to present the basis (explanatory information) for determining that the two images are similar to each other.

Figure 25 is a diagram showing an example (screen display example) of the search processing results (analysis processing results) described above. Note that the display screen 600 in Figure 25 and various other screens 610, 620 (described below) are displayed, for example, on the display unit 35b.

Display screen 600 in Figure 25 presents five main concepts C1 to C5 based on the second method.

Specifically, in the graph display area 609 of the display screen 600, the similarity Sc calculated based on equation (2) (more specifically, equation (5)) is displayed in a graph. For the top few (here, five) concepts, the contribution to the similarity between the image pairs (here, the similarity between the query image and the gallery image) is calculated and this contribution is displayed. Note that the contribution may be displayed as a numerical value (e.g., 0.18) or may be displayed as a graph (with the numerical value converted into the length of the bar graph) as shown in FIG. 25.

Here, five main concepts C1 to C5, ranked first to fifth, are determined based on the second method. The similarity (contribution) Sc is also shown for each of these five main concepts C1 to C5. Furthermore, the contribution (remaining contribution) of concepts other than the five main concepts C1 to C5 is also shown (in the "Other" column). This remaining contribution is calculated, for example, by subtracting the similarity Sc (see formula (2)) calculated using the five concepts C1 to C5 as the selected concepts from the overall contribution St (see formula (1)).

In particular, the five concepts C1 to C5 are extracted as main concepts so that they are arranged in descending order from highest to lowest. This order C1 to C5 is based on the second method and is different from the order based on the similarity Sc calculated using equation (5) (the order based on the first method).

If the first method were used, these five concepts would be extracted in the order of concepts C1, C2, C4, C3, and C5 (descending order of similarity for each concept) based on the similarity Sc calculated using equation (5). In contrast, with the second method, concepts are extracted in the order of C1, C2, C3, C4, and C5. In other words, even though the similarity Sc for concept C3 using equation (5) is smaller than the similarity Sc for concept C4 using equation (5), concept C3 is extracted as a higher-ranking concept than concept C4.

Furthermore, in the first method, based on the similarity Sc calculated using equation (5), lower-level concepts (lower-level concept vectors) of the first-ranked concept C1 (concept vector U110 corresponding to cluster G110) can also be extracted as relatively higher-level concepts. Specifically, not only the concept vector U110 (representing cluster G110) but also its lower-level concept vectors U111 and U112 (representing clusters G111 and G112 (see FIG. 11)) can be extracted as major concept vectors (concepts). This is because if concept vector U111 is similar to concept vector U110, the similarity Sc for concept vector U111 is likely to be as high as that for concept vector U110. Note that concept vector U111 is the concept vector representing lower-level cluster G111 of cluster G110, and concept vector U112 is the concept vector representing lower-level cluster G112 of cluster G110.

On the other hand, as described above, the second method searches for a concept vector that will most significantly increase the considered similarity component among concept vectors other than a concept vector already selected (considered) at a certain point in time (for example, concept vector U110). As a result, other concept vectors U that are relatively different (highly independent) from concept vector U110 are more likely to be extracted as major concept vectors (major concepts). Conversely, concept vectors U111 and U112, which represent lower-level clusters G111 and G112 of cluster G110, are less likely to be extracted as major concept vectors (major concepts). In the example of Figure 25, concept vectors U111 and U112 are not extracted as the top five major concept vectors (major concepts).

In this way, the second method makes it possible to extract concepts with high independence as main concepts. In other words, it is possible to extract main concepts with minimal overlap between concepts.

The display screen 600 in FIG. 25 also has buttons 601-605 in addition to the graph display area 609. Each button 601-605 displays the identifier of the corresponding cluster G (e.g., "G110"). The image processing device 30 and the user can uniquely identify (identify) each corresponding cluster of concepts C1-C5, etc., using this identifier (identification ID).

In addition, detailed information about each concept will be displayed as follows:

When the text portion of each concept C1 to C5 in the graph display area 609 of the display screen 600, or the buttons 601 to 605 located directly below each text portion, is pressed using a mouse or other means, a detailed information screen for the corresponding concept is displayed.

For example, when button 601 directly below concept C1 is pressed, detailed information screen 610 (see Figure 26) is displayed. Also, when button 602 directly below concept C1 is pressed, detailed information screen 620 (see Figure 27) is displayed. The same applies to the other buttons.

Figure 26 is a diagram showing a display screen 610 for detailed information about concept C1. Figure 27 is a diagram showing a display screen 620 for detailed information about concept C2. Similarly, display screens for detailed information (also referred to as detailed information screens) also exist for other concepts C3 to C5, etc. The following explanation will focus on the display screen 610 for detailed information about concept C1.

Display screen 610 has areas 611 to 614 and button 615.

Area 611 (also referred to as the upper area) is a display area for multiple images (concept constituent images) that make up the corresponding cluster G110 of the concept C1.

Area 612 (also referred to as the lower area) shows a heat map image indicating the areas (ignition areas) determined by the trained model 420 to be characteristic areas for each of the multiple images displayed in area 611. In area 612, a heat map image corresponding to each image is displayed directly below each image in the upper area 611. From the heat map in lower area 612, it can be seen that the trained model 420 is focusing on the shirt portion of each person.

Area 613 is a display area for a "concept visualization image" (described below). The concept visualization image is an image that visualizes the concept (concept vector) of cluster G110. The concept visualization image is a virtual input image that corresponds to the representative vector (concept vector) of cluster G110, and is generated using a feature visualization method (described below) or the like. The concept visualization image can also be described as an image that expresses the abstract concept (abstract concept) of cluster G110.

Area 614 is a display area for attribute information (concept name, etc.) of cluster G110. As will be described later, when the image processing device 30 receives operational input from the user (input of cluster attribute information (text input, etc.)), it stores the attribute information in the storage unit 32 and displays the attribute information (concept name, etc.) in area 614. Note that Figure 26 shows the display state after the operational input (character string "Concept C1: heavily patterned white shirt"). Before the operational input (before the character string "heavily patterned white shirt" was input), for example, only the character string "Concept C1" was displayed in area 614.

Button 615 accepts an instruction to display information about related clusters. Pressing button 615 displays a dendrogram like the one shown in the top row of Figure 11 (particularly around concept C1) and/or a Venn diagram like the one shown in the bottom row of Figure 11 (or Figure 13). This displays the existence and inclusion relationships of related clusters (specifically, same-level clusters G120 and G200, higher-level cluster G100, lower-level clusters G111 and G112, etc.) with respect to cluster G110 corresponding to concept C1. Furthermore, by clicking the corresponding position of each related cluster in the dendrogram or Venn diagram with the mouse, a detailed information display screen for that related cluster (e.g., same-level cluster G120) is displayed. Note that this is not a limitation; a display screen like the one shown in Figure 13 may also be displayed in response to the mouse click. In particular, on the display screen, each group of images constituting clusters related to cluster G110 (G110, G120, G100, G200, etc.) may be displayed together with a Venn diagram (a diagram showing the hierarchical relationships (inclusion relationships) of related clusters) near cluster G110.

The user can obtain detailed information about concept C1 from display screen 610 in Figure 26.

Specifically, the heat map in the lower region 612 indicates that the trained model 420 is focusing on the shirt portion of each person.

Furthermore, the multiple images in the upper area 611 make it possible to visually understand what images concept C1 (more specifically, its corresponding cluster) is made up of.

Furthermore, the concept visualization image in area 613 makes it possible to obtain an image that abstractly visualizes concept C1.

In addition, the attribute information in area 614 (specifically, the concept name and user notes, etc.) makes it possible to obtain the content of the concept through linguistic expression.

Here, the concept name of cluster G110 can be understood and input by the user as follows. The same applies to the concept names of other clusters.

Specifically, the user first views the detailed information screen 610 (Figure 26) regarding cluster G110. The detailed information screen 610 (more specifically, its upper area 611) displays the group of images that make up cluster G110 (see also Figure 13).

The user then presses button 615 to display a Venn diagram like the one shown in the lower part of Figure 11. Then, while referring to the Venn diagram, the user can display a detailed information screen (similar to the detailed information screen in Figure 26) about cluster G120, which is the same level as cluster G110, by, for example, pressing the position of cluster G120 within the Venn diagram. The detailed information screen about cluster G120 displays the group of images that make up cluster G120 (see Figure 13).

If necessary, a detailed information screen regarding cluster G100, which is higher than cluster G110, can also be displayed by performing the same operation.

Users can understand the characteristics of each cluster by comparing the images contained in these detailed information screens.

For example, by comparing cluster G110 with peer cluster G120, the characteristics of both clusters G110 and G120 can be understood. Specifically, it can be seen that cluster G120 is composed of images of people wearing "white shirts with little pattern," while cluster G110 is composed of images of people wearing "white shirts with a lot of pattern."

Furthermore, since the upper cluster G100 is a cluster that includes both clusters G110 and G120, it can be determined that it is a "white shirt with a pattern."

Furthermore, by comparing the higher-level cluster G100 with its peer cluster G200, the characteristics of both clusters G100 and G200 can be understood. Specifically, it can be seen that cluster G100 is composed of images of people wearing "patterned white shirts," while cluster G200 is composed of images of people wearing "light pink shirts." In other words, it can be seen that cluster G100 is "shirts other than light pink."

Through these considerations, the user can decide on "heavily patterned white shirt" as the concept name for cluster G110. The user then inputs the character string "heavily patterned white shirt" into area 614 on the detailed information screen 610. In response, the image processing device 30 accepts this input operation and registers "heavily patterned white shirt" in the memory unit 32 as the concept name for cluster G110. Furthermore, when the detailed information screen 610 is subsequently displayed, the concept name "heavily patterned white shirt" is displayed in area 614.

However, this is not limited to this, and the user may obtain all of this information by visually viewing a display screen such as that shown in Figure 13, and thereby understand the concept name of cluster G11, etc.

Furthermore, such concept name identification and input may be performed at this time (step S32) for only the user's desired cluster and its related clusters (i.e., only a relatively small number of clusters). However, this is not limited to this, and the process may be performed for all clusters, for example, immediately after step S23.

<Concept visualization image>
Here, the process of generating a concept visualization image using the Feature Visualization method will be described.

Figure 28 shows an overview of the process for generating a concept visualization image using the Feature Visualization method. The Feature Visualization method is a technique for investigating what inputs cause a specific firing (a specific intermediate output) in a certain intermediate layer of a neural network. Here, the final output from the output layer (i.e., feature vector F) is used instead of the intermediate output from the intermediate layer. In other words, it is investigated what input images cause the output vector (feature vector F) from the trained model 420 to occur.

Specifically, as shown in Figure 28, the input image that minimizes the distance d(F,U) between the output vector (i.e., feature vector F) output from the trained model 420 and the target vector (here, a certain concept vector U) is found as the concept visualization image. The feature vector F is a function (F(I)) of the image vector I of the input image. Note that here, the input image is represented by the image vector I (a vector in which the pixel values arranged two-dimensionally in the input image are rearranged one-dimensionally). Also, distances d(F,U) etc. are abbreviated as simply distance d etc.

More specifically, the process of finding the increment δI that minimizes the increment δd of this distance d using the internal parameters of the trained model 420 (acquired through machine learning) is repeated, thereby finding the input image vector I (a concept visualization image that reflects the characteristics of the concept vector U).

The increment δd of distance d is expressed by the following equation (8). Here, vector H is a constant vector (a vector with each component being a constant) calculated using the weights and activation functions of each hidden layer of the neural network. Note that equation (8) is derived by, for example, taking the sum of the relationship (expressed by gradient information, etc., of the trained model 420) between the increment δIk of value Ik (a scalar), which is the kth component of image vector I, and the increment δd of distance d(F(Ik),U) for all components k. This distance d(F(Ik),U) is the distance between the feature vector F(Ik) corresponding to the image having the kth component Ik and the concept vector U.

As shown in equation (8), the increment δd is expressed as the dot product of vector H and vector δI (the increment vector of vector I).

The process of minimizing the distance d is achieved by repeatedly setting the increment δd to the smallest value (the norm of which is the most negative value (also called the maximum negative value)). More specifically, this process is achieved by repeatedly finding the δI that minimizes the increment δd among the δIs with the same norm (magnitude). This process is equivalent to repeatedly finding the orientation of δI that can minimize the distance d (minimize the increment δd).

The direction of δI that minimizes the increment δd is the direction that minimizes the dot product with vector H (i.e., makes it the negative value of the maximum norm). Therefore, δI that minimizes the increment δd (makes it the negative value of the maximum norm) can be found as the vector δI with a specified norm (magnitude) that is oriented in the opposite direction to the constant vector H (δI when cosθ = -1). Here, the angle θ is the angle between the two vectors H and δI.

Then, the operation of adding such vector δI to vector I (see equation (9) below) is repeated many times (for example, thousands of times). Note that the initial value (initial vector) of vector I can be a vector equivalent to a random noise image (or an appropriate input image (such as an input image belonging to the cluster corresponding to concept vector U)). ε is a predetermined constant.

This process determines the input image vector I (i.e., the input image obtained by rearranging the input image vector I into a two-dimensional array) that minimizes the distance d between the feature vector F and the target vector (concept vector U).

For example, the "concept visualization image" for concept C2 (see area 623 in Figure 27) is an image generated using the Feature Visualization method based on the concept vector U400 (see the top row in Figure 20) of cluster G400 ("short white bottoms"). This concept visualization image has a part that resembles "short white bottoms (white shorts)" near the center of the image (near the area indicated by the thick dashed circle). This image shows that the concept vector U400 reflects the feature "short white bottoms."

Furthermore, the "concept visualization image" of concept C1 (see area 613 in Figure 26) is an image generated using the Feature Visualization method based on the concept vector U110 (see the top row in Figure 18) of cluster G110 ("white shirt with lots of patterns"). This concept visualization image has a part that resembles a "patterned shirt" in a location slightly above the center of the image. This image shows that the concept vector U400 reflects the feature of a "patterned (lots of patterns) shirt."

However, it is not necessarily easy for users to fully understand the content of each concept from only the "concept visualization image" of that concept. It is preferable that the concept visualization image be used as a supplementary tool.

<1-11. Effects of the embodiment>
According to the above embodiment, a plurality of hierarchical clusters are generated by performing a hierarchical clustering process on a plurality of feature vectors F (step S22 ( FIG. 4 )). Then, a vector (specifically, a concept vector U) corresponding to a specific cluster among the plurality of clusters is extracted as a concept of the specific cluster (step S23).

Therefore, it is possible to manage and understand concepts corresponding to specific hierarchical clusters using their representative vectors (concept vectors U). Furthermore, by performing analytical processing using these concept vectors U, it becomes possible to explain the basis for image similarity on a concept basis.

Furthermore, according to the above embodiment, two or more input images corresponding to a specific cluster (e.g., G110) among the multiple clusters generated by the hierarchical clustering process are determined and displayed as an image group representing the concept of the specific cluster (specific concept) (see Figures 13, 26, etc.). In other words, the two or more input images constituting the specific cluster are displayed as an image group corresponding to the concept of the specific cluster. For example, multiple input images corresponding to cluster G110 (see area 611 in Figure 26) are displayed as an image group corresponding to the concept of cluster G110. This image group is a group of mutually similar images (similar image group), and because it is an image group that expresses the concept of the specific cluster, it is also referred to as a similar image group for expressing a concept. This makes it possible to visually present to the user the concept of the specific cluster.

Furthermore, according to the above embodiment, a group of similar images for a specific cluster is displayed together with their respective heat maps (see area 612 (Figure 26) etc.). Therefore, it is possible to identify ignition areas (feature areas) within images related to a specific cluster using the heat map, and then conceptually grasp the features of that specific cluster.

Furthermore, a group of similar images for a specific cluster is displayed together with the concept visualization image (see area 613 (Figure 26) etc.). Therefore, it is possible to grasp the characteristics of a specific cluster conceptually (especially logically) through the group of similar images, while visually grasping the characteristics of the specific cluster through the concept visualization image.

Furthermore, in the above embodiment, multiple concepts (more specifically, multiple concept vectors U) corresponding to multiple hierarchical clusters are extracted. Therefore, it is possible to grasp the hierarchical relationships (parent-child relationships) and inclusion relationships between multiple concepts.

Furthermore, according to the above embodiment, when determining the similarity between two images (first image and second image) in step S31 (Figure 5), the contribution of multiple candidate concepts to the similarity between the first image and the second image is calculated. Therefore, it is possible to grasp the contribution (importance) of each concept to the similarity between the image pair.

In the above embodiment, the contribution level is calculated for each of a large number of concepts (candidate concepts), and the top few main concepts are determined by sorting, etc. based on the contribution level. However, this is not limited to this, and, for example, the contribution level to the similarity between two images may be calculated only for at least one concept (designated concept) designated by the user (based on their interests, etc.). In particular, the contribution level to the similarity between two images may be calculated only for the concept vector U (designated concept vector) corresponding to the designated concept. This allows the user to know to what extent the arbitrarily designated designated concept contributes (influences) to the similarity between the two images.

Furthermore, in the above embodiment, as shown in Figure 25, the basis for the similarity between two images (first image and second image) is displayed by the contribution (numerical value) of a predetermined number of top concepts (one to several). Therefore, it is possible to provide objective evaluation criteria regarding the basis for judgment.

In particular, the inner product of two projection vectors, F, corresponding to two images, projected onto a subspace (line) spanned by each concept's unique concept vector U, is calculated as the contribution Sc of each concept (see equation (5)). Therefore, the contribution of each concept to the similarity can be objectively presented.

Furthermore, Figure 25 shows that, among the multiple hierarchical concepts, the contribution Sc of concept C1 is higher than the contribution Sc of concept C2. This shows that, in the similarity determination by the image processing device 30, the characteristic of concept C1 ("white shirt with a lot of pattern") contributes more than the characteristic of concept C2 ("white, cropped bottoms").

In other words, it appears that the "similarity judgment" (a judgment that two images are similar to each other) is based on the fact that the person is wearing a "shirt" with specific characteristics, rather than the fact that the person is wearing "bottoms" with specific characteristics.

In addition, in Figure 25, concept C1 (cluster G110) is determined to be the concept with the highest contribution Sc. This also indicates that the contribution Sc of cluster G110 is greater than the contribution Sc of its peer cluster G120 (see Figure 11) (see the left-pointing arrow in Figure 12). In other words, the feature of cluster G110 ("white shirt with a lot of pattern") contributes more than the feature of cluster G120 ("white shirt with less pattern"). In other words, it can be seen that similarity is judged based on the feature of "white shirt with a lot of pattern" rather than the feature of "white shirt with less pattern." In other words, it can be seen that the "similarity judgment" is made by finding a particular feature in the many patterns (designs) on the shirts.

Furthermore, the fact that concept C1 (cluster G110) has the highest contribution indicates that the contribution Sc of cluster G110 is greater than the contribution Sc of its superordinate cluster G100 (see Figure 11) (see the downward arrow in Figure 12). In other words, this indicates that the feature of cluster G110 ("white shirt with many patterns"), which is a more detailed feature (lower-level feature) than the feature of cluster G100 ("white shirt with many patterns"), makes a large contribution. In other words, it can be seen that similarity is judged not simply based on "white shirt with many patterns" but also on the fact that it is "white shirt with many patterns."

The fact that concept C1 (cluster G110) has the highest contribution also indicates that the contribution Sc of cluster G110 is greater than the contribution Sc of each of its subordinate clusters G111 and G112 (see FIG. 11) (see the upward arrows in FIG. 12). That is, it is shown that the feature of cluster G110 ("white shirt with a lot of patterns") contributes more than the feature of cluster G111 ("white shirt with a lot of patterns (and the pattern is linear)") (see also FIG. 13). It is also shown that the feature of cluster G110 ("white shirt with a lot of patterns") contributes more than the feature of cluster G112 ("white shirt with a lot of patterns (and the pattern is curved)"). In other words, it can be seen that similarity is determined without taking into account the type of pattern (straight or curved).

<1-12. Modification of the first embodiment>
In the above embodiment, a vector (concept vector U) corresponding to a specific cluster is extracted as the concept of the cluster, but the present invention is not limited to this.

For example, in the first embodiment, a "subspace" corresponding to a specific cluster may be extracted as a concept of the specific cluster (a specific subspace representing the concept). In other words, a "subspace" corresponding to a specific cluster may be extracted as a concept representation of the specific cluster (a concept representation using a subspace).

In particular, the subspace spanned by the concept vector U of the specific cluster itself (specifically, the straight line (the straight line including the concept vector U) after projection using the projection matrix P corresponding to the concept vector U) may be extracted as the concept of the specific cluster. For example, the subspace spanned by the concept vector U100 (see the bottom row of Figure 11) of cluster G100 itself (i.e., the straight line including the concept vector U100) may be extracted as the concept of cluster G100.

Alternatively, a subspace spanned by a predetermined number of concept vectors U corresponding to a predetermined number (e.g., two) of lower-level clusters contained in a specific cluster may be extracted as the concept (concept representation) of that specific cluster. For example, a subspace (plane) spanned by two concept vectors U110 and U120 (see Figure 18) corresponding to two lower-level clusters G110 and G120 of cluster G100 (see the bottom of Figure 11) may be extracted as the concept of cluster G100.

In particular, in the first embodiment, when judging the similarity between two images, a specific concept vector U is extracted as the basis concept for judging that the two images are similar to each other ("similarity judgment"), but this is not limited to this. For example, a subspace (straight line) spanned by the specific concept vector U may be extracted as the basis concept for the similarity judgment.

The same applies to the second embodiment described below. In particular, in the second embodiment, a specific concept vector U is extracted (described below) as the basis concept for determining that two images are not similar to each other ("dissimilarity determination"), but this is not limiting. For example, a subspace (straight line) spanned by the specific concept vector U may be extracted as the basis concept for the "dissimilarity determination."

2. Second embodiment
In the first embodiment, a process for explaining the basis for a determination that two images are similar to each other ("similarity determination") (see FIG. 5) has been described as an example of a process for generating explanatory information regarding an inference result. However, the present invention is not limited to this. For example, a process for explaining the basis for a determination that two images are not similar to each other ("dissimilarity determination") (see FIG. 6) may be performed as a process for generating explanatory information regarding an inference result. In other words, the basis for "similarity" may be a basis for a dissimilarity determination, rather than a basis for a similarity determination. In the second embodiment, such an aspect will be described. The following description will focus on differences from the first embodiment. In the second embodiment, step S40 (see FIG. 6) is executed instead of step S30 (see FIG. 5).

Figure 29 is a conceptual diagram showing the process of determining the basis for a "dissimilarity judgment" (step S41). For convenience of illustration, Figure 29 abstracts the entire space into a three-dimensional space. The concept vector U has the same orientation as the z axis, and a straight line extending in the z direction indicates the subspace spanned by the concept vector U. The xy plane (the plane where z = 0) indicates the orthogonal complement of the subspace spanned by the concept vector U. Here, the two feature vectors q and g are feature vectors F corresponding to two images that are dissimilar to each other. Figure 29 shows that the orientations of the two feature vectors F are significantly different from each other, i.e., the two images are dissimilar to each other.

As shown in Figure 29, we consider the orthogonal complement of the subspace (straight line) indicated by a specific concept vector U corresponding to a specific concept (specific cluster) (here, a plane perpendicular to the concept vector U).

The projection vector (RF) (specifically, Rq, Rg) obtained by projecting feature vector F onto the orthogonal complementary space contains components of feature vector F excluding those explained by concept vector U (components (residual components) not yet explained by concept vector U). The fact that two such projection vectors (RF) are close to each other means that the components (residual components) not yet explained are similar (and, by extension, that the residual components do not have a significant impact on the similarity judgment (dissimilarity judgment) of two feature vectors F). Conversely, this means that the components already explained by concept vector U have a significant impact on the dissimilarity judgment of two feature vectors F.

Using this characteristic, in the second embodiment, if the distance between the projection vectors Rq, Rg obtained by projecting two feature vectors q, g onto the orthogonal complementary space is small (especially very close), the specific concept (or its corresponding specific concept vector U) is extracted as the "basis concept for the dissimilarity judgment." More specifically, a specific concept vector U for which the distance between the projection vectors Rq, Rg onto the orthogonal complementary space (plane, etc.) of the subspace (subspace (straight line) spanned by the specific concept vector U) is relatively small (compared to other concept vectors U) is extracted as the "basis concept for the dissimilarity judgment." A more detailed explanation is provided below.

When the projection matrix onto the subspace indicated by the specific concept vector U corresponding to a specific concept (specific cluster) is expressed as matrix P (see equation (3), etc.), the projection matrix R onto the orthogonal complement of that subspace is expressed as (1-P). Therefore, the projection vector obtained by projecting feature vector q onto the orthogonal complement is vector ((1-P)q), and the projection vector obtained by projecting feature vector g onto the orthogonal complement is vector ((1-P)g) (see Figure 29).

The distance between these projection vectors is |((1-P)q)-((1-P)g)|, and the square of this distance is defined as the evaluation value Sd1 (see equation (10)).

As described above, matrix P (see equation (3), etc.) is a matrix that differs depending on matrix B and, ultimately, concept vector U. Therefore, finding matrix P that minimizes evaluation value Sd1 is equivalent to finding concept vector U that minimizes evaluation value Sd1. Therefore, the concept vector U that minimizes evaluation value Sd1 is extracted as the "basis concept for dissimilarity judgment."

Furthermore, minimizing the evaluation value Sd1 is equivalent to "maximizing" the evaluation value Sd2 in equation (11). A simpler evaluation value Sd2 may also be used.

In this way, the evaluation value Sd2 (or Sd1) is used to obtain the main concept vector U as the "basis concept for the dissimilarity judgment."

In this case, a method similar to the first method described above (also referred to as the third method) or a method similar to the second method (also referred to as the fourth method) may be used.

Specifically, in the third method, the controller 31 first calculates the evaluation value Sd2 (the above-mentioned formula (11)) for a single concept vector U for each of the multiple candidate concept vectors U. Next, the controller 31 sorts the multiple concepts corresponding to the multiple candidate concept vectors U in descending order of their evaluation value Sd2. The controller 31 then determines the top few concepts as the "basis concepts for determining dissimilarity" between the two images (particularly their main concepts).

On the other hand, in the fourth method, a search process is repeatedly performed to search for a concept that maximizes evaluation value Sd2 (or minimizes evaluation value Sd1) among concepts (unselected concepts) other than those already selected (considered) at a certain point in time (at the time of the i-th iteration). This search process is repeatedly performed until a predetermined termination condition is met (for example, until a predetermined number of main concepts have been determined). Note that with each iteration, the number of selected concept vectors U increases by one, and accordingly, the rank (order) of the projection matrix P increases by one.

Each method can also calculate an evaluation value D, as shown in equation (12). This evaluation value D refers to the components (residual components related to the similarity evaluation) that have not yet been considered by the n concepts (concept vector U) selected up to a certain point in time regarding the "dissimilar" judgment. Specifically, the evaluation value D is calculated as the value (also called the residual) obtained by subtracting the degree of "dissimilar" judgment (dissimilarity judgment) explained by the n concepts (Sd2/2) from the degree of "dissimilar" judgment (dissimilarity judgment) (1-q·g). However, the evaluation value Sd2 is adjusted by multiplying it by a coefficient 1/2 so that the evaluation value D becomes zero when P becomes a unit matrix (full rank) (a β×β unit matrix) (i.e., when the concept (concept vector U) corresponding to the entire space is considered).

In the fourth method, this evaluation value D may be used (instead of evaluation value Sd2, etc.). That is, a search process may be performed to search for a concept that minimizes evaluation value D among concepts (unselected concepts) other than those already selected (considered) at a certain point in time. This search process is repeatedly performed until a predetermined termination condition is met (for example, until a predetermined number of main concepts have been determined). With each repetition, the number of selected concept vectors U increases by one, and accordingly, the rank (order) of the projection matrix P increases by one, and the evaluation value D gradually decreases.

Using the third or fourth method described above, a specific concept vector U from among the multiple concept vectors U corresponding to multiple clusters is extracted as the "basis concept for dissimilarity judgment." This specific concept vector U is the concept vector that minimizes the distance between the projection vectors (Rq) and (Rg) of each feature vector q and g onto the orthogonal complement of the subspace corresponding to it (that specific concept vector U).

In other words, of the multiple subspaces corresponding to the multiple clusters, the concept vector U that spans the subspace that makes the distance between the projection vectors ((1-P)q) and ((1-P)g) of the two feature vectors q and g onto the orthogonal complementary space relatively smaller (than other subspaces) is extracted as the "basis concept for dissimilarity judgment."

In other words, among the multiple concepts corresponding to the multiple clusters, the concept that, if removed, would result in the two images being judged to be similar to each other is extracted as the "basis concept for the dissimilarity judgment."

As a result of the above processing, the following "basis concepts for dissimilarity judgments" can be extracted:

For example, if an image of a person wearing a light pink shirt and an image of a person wearing a plain wine-red shirt are determined to be dissimilar, concept vector U200, concept vector U310, etc. may be extracted as the "basis concept for the dissimilarity determination" (see Figure 11, etc.).

Alternatively, (although not shown in Figure 11, etc.) if there is a higher-level cluster of "blue shirt" and a lower-level cluster of "blue and checked shirt," an image of a person wearing a blue shirt and an image of a person wearing a blue and checked shirt may be determined to be dissimilar. The concept that can be extracted to support this dissimilarity determination may include the concept vector of the "blue shirt" cluster and the concept vector of the "blue and checked shirt" cluster.

In this way, the concept C (concept vector U) that best reflects the characteristics of both images being compared can be extracted as the main concept.

Furthermore, in step S42, a display similar to that shown in Figures 25 and 26 is displayed. However, a display is displayed to explain the basis concept of the "dissimilarity judgment" rather than the basis concept of the "similarity judgment."

Specifically, for example, similar to Figure 25, the evaluation values Sd2 (or Sd1) of the top few (e.g., 2 to 5) main concepts are displayed in a graph. The evaluation values D (residuals) may also be displayed. Furthermore, similar to Figure 26, etc., detailed information about each main concept (constituent images of the corresponding cluster, heat map images, concept visualization images, etc.) is displayed.

In addition, similar to Figure 26, detailed information about each of the top few main concepts (basis concepts for "dissimilarity judgment") is displayed.

By performing the above processing, when an image pair is dissimilar, it is possible to understand the basis for determining that they are dissimilar (the basis for determining dissimilarity).

In this second embodiment, a specific concept vector U is extracted as the "basis concept for dissimilarity judgment." However, as mentioned above, this is not limited to this, and for example, a subspace (straight line) spanned by the specific concept vector U may also be extracted as the "basis concept for dissimilarity judgment."

<3. Modifications, etc.>
Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described contents.

For example, the processing of subphase PH3a (step S20) does not necessarily have to be performed after second phase PH2 (step S12), but may instead be performed immediately after first phase PH1 (step S11).

In addition, in each of the above embodiments, in sub-phase PH3a, multiple input images 211 used in machine learning are used as input images 210 for the trained model 420. However, this is not limited to this, and multiple input images other than the multiple input images 211 (for example, input image 213) may be used. However, it is preferable to use the multiple input images 211 used in training the trained model 420 as input images 210 for the trained model 420 rather than using the multiple input images other than the multiple input images (input image 213, etc.). This is because it is believed that a relatively accurate distribution is obtained as the distribution of feature vectors 250 for the multiple input images 210 (output distribution from the trained model 420).

Furthermore, while the above embodiments illustrate examples in which the present invention is applied to person recognition, the present invention is not limited to this. For example, the present invention may also be applied to product recognition. Alternatively, the present invention may also be applied to lesion recognition (lesion detection), etc.

Furthermore, while the above embodiments illustrate examples in which the present invention is applied to metric learning (distance learning), the present invention is not limited to this and may also be applied to class classification learning, etc.

For example, this may be applied to class classification learning using a trained model 420 that includes a feature extraction layer (configured as a CNN, etc.) that extracts image features and a fully connected layer that performs classification processing, etc. based on the features extracted in the feature extraction layer. In particular, the intermediate output vector from the trained model 420 (the vector output from the feature extraction layer and input to the fully connected layer next to the feature extraction layer) may be acquired as the feature vector F in feature space. In other words, the feature vector F output from the trained model 420 is not limited to the vector ultimately output from the trained model 420 (final output), but may also be a vector intermediately output from the trained model 420 (intermediate output). Then, hierarchical clustering processing, concept vector extraction processing, etc. may be performed on the feature vector F.

1 Image processing system 30 Image processing device (information processing device)
31 Controller 210, 211, 213, 215 Person image (input image)
250, 251, 253, 255, F Feature vector 400, 410, 420 Learning model 501 Separation plane 600, 610, 620 Display screen C Concept G Cluster U Concept vector

Claims (8)

a control unit that acquires a plurality of feature vectors output from a machine-learned learning model corresponding to input of a plurality of input images to the learning model, generates a plurality of hierarchical clusters by executing a hierarchical clustering process on the plurality of feature vectors, and extracts a subspace or vector corresponding to a specific cluster from the plurality of clusters as a concept of the specific cluster;
An information processing device comprising: The information processing device of claim 1, wherein the control unit extracts a representative vector related to the specific cluster as a concept of the specific cluster. The information processing device described in claim 2, characterized in that the control unit generates a concept visualization image, which is a virtual input image corresponding to the representative vector and which visualizes the concept of the specific cluster, based on the representative vector and the learning model, and displays the concept visualization image on a display unit. a receiving unit that receives input of attribute information of the specific cluster;
The information processing device according to claim 1 , further comprising: The information processing device described in any one of claims 1 to 4, characterized in that when determining the similarity between the first image and the second image based on a first feature vector output from the learning model in response to an input of a first image to the learning model and a second feature vector output from the learning model in response to an input of a second image to the learning model, the control unit calculates the contribution of at least one concept among a plurality of concepts corresponding respectively to the plurality of clusters to the determination of the similarity between the first image and the second image. The information processing device of any one of claims 1 to 4, characterized in that when the control unit determines the similarity between the first image and the second image based on a first feature vector output from the learning model in response to a first image input to the learning model and a second feature vector output from the learning model in response to a second image input to the learning model, the control unit extracts, from among a plurality of subspaces corresponding to the plurality of clusters, a subspace that relatively reduces the distance between the projection vectors of the first and second feature vectors onto the orthogonal complementary space, or a vector spanning the subspace, as a concept that serves as a basis for determining that the first image and the second image are dissimilar to each other. a control unit that acquires a plurality of feature vectors output from a machine-learned learning model corresponding to input of a plurality of input images to the learning model, generates a plurality of hierarchical clusters by executing a hierarchical clustering process on the plurality of feature vectors, and determines two or more input images corresponding to a specific cluster among the plurality of clusters as a group of images representing the concept of the specific cluster;
An information processing device comprising: a) obtaining a plurality of feature vectors output from a machine-learned learning model in response to input of a plurality of input images to the learning model;
b) generating a plurality of hierarchical clusters by performing a hierarchical clustering process on the plurality of feature vectors;
c) extracting a subspace or vector corresponding to a specific cluster from among the plurality of clusters as a concept of the specific cluster;
An information processing method comprising: