JP7775435B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Technology related to image search using a Visual-Semantic Embedding (VSE) model has been disclosed in the past. For example, in image search using a VSE model, a specified search keyword and the image to be searched are each converted into feature vectors, and images corresponding to the similarity between the two feature vectors are output as search results (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2022-180941

However, conventional technology only searches for images with similar feature vectors to the search keywords, and does not consider providing thumbnail images that reflect the user's search intent.

The present invention has been made in consideration of the above, and aims to provide an information processing device, information processing method, and information processing program that can provide thumbnail images that reflect the user's search intent.

To solve the above-mentioned problems and achieve the objectives, the information processing device of the present invention includes a search unit that searches for search targets based on a search query specified by a user, a selection unit that selects from each of the search targets, among multiple candidate images linked to the search targets included in the search results by the search unit, the candidate image that has the highest relationship to the search query, and a provision unit that provides the candidate image selected by the selection unit as a thumbnail image of the search target.

This invention makes it possible to provide thumbnail images that reflect the user's search intent.

FIG. 1 is a diagram illustrating an example of the configuration of a provision system according to an embodiment. FIG. 2 is a diagram illustrating an example of information processing according to the embodiment. FIG. 3 is an explanatory diagram of a distributed representation space according to the embodiment. FIG. 4 is a diagram illustrating an outline of a process for generating thumbnail images according to the embodiment. FIG. 5 is a block diagram illustrating an example of the configuration of an information processing apparatus according to the embodiment. FIG. 6 is a diagram illustrating an example of information stored in a user information storage unit according to the embodiment. FIG. 7 is a diagram illustrating an example of information stored in a product information storage unit according to the embodiment. FIG. 8 is a diagram illustrating an example of a method for collecting learning data according to the embodiment. FIG. 9 is a diagram illustrating an example of learning data according to the embodiment. FIG. 10 is a diagram illustrating an example of a composite image according to the embodiment. FIG. 11 is a flowchart illustrating an example of a processing procedure for information processing according to the embodiment. FIG. 12 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the information processing apparatus according to the embodiment.

Below, detailed explanations will be given with reference to the drawings of the information processing device, information processing method, and information processing program (hereinafter referred to as "embodiments") according to the present application. Note that the information processing device, information processing method, and information processing program according to the present application are not limited to these embodiments. In addition, the following explanations will be given for the case where the search target is a product sold in an online shopping mall.

[Embodiment]
1.1 System
First, the provision system shown in Fig. 1 will be described. Fig. 1 is a diagram showing an example of the configuration of the provision system according to the embodiment. As shown in Fig. 1, the provision system includes an information processing device 1 and a user terminal 10. The information processing device 1 and the user terminal 10 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via a wired or wireless connection.

The information processing device 1 provides an e-commerce service that offers (searches, sells, etc.) products including clothing (also known as fashion items, including clothes, underwear, footwear (also known as shoes), headwear (such as caps and hats), bags, and accessories), cosmetics (also known as fashion items), etc. The information processing device 1 also provides a coordination service that accepts posts from users of content (images, videos, articles, etc.) showing outfits using multiple pieces of clothing, content showing makeup using one or more cosmetics, and provides (searches, distributes, etc.) the content to other users. For example, the information processing device 1 can be realized by a server device, a cloud system, etc.

In addition, as described below, the information processing device 1 provides various services to users using a pre-trained distributed representation space. Images and keywords (including tags, etc.) are projected into the distributed representation space according to the embodiment, and the closeness between images, images and keywords, and keywords is projected. The closer two elements are semantically, the closer they are trained to be in the distributed representation space (and vice versa). In other words, the similarity between images, images and keywords, and keywords can be measured as distance in the distributed representation space. Closeness in the distributed representation space according to the embodiment includes, for example, closeness based on Euclidean distance and closeness based on cosine distance.

The information processing device 1 may also function as a web server that provides websites related to services. The information processing device 1 may also be a device that distributes information to the user terminal 10 to be displayed in applications related to various services installed on the user terminal 10. The information processing device 1 may also be a device that distributes the application data itself.

The information processing device 1 may also function as a distribution device that distributes control information to the user terminal 10. Here, the control information is written in, for example, a script language such as JavaScript (registered trademark) or a style sheet language such as CSS (Cascading Style Sheets). Note that the application itself distributed from the information processing device 1 may also be considered as control information.

The user terminal 10 is an information processing device used by a user. The user terminal 10 may be realized, for example, as a smartphone, tablet terminal, notebook PC (Personal Computer), desktop PC, mobile phone, or PDA (Personal Digital Assistant). The user terminal 10 also displays information distributed by the information processing device 1 or a server device that provides a specified service, using a web browser or application.

1.2 Information Processing
Next, information processing according to an embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of information processing according to an embodiment. As shown in Fig. 2, the information processing device 1 functions as a web server that provides information about fashion items, and first accepts a search query specified by a user from a user terminal 10 (step S1).

The information processing device 1 searches for products based on the search query (step S2). For example, products tagged with keywords corresponding to the search query are output as search results.

Next, the information processing device 1 selects a thumbnail image for each product included in the search results (step S3). More specifically, the information processing device 1 selects, from among multiple candidate images associated with each product, the candidate image that has the highest relationship with the search query for each product included in the search results.

For example, the information processing device 1 selects thumbnail images for each product based on the similarity between the search query (keywords) and each candidate image in the distributed representation space. More specifically, the search query indicates the user's search intent, and the thumbnail image clicked (viewed) by the user who specified the search query (it may be a thumbnail image that led to a purchase rather than just a click) can be considered an image that better reflects the user's search intent. For example, the information processing device 1 can project each search query, candidate image, etc. into the distributed representation space using technologies related to distributed representation spaces, such as VSE (Visual-Semantic Embedding) and CLIP (Contrastive Language-Image Pre-training).

For example, the information processing device 1 can associate a search query specified by a user with a thumbnail image clicked by the user and use the associated data as learning data, thereby selecting thumbnail images that are highly related to the search query, i.e., that reflect the user's search intent. Furthermore, for example, the information processing device 1 can associate a search query specified by a user with a thumbnail image not clicked by the user and use the associated data as negative example learning data. Note that in this embodiment, in addition to the search query and candidate images, thumbnail images are selected from the candidate images based on the attributes of the user who specified the search query.

Here, the distributed representation space according to the embodiment will be explained using Figure 3. Figure 3 is an explanatory diagram of the distributed representation space according to the embodiment. Note that here, the process of selecting a thumbnail image from multiple candidate images linked to one product will be explained.

As shown in FIG. 3, the information processing device 1 first embeds multiple candidate images linked to products into a distributed representation space (step S11), and then embeds the search query and user attributes into the distributed representation space (steps S12 and S13). Note that user attributes include, for example, information about the user's age, gender, body type, foot shape, skin color, residence, and the user's click log, search log (search history), purchase log (purchase history), and browsing log (browsing history). User attributes may also include, for example, information about the age, gender, body type, foot shape, skin color, residence, click log, search log, purchase log, and browsing log of users who have a predetermined relationship with the user (including a parent-child relationship, etc.).

In the example shown in Figure 3, the feature vectors of each candidate image are indicated as "image vectors Vi1 to Vi5," the feature vector of the search query is indicated as "query vector Vq," and the feature vector of the user attribute is indicated as "user vector Vu." Note that hereinafter, when there is no need to distinguish between image vectors Vi1 to Vi5, they will be referred to as "image vector Vi."

For example, the information processing device 1 generates a ranking of the image vectors Vi1 to Vi5 according to their positional relationship with the query vector Vq and the user vector Vu. For example, the information processing device 1 generates a ranking in descending order, starting with the image vector Vi having the smallest area of the space (triangle) connecting each of the image vectors Vi1 to Vi5 with the query vector Vq and the user vector Vu. Note that the information processing device 1 may also generate a ranking of the image vectors Vi1 to Vi5 in descending order of similarity to either the query vector Vq or the user vector Vu.

Then, the information processing device 1 selects the candidate image ranked first as the thumbnail image. That is, the information processing device 1 selects, from among the image vectors Vi, the candidate image corresponding to the image vector Vi that is similar to the query vector Vq and the user vector Vu, respectively, as the thumbnail image.

In other words, the information processing device 1 selects a candidate image to be used as a thumbnail image from among the candidate images associated with the product, depending on the search query and user attributes. More specifically, from the image vectors Vi corresponding to each candidate image, it selects an image vector Vi that is similar to a query vector Vq corresponding to the search query and a user vector Vu corresponding to the user attributes, and selects this as the thumbnail image.

This allows the information processing device 1 to select a thumbnail image that reflects the user's search intent from multiple candidate images linked to the product. In the example shown in Figure 3, the area of the space connecting the image vector Vi4 with the query vector Vq and the user vector Vu is the smallest, indicating that the candidate image corresponding to the image vector Vi4 will be selected as the thumbnail image.

When the information processing device 1 selects a thumbnail image from each candidate image for each product included in the search results of step S2, it provides the selected thumbnail image to the user terminal 10, as shown in FIG. 2 (step S4).

The information processing device 1 can also generate thumbnail images for products for which there is no image vector Vi that satisfies the thumbnail image selection conditions from each image vector Vi in the distributed representation space. That is, for products for which there is no appropriate thumbnail image, the information processing device 1 generates a new thumbnail image based on the search query. Note that the information processing device 1 may register multiple candidate images generated in advance, and if there is no thumbnail image that satisfies the selection conditions, select and provide a thumbnail image from the candidate images registered in advance.

Figure 4 is a schematic diagram of the thumbnail image generation process according to the embodiment. For example, the information processing device 1 generates a composite image Ic, which is a thumbnail image synthesized by combining a product image It of a product for which an appropriate thumbnail image is not available and a composite image Is corresponding to the search query.

For example, the composite images Is include candidate images linked to other products and posted images (e.g., outfit images) posted by each user to a coordination service. Posted images include photos tagged with fashion items worn by each user.

For example, the information processing device 1 extracts each synthesis image Is based on the search query from candidate images linked to other products and posted images posted by each user. For example, the information processing device 1 extracts a synthesis image Is from each image based on the similarity between the image vector Vi of each image in the distributed representation space and the query vector Vq of the search query.

Next, the information processing device 1 uses various image generation AIs to synthesize a composite image Ic by combining the product image It and the composite image Is.

In this case, for example, the information processing device 1 repeatedly generates the composite image Ic until the area of the space connecting the image vector Vi of the composite image Ic, the query vector Vq of the search query, and the user vector Vu of the user attributes becomes equal to or less than a threshold, thereby generating a composite image Ic that is adapted to the search query specified by the user.

In this way, the information processing device 1 automatically generates thumbnail images for products that do not have thumbnail images corresponding to the search query. This allows the information processing device 1 to provide appropriate thumbnail images that correspond to the search query entered by the user.

2. Information Processing Device
Next, a configuration example of the information processing device 1 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration example of the information processing device 1 according to the embodiment. As shown in Fig. 5, the information processing device 1 includes a communication unit 2, a storage unit 3, and a control unit 4. Note that the information processing device 1 may also include an input unit (e.g., a keyboard, a mouse, etc.) that accepts various operations from an administrator who uses the information processing device 1, and a display unit (e.g., a liquid crystal display, etc.) that displays various information.

The communication unit 2 is realized, for example, by a network interface card (NIC). The communication unit 2 is connected to a communication network such as 4G (4th Generation) or 5G (5th Generation) via a wired or wireless connection, and transmits and receives information to and from each of the user terminals 10 and the like via the communication network.

The storage unit 3 is realized, for example, by a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. The storage unit 3 has a user information storage unit 31, a product information storage unit 32, and a model storage unit 33.

The user information storage unit 31 stores information about each user. Figure 6 is a diagram showing an example of information stored in the user information storage unit 31 according to an embodiment. As shown in Figure 6, the user information storage unit 31 stores information items such as "user ID" and "user information" in association with each other.

"User ID" indicates identification information for identifying a user. "User Information" indicates user information. In the example shown in Figure 6, conceptual information such as "User Information #1" and "User Information #2" is stored in "User Information," but in reality, information on user attributes such as age and gender, and identification information for products (items) owned by the user are stored.

The product information storage unit 32 stores product information. The product information storage unit 32 contains information about products sold in the online shopping mall provided by the information processing device 1. Figure 7 is a diagram showing an example of information stored in the product information storage unit 32 according to an embodiment.

As shown in FIG. 7, the product information storage unit 32 stores information such as "product ID" and "image information" in association with each other. "Product ID" indicates identification information for identifying a product. "Image information" is image information of a product identified by the corresponding product ID. In this embodiment, multiple candidate images are stored as image information for each product.

The model storage unit 33 stores a model. For example, the model is a learning model that projects each image, search query (keyword), and user information into the same distributed representation space. For example, the learning model is a model that has been trained by the learning unit 42, which will be described later, and the learning model can be, for example, a VSE model.

Next, the control unit 4 will be described. The control unit 4 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or MPU (Micro Processing Unit) executing various programs (equivalent to examples of information processing programs) stored in a storage device within the information processing device 1 using RAM as a working area. The control unit 4 is also, for example, a controller, and is realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

As shown in FIG. 5, the control unit 4 includes an acquisition unit 41, a learning unit 42, a search unit 43, a selection unit 44, a generation unit 45, and a provision unit 46, and realizes or executes the information processing functions and actions described below. Note that the internal configuration of the control unit 4 is not limited to the configuration shown in FIG. 5, and may be other configurations that perform the information processing described below. Furthermore, the connection relationships between the processing units in the control unit 4 are not limited to the connection relationships shown in FIG. 5, and may be other connection relationships.

The acquisition unit 41 acquires various types of information. For example, the acquisition unit 41 acquires user information from the user terminal 10 and registers it in the user information storage unit 31. The acquisition unit 41 also acquires product information from the terminals of store owners (not shown) in the online shopping mall and registers it in the product information storage unit 32.

The acquisition unit 41 also accepts posts from users, such as content (images, videos, articles, etc.) showing outfits using multiple pieces of clothing, and content showing makeup using one or more cosmetics. For example, such content is provided (searched, distributed, etc.) to other users through a platform such as a social networking site. Note that such a platform may be a platform provided by the information processing device 1.

The learning unit 42 learns the model (learning model) stored in the model storage unit 33. First, a method for collecting learning data will be described. Figure 8 is a diagram showing an example of a method for collecting learning data according to an embodiment.

As shown in FIG. 8, for example, the information processing device 1 provides an annotation app to the user terminal 10 and acquires learning data for the learning model through the annotation app. For example, as shown in FIG. 8, the annotation app displays a search query to be learned and multiple images.

Then, the user selects an image that matches the search query through the annotation app, and the information processing device 1 acquires learning data that links the search query with the image selected by the user. In the example shown in FIG. 8, the search query is "golf," and the user selects an image that is associated with "golf." Note that the search query may also be "golf knit sweater," and the user may select an image that is associated with "golf knit sweater."

The information processing device 1 can collect learning data of images corresponding to search queries by having the user select images corresponding to each search query through the annotation app.

Note that the annotation app is not limited to the example shown in Figure 8; it may display two images simultaneously, change the layout of multiple images depending on the user, display multiple images in a circular arrangement, or display multiple images in a constantly changing layout. This eliminates the bias that images are more likely to be selected depending on their layout.

In addition to the above example, the information processing device 1 may also collect learning data from, for example, log information for each user in an online shopping mall. Figure 9 is a diagram showing an example of learning data according to an embodiment.

As shown in Figure 9, the learning data includes items such as "user," "search query," "clicked image," and "unclicked image." "User" indicates the user's attributes. "Search query" indicates the search query specified by the corresponding user. The learning data may also include information such as the time it took the user to select an image, images displayed at the same time, and the position (display location) of the image selected by the user.

"Clicked images" refers to images of products included in the search results for the corresponding search query that the corresponding user clicked on (e.g., thumbnail images), while "unclicked images" refers to images of products included in the search results for the corresponding search query that the corresponding user did not click on.

In other words, in this case, it is possible to collect learning data based on the logs of each user in the online shopping mall. After collecting this learning data, the learning unit 42 trains the learning model based on the collected learning data. Specifically, the learning unit 42 trains the learning model so that, in the distributed representation space, the feature vector of the search query and the feature vector of the image linked to the search query become similar feature vectors in the distributed representation space.

More specifically, the learning unit 42 trains the learning model so that similar images have similar feature vectors, similar search queries (keywords) have similar feature vectors, and search queries that serve as learning data and images linked to those search queries have similar feature vectors.

The learning unit 42 also trains a learning model that can project each user's user attributes into a distributed representation space. For example, user attributes include so-called demographic attributes such as the user's age, gender, and residence, as well as the user's log in the online shopping mall, which includes search history, browsing history, purchase history, etc.

The learning unit 42 also generates a pseudo-ranking of thumbnail images that express search intent using a method such as Serial Rank, and evaluates the learning model. The learning unit 42 also trains the learning model so that similar user attributes become similar feature vectors in the distributed representation space. Note that the learning data is not limited to the above example, and may also include logs of each user in the coordination service. User logs in the coordination service include browsing history, search history, posting history, product purchase history through the coordination service, etc.

The search unit 43 searches for a search target (product) based on a search query specified by the user. Specifically, the search unit 43 acquires a user ID and search query from the user terminal 10, and searches the product information storage unit 32 for products that match the search query specified by the user. The search unit 43 then passes the user ID and product information included in the search results to the selection unit 44.

The selection unit 44 selects, for each product, a candidate image that has the highest relationship with the search query from among multiple candidate images linked to the product included in the search results by the search unit 43. Specifically, the selection unit 44 uses a learning model to project the candidate image, user attributes, and search query for each product into a distributed representation space, and calculates the area of the space connecting each image vector Vi, user vector Vu, and query vector Vq in the distributed representation space (see Figure 2).

For example, the selection unit 44 selects as the thumbnail image the candidate image corresponding to the image vector Vi with the smallest area of such space. At this time, for example, if the area of the space connecting the image vector Vi, the user vector Vu, and the query vector Vq is equal to or greater than a threshold, i.e., if there is no thumbnail image corresponding to the search query, the selection unit 44 passes information about the product to the generation unit 45 and instructs it to generate a thumbnail image.

The generation unit 45 generates thumbnail images of products included in the search results by the search unit 43 based on the relationship between the search query and the image. The generation unit 45 generates thumbnail images for products for which the selection unit 44 has instructed the generation of thumbnail images, i.e., for products for which multiple candidate images associated with the product do not satisfy the thumbnail image selection conditions.

For example, the generation unit 45 generates, as thumbnail images, coordinated images that include products included in the search results by the search unit 43. The coordinated images are images that use multiple pieces of clothing. For example, the generation unit 45 generates coordinated images by combining multiple candidate images that are linked to products corresponding to each category, such as tops, bottoms, and shoes.

First, the generation unit 45 extracts images to be synthesized based on the search query. For example, the generation unit 45 uses a learning model to project product images of products sold in the online shopping mall and images posted by users into a distributed representation space, and extracts images in the distributed representation space that are similar to the query vector Vq of the search query as images to be synthesized.

For example, based on the product image It for which a thumbnail image is to be generated and the search query, the generation unit 45 extracts, as an image to be synthesized, an image in the distributed representation space that is similar to the search query and that suits the product. For example, the generation unit 45 extracts a plurality of different patterns of images to be synthesized depending on the bone structure, style, season, etc. of the model appearing in the posted image. Note that the generation unit 45 may also extract a plurality of different patterns of images to be synthesized depending on the compatibility between the product (e.g., a top) and the model appearing in the posted image, the compatibility with other products (e.g., a skirt) worn by the model appearing in the posted image, and the compatibility with the background appearing in the posted image.

When extracting images to be composited from images posted by users, the generation unit 45 may, for example, exclude posted images that include products from brands that are not sold in the online shopping mall from the candidates for images to be composited.

For example, posted images are tagged with products (items) that each user is wearing, and the generation unit 45 can use these tags to identify posted images that include products that are not available in the online shopping mall.

In addition, the generation unit 45 may generate a composite image by overwriting products that are not sold in the online shopping mall, for example, when generating a composite image using the images to be composited, without excluding all posted images from the list of images to be composited. In other words, by not including products from brands that are not sold in the online shopping mall in thumbnail images, it is possible to prevent copyright infringement, for example.

The generation unit 45 may prepare images corresponding to each search query in advance and extract (select) images to be synthesized from among these images.

Once the generation unit 45 has finished extracting the composite image, it uses image generation AI to generate a composite image from the product image It of the target product and the composite image. For example, the generation unit 45 generates a collage image by arranging each product image (each item image) on the composite image, and then generates a natural-looking thumbnail image from the collage image. In this case, the generation unit 45 may, for example, extract cut-out images of the items used, background, etc. from each composite image, and then generate the thumbnail image.

The generation unit 45 may also change the instructions given to the image generation AI depending on the user and the search query. For example, if the user is male, the generation unit 45 may instruct the image generation AI to generate an image in which a male is the model.

Also, suppose the user who originated the search is a man, and the search query is "women, dress, stylish," and the man is searching for a gift for a woman. In this case, the generation unit 45 instructs the image generation AI to generate an image of a woman as the model, based on the search query "women." Note that whether the user or the search query is given priority may be determined in advance, for example, on a rule-based basis.

Furthermore, when generating a composite image, the generation unit 45 may, for example, include in the composite image products that the user has previously purchased. Note that products that the user has previously purchased can be identified, for example, from the user's purchase history in an online shopping mall, but they may also be identified from posted images (coordination images) posted by the user.

Figure 10 is a diagram showing an example of a composite image according to an embodiment. For example, as shown in Figure 10, the generation unit 45 extracts product images of products previously purchased by the user from the user information storage unit 31, and generates a composite image including the extracted product images. Note that at this time, the generation unit 45 may generate a composite image including product images of products selected from the user's log. In this case, the generation unit 45 may extract product images related to products registered as favorites by the user, products previously viewed, or products purchased by users similar to the target user. In the example shown in Figure 10, the products previously purchased by the user are a bag and a necklace. The generation unit 45 generates a composite image Ic by further using product image Ib corresponding to the bag and product image In corresponding to the necklace as images to be combined.

In other words, in this case, the composite image Ic is generated by coordinating a product image (a dress in Figure 10) that corresponds to the search query specified by the user, in other words, a product image that matches the user's search intent, with a bag or necklace that the user has previously purchased.

In other words, the composite image Ic can show the user an example of how the dress currently being sold looks when worn with a bag or necklace that the user currently owns.

In this way, the generation unit 45 can generate a composite image Ic that is individually optimized for the user, which is expected to increase the user's purchasing motivation. Note that, for example, the generation unit 45 may generate the composite image Ic by combining products specified by the user. In this case, the user may select a product from products that they have purchased in the past, or, for example, may select an image of the product that they have photographed.

The providing unit 46 provides the user with the thumbnail image selected by the selecting unit 44. At this time, for products for which no thumbnail image was selected by the selecting unit 44, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as the thumbnail image.

This allows the providing unit 46 to provide thumbnail images that reflect the user's search intent.

3. Processing Flow
Next, a processing procedure executed by the information processing device 1 according to the embodiment will be described with reference to Fig. 11. Fig. 11 is a flowchart showing an example of the processing procedure of information processing according to the embodiment. The processing described below is repeatedly executed by the information processing device 1 at a predetermined cycle each time a search query is acquired.

As shown in FIG. 11, the information processing device 1 accepts a search query from a user (step S101). Next, the information processing device 1 searches for products that match the search query (step S102).

Next, the information processing device 1 embeds multiple candidate images linked to the products included in the search results, the search query, and user attributes into a distributed representation space (step S103). Next, the information processing device 1 selects thumbnail images from the candidate images based on each feature vector in the distributed representation space (step S104).

Next, the information processing device 1 determines whether there are any products whose candidate images do not satisfy the selection conditions (step S105), and if there are any products that do not satisfy the selection conditions (step S105: Yes), it generates thumbnail images of those products (step S106).

Furthermore, if thumbnail images have been selected from all products (step S105; No), the information processing device 1 proceeds to processing in step S107. Then, the information processing device 1 provides the thumbnail images to the user (step S107) and ends the processing.

4. Modifications
In the above-described embodiment, the information processing device 1 provides thumbnail images related to products. However, the present invention is not limited to this. For example, the present invention may be applied when providing thumbnail images for various web services such as social networking sites and video sites. Furthermore, the information processing device 1 may provide the present invention when providing thumbnail images related to outfits in a coordination service. For example, the selection unit may select, from among multiple candidate images (posted images) associated with outfits included in the search results by the search unit, a candidate image that has the highest relationship with the search query from each outfit and provide it as a thumbnail image.

In the above-described embodiment, the providing unit 46 provides the composite image Ic generated by the generating unit 45 as a thumbnail image for a product for which no thumbnail image was selected by the selecting unit 44, but this is not limited to this. Even if there is no candidate image that reflects the user's search intent among the multiple candidate images linked to the product, if posted images showing outfits using the product have been posted to a coordination site and among them there is a posted image that reflects the user's search intent, that posted image may be provided as a thumbnail image. Specifically, the information processing device 1 embeds multiple posted images showing outfits using the products included in the search results, the search query, and user attributes into a distributed representation space. Next, the information processing device 1 selects a thumbnail image from the posted images based on each feature vector in the distributed representation space.

In addition, instead of embedding all posted images showing outfits using products included in the search results into the distributed representation space, the information processing device 1 may search the outfit coordination site in advance using the same search query and embed the posted images included in the search results into the distributed representation space.

In other words, the search targets of the search unit 43 may be expanded to include, for example, coordinated outfit images posted on a coordination service, in addition to products sold in the online shopping mall. By expanding the search targets in this way, it is possible to provide search results that reflect the user's search intent.

5. Effects
The information processing device 1 according to the embodiment includes a search unit 43 that searches for products based on a search query specified by a user, a selection unit 44 that selects, from each product, a candidate image that has the highest relationship with the search query from among multiple candidate images linked to the products included in the search results by the search unit 43, and a provision unit that provides the candidate image selected by the selection unit 44 as a thumbnail image of the product.

The selection unit 44 also projects the search query and candidate images into the distributed representation space, and selects candidate images in the distributed representation space that are similar to the search query. The selection unit 44 also selects candidate images based on the user information of the user who specified the search query, and selects the candidate image that has the smallest area of the space connecting the user information, search query, and candidate images in the distributed representation space onto which the user information is projected.

In addition, during the learning phase, the selection unit 44 selects candidate images using a model trained to ensure that candidate images that have been pre-annotated as images corresponding to the search query and the search query are similar vectors in the distributed representation space.

In addition, during the learning phase, the selection unit 44 selects candidate images using a model trained to ensure that the search query specified by the user and the image selected by the user when using the search query are similar vectors in the distributed representation space.

The information processing device 1 also includes a generation unit 45 that generates a composite image using a composite image that satisfies the search conditions for a product that does not satisfy the selection conditions of the candidate image by the selection unit 44, and the provision unit 46 provides the composite image generated by the generation unit 45 as a thumbnail image.

By using any one or a combination of the above processes, the information processing device according to the present application can provide thumbnail images that reflect the user's search intent.

6. Hardware Configuration
The information processing device 1 according to the embodiment described above is realized by, for example, a computer 1000 configured as shown in Fig. 12. Fig. 12 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information processing device according to the embodiment. The computer 1000 has a CPU 1100, a RAM 1200, a ROM 1300, a HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls each component. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

HDD 1400 stores programs executed by CPU 1100 and data used by such programs. Communication interface 1500 receives data from other devices via network (communication network) N and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via network N.

The CPU 1100 controls output devices such as displays and printers, and input devices such as keyboards and mice (in Figure 12, the output devices and input devices are collectively referred to as "input/output devices") via the input/output interface 1600. The CPU 1100 acquires data from the input devices via the input/output interface 1600. The CPU 1100 also outputs generated data to the output devices via the input/output interface 1600.

Media interface 1700 reads programs or data stored on recording medium 1800 and provides them to CPU 1100 via RAM 1200. CPU 1100 loads the programs from recording medium 1800 onto RAM 1200 via media interface 1700 and executes the loaded programs. Recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

For example, when the computer 1000 functions as an information processing device according to an embodiment, the CPU 1100 of the computer 1000 executes programs loaded onto the RAM 1200 to realize the functions of the control unit 4. The CPU 1100 of the computer 1000 reads and executes these programs from the recording medium 1800, but as another example, the CPU 1100 may obtain these programs from another device via the network N.

[7. Other]
Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown.

Furthermore, the components of each device shown in the figure are functional concepts and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

For example, the information processing device described above may be implemented using multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

Furthermore, the above-described embodiments and variations can be combined as appropriate to the extent that the processing content is not contradictory.

Furthermore, the "parts" (sections, modules, units) mentioned above can be read as "means" or "circuits." For example, an acquisition unit can be read as an acquisition means or an acquisition circuit.

Reference Signs List 1 Information Processing Device 2 Communication Unit 3 Storage Unit 4 Control Unit 10 User Terminal 31 User Information Storage Unit 32 Product Information Storage Unit 33 Model Storage Unit 41 Acquisition Unit 42 Learning Unit 43 Search Unit 44 Selection Unit 45 Generation Unit 46 Provision Unit

Claims (3)

a search unit that searches for search targets based on a search query specified by a user;
a selection unit that selects, from among a plurality of candidate images linked to the search target included in the search results by the search unit, the candidate image that has the highest relationship with the search query and the user information of the user from each of the search targets;
a providing unit that provides the candidate image selected by the selecting unit as a thumbnail image to be searched. 1. A computer-implemented information processing method, comprising:
a search step of searching for a search target based on a search query specified by a user;
a selection step of selecting, from each of the search targets included in the search results by the search step, a candidate image that has the highest relationship between the search query and the user information of the user, from among a plurality of candidate images linked to the search targets;
a providing step of providing the candidate image selected in the selection step as a thumbnail image to be searched. a search procedure for searching for a search target based on a search query specified by a user;
a selection step of selecting, from each of the search targets, a candidate image that has the highest relationship between the search query and the user information of the user from among a plurality of candidate images associated with the search targets included in the search results of the search step;
a providing step of providing the candidate image selected by the selection step as a thumbnail image to be searched for.