JP7362076B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

2. Description of the Related Art Conventionally, in order to enable images to be searched from text information, there has been known a technique for generating text information (caption, tag, etc.) indicating the content of an image. For example, in order to make it possible to search for a person included in an image, a person area including the person is identified in the image, and the person area is divided into a plurality of partial areas. A technique is known in which a query element is generated in each of a plurality of partial areas, and a search query for searching for a person is generated by combining the query elements of the plurality of partial areas.

Japanese Patent Application Publication No. 2016-162414 JP2019-219988A

There is a need for technology that improves image search accuracy.

The information processing device according to the present application obtains region information regarding an object region, which is a divided region including an object having a structure, out of an image divided into regions using a segmentation technology, and information about an object region estimated using a pose estimation technology. an attribute information extraction unit that extracts attribute information regarding attributes of the object based on structural information regarding the structure of the object; and a sentence generated based on the attribute information extracted by the attribute information extraction unit, The image forming apparatus includes a model generation unit that generates a VSE (Visual-Semantic Embedding) model that is trained to associate a text indicating the content of an image with the image and embed the image in a common space.

FIG. 1 is a diagram for explaining an overview of information processing according to an embodiment. FIG. 2 is a diagram for explaining a VSE (Visual-Semantic Embedding) model and a concept classifier according to the embodiment. FIG. 3 is a diagram illustrating a configuration example of an information processing device according to an embodiment. FIG. 4 is a diagram showing an attribute information extraction processing procedure and a VSE model generation processing procedure according to the embodiment. FIG. 5 is a diagram showing an information processing procedure according to the embodiment. FIG. 6 is a diagram showing an information processing procedure according to a modification. FIG. 7 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An information processing apparatus, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail below with reference to the drawings. Note that the information processing apparatus, information processing method, and information processing program according to the present application are not limited to this embodiment. Further, in each of the embodiments below, the same parts are given the same reference numerals, and redundant explanations will be omitted.

(Embodiment)
[1. Introduction]
2. Description of the Related Art Conventionally, techniques for extracting attribute information regarding attributes of an object (for example, a person) included in an image from an image are known. For example, a pose estimation technology (hereinafter also referred to as pose estimation technology) that extracts pose information regarding the pose of a person included in an image is known. Furthermore, a segmentation technique is known that extracts information regarding a region of a person or a region of clothing included in an image.

However, although segmentation techniques can accurately extract regions of people and clothing included in images, they cannot extract the postures of people included in images. Further, although posture estimation techniques can accurately extract the posture of a person included in an image, it is difficult to accurately extract a region of a person or a region of clothing included in an image.

In contrast, the information processing device 100 according to one embodiment combines segmentation technology and posture estimation technology to determine the region of the person included in the image, the region of clothing, and the posture of the person included in the image. It can be extracted with high precision. FIG. 1 is a diagram for explaining an overview of information processing according to an embodiment. The information processing shown in FIG. 1 is performed by an information processing apparatus 100 (see FIG. 3), which will be described later.

As shown in FIG. 1, an information processing apparatus 100 according to an embodiment divides an image into regions using a segmentation technique, and provides human region information regarding a human region, which is a divided region including a person, of the divided image. Extract. Here, the information processing apparatus 100 may extract a person area including the entire person from the head to the toes for each person by dividing the image into regions using a segmentation technique. In addition, the information processing device 100 divides the image into regions using segmentation technology, thereby dividing the image into regions that include each body part of the person (for example, hair, face, and each fashion item worn by the person). Body part area information regarding a body part area that is a region may be extracted for each body part.

Here, the segmentation technology is a technology that uses an image as input, divides regions at the pixel level, and attaches labels to the regions. Segmentation techniques can be roughly divided into three types based on the meaning of labeling. Semantic segmentation is a technique that classifies all pixels on an image into classes. Mantic segmentation divides an image into regions for each type of object. Instance segmentation is a technology that divides the region of each object and recognizes the type of object. Instance segmentation divides an image into regions for each object. Furthermore, panoptic segmentation is a technology that combines semantic segmentation and instance segmentation. Panoptic segmentation performs instance segmentation on objects such as people, animals, and cars (also called the countable class, also called the thing class), and performs instance segmentation on objects such as people, animals, and cars (also called the countable class, or thing class), and performs instance segmentation on backgrounds such as the sky, roads, and lawns (also called the uncountable class, or stuff class). ) is a technology that performs semantic segmentation. The information processing apparatus 100 may segment the image into regions using semantic segmentation, instance segmentation, or panoptic segmentation techniques. Further, the information processing apparatus 100 may segment the image into regions by combining techniques of semantic segmentation, instance segmentation, and panoptic segmentation.

Furthermore, the information processing apparatus 100 extracts skeletal information regarding the skeleton of a person included in the image using a posture estimation technique. Specifically, the information processing apparatus 100 may extract skeletal information from an image using any known technique related to pose estimation. For example, the information processing device 100 may extract skeletal information using a posture estimation technique that estimates the posture (skeleton) of a person or animal from a video or still image using a deep learning model called a posture estimation model. . Note that when multiple people are included in one image, the information processing device 100 detects feature points for the multiple people through posture estimation processing, and estimates skeletal information regarding the skeletons of the multiple people. You may. The information processing apparatus 100 can precisely estimate the body part of a person in an image by using posture estimation technology.

For example, OpenPose (“OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, Zhe Cao et al., 2018) is known as an example of a pose estimation model. OpenPose is a posture estimation model that detects feature points (also referred to as key points) indicating the physical characteristics of a person included in an image and estimates the posture of the person by connecting the feature points. For example, OpenPose estimates joint points indicating the positions of the joints of a person's body as feature points of the person's body included in an image, and connects the joint points to generate a skeletal model indicating the skeleton of the person's body. is detected as a person's posture. Furthermore, for example, among posture estimation models of the type that detect feature points, there are models that can estimate 30 types of body parts of a person in an image. Specifically, when a pose estimation model of the type that detects feature points is used, the body parts of the person in the image are the head, eyes (right, left), ears (right, left), nose, neck, and shoulders. (right, center, left), elbow (right, left), spine, wrist (right, left), hand (right, left), thumb (right, left), fingertip (right, left), waist (right, center) , left), knee (right, left), ankle (right, left), and foot (right, left).

Furthermore, DensePose (reference URL: http://openaccess.thecvf.com/content_cvpr_2018/html/Guler_DensePose_Dense_Human_CVPR_2018_paper.html) is known as an example of a posture estimation model. DensePose is a posture estimation model that detects a human region of a person in a two-dimensional image and generates a three-dimensional body surface model corresponding to the detected human region. More specifically, DensePose takes an RGB image as input and estimates the UV coordinates of the three-dimensional surface of a person in the RGB image. Using DensePose, it is possible to estimate the UV coordinates of the 3D body surface from the human region in the 2D image, so each human body part (part of the person's body) in the human region in the 2D image can be precisely estimated. It can be estimated that Using DensePose, it is possible to estimate 24 types of body parts of a person in an image. Specifically, when DensePose is used, from an RGB image, the body parts of the person in the image are: head (left, right), neck, torso, arms (left, right/top, front/front, back), Legs (left, right/thigh, calf/front, back), hands (left, right), and feet (left, right) can be identified.

Next, the information processing apparatus 100 extracts person attribute information regarding the attributes of the person included in the image based on the extracted person area information and skeleton information. For example, the information processing apparatus 100 extracts posture information regarding a person's posture based on extracted skeletal information, as an example of person attribute information. Further, the information processing apparatus 100 extracts, as an example of person attribute information, motion information regarding a person's motion, based on a pattern of temporal changes in the extracted skeletal information, for example. For example, the information processing apparatus 100 extracts information that the person included in the image has changed his or her posture from a standing state to a sitting state. Furthermore, as an example of person attribute information, the information processing device 100 extracts information regarding clothing worn by a person, hairstyle, and facial expression based on information on body part regions of each extracted body part, for example. . For example, the information processing device 100 extracts information that the person included in the image is wearing blue clothes. Subsequently, the information processing apparatus 100 generates a sentence indicating the content of the image based on the extracted person attribute information. For example, the information processing device 100 determines the content of the image based on information that the person included in the image has changed his or her posture from a standing state to a sitting state and information that the person's clothing is blue. As an example of a sentence indicating ``A person wearing blue clothes sat on a red sofa in a restaurant.'' may be generated.

As described above, the information processing apparatus 100 according to an embodiment uses person area information regarding a person area, which is a divided area including a person, of an image divided into areas using a segmentation technique, and a posture estimation technique. Personal attribute information regarding the attributes of the person is extracted based on skeletal information regarding the person's skeleton estimated using . Further, the information processing device 100 according to an embodiment learns to associate a sentence generated based on the extracted person attribute information and indicates the content of the image with the image and embed it in a common space. generate a VSE model.

In this way, the information processing device 100 according to one embodiment can accurately determine the region of a person and the clothing region included in an image, and the posture of a person included in an image by combining the segmentation technology and the posture estimation technology. Can be extracted well. Thereby, the information processing device 100 can hierarchically decompose the human region and the components of the human region (for example, joint position information of each joint, body part region, fashion item region, etc.) included in the image. can. Furthermore, the information processing apparatus 100 can appropriately extract person attribute information regarding attributes of a person from the person region and skeletal information based on the hierarchical relationship of the decomposed components. Further, the information processing apparatus 100 can appropriately generate a sentence indicating the content of the image based on the person attribute information appropriately extracted from the image. Therefore, the information processing apparatus 100 according to one embodiment can improve image search accuracy.

Note that in the above example, the information processing apparatus 100 extracts person area information from an image using segmentation technology, and extracts person attribute information based on the extracted person area information. Not limited to. For example, the information processing apparatus 100 uses segmentation technology to extract a human region from an image. Subsequently, the information processing apparatus 100 may extract, from the human region, skeletal information regarding the skeleton of the person included in the human region, using a posture estimation technique. In this way, the information processing device 100 uses the information of the human region extracted using the segmentation technique to estimate the skeletal information using the posture estimation technique, thereby estimating the posture information more accurately. can do. Furthermore, the information processing apparatus 100 may use information that is a combination of image information, person area information, and skeletal information in order to estimate person attribute information. Thereby, the information processing apparatus 100 can estimate person attribute information with higher accuracy.

In addition, in recent years, in order to efficiently organize and manage a huge amount of video content, technology has been introduced that automatically analyzes the content of images and automatically generates and adds text information (captions, tags, etc.) that indicates the content of the images. It has been known. This makes it possible to search for images based on text information. Here, the "image" in this specification may be a moving image such as a video, or each scene (still image) included in the video.

Against this background, research is known that compares two typical methods for retrieving images from text information. Specifically, (1) a trained machine learning model (hereinafter also referred to as a concept classifier) that has previously learned detection targets (hereinafter also referred to as concepts) such as objects, people, scenes, and actions contained in images; (2) image features indicating the characteristics of the image and linguistic features indicating the characteristics of the linguistic expression expressing the content of the image in language are associated and embedded. There is a known study that compared a method of searching for images that match a query sentence using a common space using 2020 2nd International Conference on Intelligent Medicine and Image Processing, April 2020, Pages 103‐107, https://doi.org/10.1145/3399637.3399657).

According to the above research, it has been shown that the method using a concept classifier shown in (1) and the method using VSE shown in (2) are complementary. Therefore, the inventor of the present invention proposes a technique for improving image search accuracy by integrating the method using a concept classifier shown in (1) and the method based on VSE shown in (2). Specifically, the information processing apparatus 100 according to one embodiment extracts an appropriate concept from an image using a concept classifier, and searches again for an image similar to the extracted concept using a VSE model. For example, the information processing device 100 may extract concepts using a VSE model (hereinafter sometimes referred to as a VSE model) that has been learned to associate visual graph information and character graph information and embed them in a common space. Search again for images similar to . As a result, the information processing apparatus 100 according to an embodiment can, for example, extract an appropriate concept from among concepts (for example, implicit concepts) that are not explicitly stated in the query sentence received from the user. You can re-search for images similar to the concept you created. Therefore, according to an embodiment of the present invention, image search accuracy can be improved.

From here, the VSE model and concept classifier according to the embodiment will be explained using FIG. 2. FIG. 2 is a diagram for explaining the VSE model and concept classifier according to the embodiment.

The left side of FIG. 2 shows an example of the VSE model according to the embodiment. For example, in the VSE model shown on the left side of Figure 2, when searching for a sentence from an image, the image is input to the VSE model, and a feature vector (hereinafter also referred to as image feature vector) corresponding to the image feature amount indicating the characteristics of the image is used. ) is generated. Next, the VSE model maps the generated image feature vector to a common space with a feature vector corresponding to a sentence feature indicating the sentence feature (hereinafter also referred to as sentence feature vector), and then maps the image feature vector to a common space. A sentence corresponding to a feature vector of a sentence similar to the vector is set as a search result. For example, the VSE model outputs, as a search result, a sentence corresponding to a sentence feature vector in which the degree of similarity between the image feature vector and the sentence feature vector exceeds a predetermined threshold.

Further, the VSE model can extract image features from an image. The VSE model may be implemented using any known technique that can extract image features from images. For example, the VSE model may include a Convolutional Neural Network (CNN). Then, the VSE model may use CNN to extract image features from the image. For example, VSE models include ResNet (Residual Network) (Kaiming He et al., 2015), AlexNet (Krizhevsky et al., 2012), VGGNet (Simonyan et al., 2014), which were developed for object recognition. GoogLeNet (Szegedy et al., 2014), SENet (Squeeze-and-Excitation Networks) (Jie Hu et al., 2018)), EfficientNet (Tan et al., 2019), or ZFNet (Matthew et al., 2013) may be provided. Then, the VSE model may extract image features from the image using ResNet, AlexNet, VGGNet, GoogleLeNet, SENet, EfficientNet, or ZFNet. In addition, for example, the VSE model is based on Faster R-CNN (Shaoqing Ren et al., 2015), YOLO (You Look Only Onse) (Joseph Redmon et al., 2015), which was developed for object detection, or SSD (Single Shot MultiBox Detector) (Wei Liu., 2015). The VSE model may then extract image features from the image using Faster R-CNN, YOLO, or SSD.

Furthermore, when searching for an image from a sentence, the VSE model shown on the left side of FIG. 2 inputs the sentence to the VSE model and generates a feature vector of the sentence. Next, the VSE model maps the generated sentence feature vector to a common space with the image feature vector, and then sets the image corresponding to the image feature vector similar to the sentence feature vector as a search result. For example, the VSE model outputs, as a search result, an image corresponding to an image feature vector in which the degree of similarity between the sentence feature vector and the image feature vector exceeds a predetermined threshold.

Further, the VSE model extracts linguistic features indicating the characteristics of the linguistic expression from the linguistic expression (sentence, phrase, keyword, etc.). The VSE model may be realized by any known technique that can extract linguistic features. For example, the VSE model may include a recurrent neural network (RNN). Then, the VSE model may extract linguistic features from the linguistic expression using RNN. Further, for example, the VSE model may include a GRU (Gated Recurrent Unit) or a LSTM (Long Short Term Memory). Then, the VSE model may use GRU or LSTM to extract linguistic features from the linguistic expression. For example, VSE models include Transformer (Ashish Vaswani et al., 2017), Transformer-based BERT (Bidirectional Encoder Representations from Transformers), GPT-3 (Generative Pre-Training 3), or T5 (Text-to-Text Transfer Transformer). Then, the VSE model may extract linguistic features from the linguistic expression using Transformer, BERT, GPT-3, or T5.

The right side of FIG. 2 shows an example of the output result of the concept classifier according to the embodiment. The concept classifier is a trained machine learning model that is trained to output a concept similarity indicating the degree of similarity between a concept included in the image and the image when an image including a concept is input. Here, the concepts that the concept classifier learns (also called detection targets) include not only objects and people included in the image, but also scenes of the image and people and animals included in the image. It includes concepts such as actions (running, sitting, etc.). For example, the output result of the concept classifier shown on the right side of Figure 2 is that when an image of a man standing next to a motorcycle is input to the concept classifier, the color of the man's hair in the image, the color of the man's hair, The system detects objects such as clothing worn by men, body parts of men, the color of motorcycles, mountains and oceans in the background, and red bridges, and outputs the class (category) of the objects. Although not shown on the right side of FIG. 2, the concept classifier outputs the class of the concept included in the image as well as the probability that the concept included in the image corresponds to the class (category) of the concept. In this way, when an image including a concept is input, the concept classifier detects the concept included in the image and estimates the class of the detected concept. That is, when an image including a concept is input, the concept classifier outputs the probability that each concept included in the image corresponds to the estimated class of each concept as the concept similarity.

[2. Configuration of information processing device]
Next, the configuration of the information processing apparatus according to the embodiment will be described using FIG. 3. FIG. 3 is a diagram illustrating a configuration example of an information processing device according to an embodiment. As shown in FIG. 3, the information processing device 100 includes a communication section 110, a storage section 120, an input section 130, an output section 140, and a control section 150.

(Communication Department 110)
The communication unit 110 is realized by, for example, a NIC (Network Interface Card), a modem chip, an antenna module, and the like. Further, the communication unit 110 is connected to a network N (not shown) by wire or wirelessly.

(Storage unit 120)
The storage unit 120 is realized by, for example, a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. For example, the storage unit 120 stores a video pool that is a database of images that are multiple videos or scenes included in each of the multiple videos. Furthermore, the storage unit 120 stores a caption pool that is a database of character strings that are a plurality of sentences or each text included in each of a plurality of sentences.

(Input section 130)
The input unit 130 receives inputs for various operations from the user. For example, the input unit 130 may receive various operations from the user via a display screen (for example, the output unit 140) using a touch panel function. Further, the input unit 130 may accept various operations from buttons provided on the information processing device 100 or a keyboard or mouse connected to the information processing device 100. For example, the input unit 130 may accept input of a query sentence from the user. Furthermore, the input unit 130 may accept input of a query image from the user.

(Output section 140)
The output unit 140 is a display screen realized by, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display, and is a display device for displaying various information. The output unit 140 displays various information under the control of the control unit 150. Note that when a touch panel is employed in the information processing apparatus 100, the input section 130 and the output section 140 are integrated. Furthermore, in the following description, the output unit 140 may be referred to as a screen.

(Control unit 150)
The control unit 150 is a controller that controls the information processing device 100 using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array). This is achieved by executing various programs (corresponding to an example of an information processing program) stored in an internal storage device using a storage area such as a RAM as a work area. In the example shown in FIG. 3, the control unit 150 includes an attribute information extraction unit 151, a model generation unit 152, a reception unit 153, an acquisition unit 154, a search unit 155, an extraction unit 156, and a generation unit 157. .

(Attribute information extraction unit 151)
The attribute information extraction unit 151 extracts region information regarding an object region, which is a divided region including an object having a structure, from an image divided into regions using a segmentation technique. For example, the attribute information extraction unit 151 extracts person area information regarding a person area, which is a divided area including a person, as an example of an object having a structure. For example, the attribute information extraction unit 151 may extract item area information regarding an item area, which is a divided area including each fashion item worn by a person, as an example of person area information.

Further, the attribute information extraction unit 151 may extract body part area information regarding a body part area, which is a divided area including each body part of a person, as an example of person area information. For example, the attribute information extraction unit 151 may extract information regarding a person's hair region as an example of body part region information. Further, the attribute information extraction unit 151 may extract information regarding a person's face area as an example of body part area information.

Furthermore, the attribute information extraction unit 151 extracts structural information regarding the structure of the object estimated using the pose estimation technique. For example, the attribute information extraction unit 151 extracts, as an example of structural information, skeletal information regarding a human skeleton estimated using a posture estimation technique. For example, the attribute information extraction unit 151 extracts joint position information of each joint of a person as an example of skeletal information.

Furthermore, after extracting the area information and structure information, the attribute information extraction unit 151 extracts attribute information regarding the attributes of the object based on the extracted area information and structure information. For example, the attribute information extraction unit 151 extracts person attribute information regarding the attributes of the person included in the image based on the extracted person area information and skeleton information.

For example, the attribute information extraction unit 151 extracts posture information regarding the posture of the person based on joint position information of each joint of the extracted person as an example of the person attribute information. For example, the attribute information extraction unit 151 may extract posture information such as whether the person is standing, sitting, or raising their right hand.

Further, the attribute information extraction unit 151 extracts motion information regarding the motion of the person, as an example of the person attribute information, based on a pattern of temporal changes in joint position information of each joint of the extracted person. For example, the attribute information extraction unit 151 may extract motion information indicating that the person changed their posture from a standing state to a sitting state. Furthermore, the attribute information extraction unit 151 may extract motion information such as whether the person is running, walking, or waving their right hand.

Further, the attribute information extraction unit 151 extracts item attribute information regarding the attributes of a fashion item worn by the person, as an example of person attribute information, based on the extracted item area information. For example, the attribute information extraction unit 151 extracts information indicating the type, color, shape, material, etc. of fashion items such as clothes, shoes, hats, and bags worn by a person as an example of item attribute information. good.

Further, the attribute information extraction unit 151 extracts body part attribute information regarding attributes of a person's body parts as an example of person attribute information. For example, the attribute information extraction unit 151 extracts information regarding a person's hairstyle based on information regarding the hair region of the person as an example of body part attribute information. For example, the attribute information extraction unit 151 may extract information indicating the type, color, shape, and hair type of a person's hairstyle. Further, the attribute information extraction unit 151 extracts information regarding the facial expression of the person based on information regarding the facial area of the person as an example of body part area information. For example, the attribute information extraction unit 151 may extract information indicating the type of facial expression of the person (smiling, angry, etc.), the degree of facial expression (slightly smiling, very angry, etc.), and the like.

In the above example, the attribute information extraction unit 151 extracts person attribute information related to the attributes of a person included in an image as an example of an object having a structure. Attribute information of any object other than a person may be extracted as long as the object has the following. For example, the attribute information extraction unit 151 may extract attribute information regarding the attributes of a vehicle that is formed by combining parts such as a door portion, a window glass portion, and a tire portion, as an example of an object having a structure. For example, the attribute information extraction unit 151 extracts, as region information, vehicle region information regarding a vehicle region that is a divided region including an object that is a vehicle, and structure information that is estimated using a posture estimation technique. Based on the skeleton information regarding the skeleton of the vehicle, vehicle attribute information regarding the attributes of the vehicle is extracted as attribute information.

(Model generation unit 152)
The model generation unit 152 is trained to associate sentences with images, which are generated based on the attribute information extracted by the attribute information extraction unit 151, and which indicate the content of images, and embed them in a common space. Generate a VSE model. Specifically, the model generation unit 152 associates the image with a sentence that is generated based on the person attribute information extracted by the attribute information extraction unit 151 and that indicates the content of the image, and stores it in a common space. Generate a VSE model trained to embed.

For example, the model generation unit 152 generates a person attribute feature vector indicating the characteristics of the person attribute information extracted by the attribute information extraction unit 151. For example, the model generation unit 152 generates a human region feature vector indicating the characteristics of the extracted human region using a segmentation technique. Furthermore, the model generation unit 152 generates a skeletal information feature vector indicating the characteristics of the person's skeletal information estimated using the posture estimation technique. Subsequently, the model generation unit 152 generates a person attribute feature vector by connecting the generated person region feature vector and skeleton information feature vector. Furthermore, the model generation unit 152 generates a text feature vector that indicates the features of the text that indicates the content of the generated image. After generating a person attribute feature vector and a text feature vector, the model generation unit 152 generates a VSE model by learning the VSE model so that the generated person attribute feature vector and text feature vector are similar in a common space. .

(Reception Department 153)
The reception unit 153 receives a query sentence input by a user. For example, the reception unit 153 accepts a query sentence input by a user via the input unit 130. Here, the query sentence in this specification does not have to be a complete sentence, and may be a keyword or a phrase, for example. In the following, the query sentence that the reception unit 153 first receives from the user will be referred to as a "first query sentence." For example, the reception unit 153 accepts the phrase "person in a car" from the user as an example of the first query sentence.

(Acquisition unit 154)
The acquisition unit 154 acquires images from the video pool. For example, when the reception unit 153 receives the first query sentence, the acquisition unit 154 refers to the storage unit 120 and acquires from the video pool an image that is a plurality of videos or each scene included in each of the plurality of videos. For example, the acquisition unit 154 acquires N images #11 to #1N (N is a natural number) from the video pool.

(Search unit 155)
The search unit 155 uses the VSE model generated by the model generation unit 152 to search for a first image related to the first query sentence received by the reception unit 153. Specifically, when the acquisition unit 154 acquires an image, the search unit 155 inputs the set of the first query sentence accepted by the reception unit 153 and the image acquired by the acquisition unit 154 to the VSE model. For example, the search unit 155 inputs a set of the first query sentence "person in a car" and each of N images #11 to #1N to the VSE model.

Subsequently, the search unit 155 outputs the first similarity between the first query sentence and the image from the VSE model. For example, the search unit 155 outputs the degrees of similarity #11 to #1N between the first query sentence and each of the N images #11 to #1N, respectively. Subsequently, the search unit 155 searches for a first image whose output first similarity exceeds the first threshold. For example, suppose that similarity #11 to similarity #13 exceed the first threshold, but similarity #14 to similarity #1N are less than the first threshold. At this time, the search unit 155 selects images #11 to #13 whose first similarity with the first query sentence exceeds the first threshold from among the N images #11 to #1N as the first images. get.

Note that instead of searching for a first image whose output first similarity exceeds the first threshold, the search unit 155 searches several first images in order from the one with the highest output first similarity. It's fine. For example, among the degrees of similarity #11 to #1N between the first query sentence and N images #11 to #1N, the degree of similarity #11 is the highest, and the larger N is, the higher the degree of similarity is. Suppose it is low. At this time, the search unit 155 selects, for example, three images #11 to #13 from among the N images #11 to #1N in order from the one with the highest first similarity to the first query sentence. It may be acquired as the first image.

(Extraction unit 156)
The extraction unit 156 extracts a concept related to the first image. Specifically, when an image including a concept is input, the extraction unit 156 uses a concept identification which is a trained machine learning model that is trained to output the concept similarity between the concept included in the image and the image. A concept related to the first image is extracted from the first image using a device. For example, when the search unit 155 searches for a first image, the extraction unit 156 inputs the first image searched by the search unit 155 to the concept classifier. For example, the extraction unit 156 inputs each of images #11 to #13, which are the first images searched by the search unit 155, to the concept classifier. Note that, for the sake of simplicity, a case will be described below in which the first image is only image #11.

Subsequently, the extraction unit 156 outputs the concept similarity between the concept included in the first image and the first image from the concept classifier. For example, the extraction unit 156 outputs "90%", which is the concept similarity #21, between "car_interior", which is a concept included in image #11, and image #11, from the concept classifier. Further, for example, the extraction unit 156 outputs "80%", which is the concept similarity #22, between "automobile", which is a concept included in image #11, and image #11, from the concept discriminator. Further, for example, the extraction unit 156 outputs "70%", which is the concept similarity #23 between "motorcycle", which is a concept included in image #11, and image #11, from the concept discriminator.

Subsequently, the extraction unit 156 extracts concepts whose output concept similarity exceeds the concept threshold. For example, assume that the concept threshold is "85%". At this time, the extraction unit 156 extracts "car_interior", which is a concept whose concept similarity exceeds the concept threshold of "85%". Subsequently, the extraction unit 156 extracts hidden concepts, which are concepts that are not included in the first query sentence, from among the concepts whose output concept similarity exceeds the concept threshold. For example, the extraction unit 156 extracts a concept that includes a character string that is not included in the first query sentence as a hidden concept. For example, the extraction unit 156 extracts "car_interior", which is a concept that includes the character string "interior" that is not included in the first query sentence "person in a car", as a hidden concept.

Note that instead of extracting concepts whose output concept similarity exceeds the concept threshold, the extraction unit 156 may extract several concepts in descending order of output concept similarity. For example, among the concept similarities #11 to #1M between the first image and M concepts #11 to #1M (M is a natural number), the concept similarity #11 is the highest. , M is larger, the concept similarity is lower. At this time, the extraction unit 156 extracts, for example, three concepts #11 to #13 from the M concepts #11 to #1M in order of concept similarity to the first image. It's fine.

(Generation unit 157)
Hereinafter, the query sentence generated based on the hidden concept extracted by the extraction unit 156 will be referred to as a "second query sentence." The generation unit 157 generates a second query sentence based on the hidden concept extracted by the extraction unit 156. For example, the generation unit 157 may update the first query sentence based on the hidden concept extracted by the extraction unit 156 to generate the second query sentence. For example, the generation unit 157 may generate a second query sentence that includes “car_interior”, which is the hidden concept extracted by the extraction unit 156. The output unit 140 outputs the second query statement generated by the generation unit 157. For example, the output unit 140 outputs “car_interior” as an example of the second query sentence generated by the generation unit 157. The reception unit 153 receives the second query sentence output by the output unit 140 from the user. For example, the reception unit 153 receives “car_interior”, which is the second query sentence output by the output unit 140, from the user.

Note that instead of the generation unit 157 generating the second query sentence, the user may generate the second query sentence based on the hidden concept output by the output unit 140. The reception unit 153 may receive a second query sentence generated by the user from the user.

Further, when the reception unit 153 receives the second query sentence, the acquisition unit 154 refers to the storage unit 120 and acquires from the video pool an image that is a plurality of videos or each scene included in each of the plurality of videos. The search unit 155 uses the VSE model to search again for the second image related to the second query sentence received by the reception unit 153. For example, the search unit 155 searches again for the second image related to “car_interior”, which is the second query sentence accepted by the reception unit 153. For example, the search unit 155 inputs the set of the second query sentence received by the reception unit 153 and the image acquired by the acquisition unit 154 into the VSE model. Subsequently, the search unit 155 outputs the second similarity between the image and the second query sentence from the VSE model. Subsequently, the search unit 155 searches again for a second image whose output second similarity exceeds the second threshold. The output unit 140 outputs the second image searched again by the search unit 155 as a search result. In this way, the output unit 140 outputs, for example, the second image related to "car_interior", which is the second query sentence, and which is generated based on the hidden concept "car_interior", as a search result.

In the above example, the extraction unit 156 extracts "car_interior", which is a concept that includes the character string "interior" that is not included in the first query sentence "person in a car", as a hidden concept. , another example will be explained. For example, the reception unit 153 receives the phrase "destroyed old building" from the user as an example of the first query sentence. The search unit 155 uses the VSE model to search for a first image related to “destroyed old building”, which is the first query sentence accepted by the reception unit 153. The extraction unit 156 inputs the first image to the concept classifier and extracts the concept "ruin" included in the first image. Subsequently, the extraction unit 156 extracts "ruin", which is a concept that includes the character string "ruin" that is not included in the first query sentence "destroyed old building", as a hidden concept.

[3. Information processing flow]
Next, an information processing procedure according to the embodiment will be described using FIG. 4. FIG. 4 is a flowchart illustrating an example of information processing according to the embodiment. In FIG. 4, the attribute information extraction unit 151 acquires an image to be processed (step S11). For example, the attribute information extraction unit 151 acquires an image to be processed from a video pool.

Next, the attribute information extraction unit 151 uses segmentation technology to extract person area information regarding a person area including a person from the image. Further, the attribute information extraction unit 151 extracts skeletal information of a person included in the image using a posture estimation technique (step S12).

Next, the attribute information extraction unit 151 extracts person attribute information regarding the attributes of the person included in the image based on the person area information and the skeleton information (step S13). After extracting the person attribute information, the attribute information extraction unit 151 generates a sentence indicating the content of the image based on the extracted person attribute information (step S14).

The model generation unit 152 generates a VSE model learned to associate the text and image generated by the attribute information extraction unit 151 and embed them in a common space (step S15).

Next, the information processing procedure according to the embodiment will be described using FIG. 5. FIG. 5 is a flowchart illustrating an example of information processing according to the embodiment. In FIG. 5, the reception unit 153 receives the first query sentence input by the user (step S101). When the reception unit 153 receives the first query sentence, the acquisition unit 154 acquires images that are a plurality of videos or each scene included in each of the plurality of videos from the video pool (step S102).

The search unit 155 inputs the set of the first query sentence received by the reception unit 153 and the image acquired by the acquisition unit 154 into the VSE model. Subsequently, the search unit 155 outputs the first similarity between the first query sentence and the image from the VSE model (step S103). Subsequently, the search unit 155 searches for a first image whose output first similarity exceeds the first threshold (step S104).

The extraction unit 156 inputs the first image searched by the search unit 155 to the concept classifier (step S105). Subsequently, the extraction unit 156 outputs the concept similarity between the concept included in the first image and the first image from the concept classifier (step S106). Subsequently, the extraction unit 156 extracts concepts whose output concept similarity exceeds the concept threshold. Subsequently, the extraction unit 156 extracts hidden concepts that are not included in the first query sentence from among the concepts whose output concept similarity exceeds the concept threshold (step S107).

The output unit 140 outputs the hidden concept extracted by the extraction unit 156 (step S108). The user updates the first query sentence based on the hidden concept output by the output unit 140 and generates a new second query sentence (step S109). The receiving unit 153 receives the second query sentence updated based on the hidden concept output by the output unit 140 from the user. For example, the reception unit 153 receives the second query sentence updated by the user from the user via the input unit 130 (step S110). When the reception unit 153 receives the second query sentence, the acquisition unit 154 acquires images that are a plurality of videos or each scene included in each of the plurality of videos from the video pool (step S111).

The search unit 155 inputs the set of the second query sentence accepted by the reception unit 153 and the image acquired by the acquisition unit 154 into the VSE model. Subsequently, the search unit 155 outputs the second similarity between the image and the second query sentence from the VSE model (step S112). Subsequently, the search unit 155 searches again for a second image whose output second similarity exceeds the second threshold (step S113). The output unit 140 outputs the second image searched again by the search unit 155 as a search result (step S114).

[4. Modified example]
Next, an information processing procedure according to a modified example will be described using FIG. 6. FIG. 6 is a flowchart illustrating an example of information processing according to a modification. In FIG. 6, the reception unit 153 receives a query image input by a user (step S201). For example, the reception unit 153 accepts a query image input by a user via the input unit 130. Here, the query image in this specification does not have to be the entire image, but may be, for example, a part of the image.

When the reception unit 153 receives the query image, the acquisition unit 154 refers to the storage unit 120 and acquires a character string that is a plurality of sentences or each text included in each of the plurality of sentences from the caption pool (step S202). .

Further, when the acquisition unit 154 acquires a character string, the search unit 155 inputs the set of the query image accepted by the reception unit 153 and the character string acquired by the acquisition unit 154 to the VSE model. Subsequently, the search unit 155 outputs the third degree of similarity between the query image and the character string from the VSE model (step S203). Subsequently, the search unit 155 searches for a character string whose output third similarity exceeds the third threshold (step S204).

The generation unit 157 generates a third query sentence based on the character string searched by the search unit 155. The output unit 140 outputs the third query statement generated by the generation unit 157. The reception unit 153 receives the third query sentence output by the output unit 140 from the user (step S205). When the reception unit 153 receives the third query sentence, the acquisition unit 154 acquires images that are a plurality of videos or each scene included in each of the plurality of videos from the video pool (step S206).

The search unit 155 inputs the set of the third query received by the reception unit 153 and the image acquired by the acquisition unit 154 into the VSE model. Subsequently, the search unit 155 outputs the first similarity between the image and the third query sentence from the VSE model (step S207). Subsequently, the search unit 155 searches for a third image whose output first similarity exceeds the first threshold (step S208).

The extraction unit 156 inputs the third image searched by the search unit 155 to the concept classifier (step S209). Subsequently, the extraction unit 156 outputs the concept similarity between the concept included in the third image and the third image from the concept classifier (step S210). Subsequently, the extraction unit 156 extracts concepts whose output concept similarity exceeds the concept threshold. Subsequently, the extraction unit 156 extracts hidden concepts that are not included in the third query sentence from among the concepts whose output concept similarity exceeds the concept threshold (step S211).

The output unit 140 outputs the hidden concept extracted by the extraction unit 156 (step S212). The user updates the third query sentence based on the hidden concept output by the output unit 140 and generates a new fourth query sentence (step S213). The reception unit 153 receives from the user a fourth query sentence that has been updated based on the hidden concept output by the output unit 140. For example, the receiving unit 153 receives the fourth query statement updated by the user from the user via the input unit 130 (step S214). When the reception unit 153 receives the fourth query sentence, the acquisition unit 154 acquires images that are a plurality of videos or each scene included in each of the plurality of videos from the video pool (step S215).

The search unit 155 inputs the set of the fourth query received by the reception unit 153 and the image acquired by the acquisition unit 154 into the VSE model. Subsequently, the search unit 155 outputs the second similarity between the image and the fourth query sentence from the VSE model (step S216). Subsequently, the search unit 155 searches again for a fourth image whose output second similarity exceeds the second threshold (step S217). The output unit 140 outputs the fourth image searched again by the search unit 155 as a search result (step S218).

Note that in step S205, the user can change the third query sentence output by the output unit 140. The reception unit 153 receives from the user, via the input unit 130, the third query sentence that has been changed by the user.

[5. effect〕
As described above, the information processing device 100 according to the embodiment includes the attribute information extraction section 151 and the model generation section 152. The attribute information extraction unit 151 extracts region information regarding an object region, which is a divided region including an object having a structure, out of an image divided into regions using a segmentation technology, and an object estimated using a pose estimation technology. Attribute information regarding the attributes of the object is extracted based on the structural information regarding the structure of the object. The model generation unit 152 is trained to associate sentences with images, which are generated based on the attribute information extracted by the attribute information extraction unit 151, and which indicate the content of images, and embed them in a common space. Generate a VSE (Visual-Semantic Embedding) model.

In this way, the information processing device 100 combines segmentation technology and pose estimation technology to identify regions of structured objects (for example, vehicles and people) included in an image, and the constituent elements of objects (each part and each part). It is possible to accurately extract the region of the object (part) and the posture of the object included in the image. Thereby, the information processing apparatus 100 can hierarchically decompose the object region and the constituent elements of the object region (for example, the structure of the object, the region of each part, etc.) included in the image. Further, the information processing apparatus 100 can appropriately extract attribute information regarding the attributes of the object from the object region and structure information based on the hierarchical relationship of the decomposed components. Further, the information processing apparatus 100 can appropriately generate a sentence indicating the content of the image based on attribute information appropriately extracted from the image. Therefore, the information processing device 100 according to the present invention can improve image search accuracy.

Further, the attribute information extraction unit 151 extracts, as area information, person area information regarding a person area, which is a divided area including an object that is a person, from the image, and person area information regarding a person area, which is a divided area including an object that is a person, and structure information, which is estimated using posture estimation technology. Personal attribute information regarding the attributes of the person is extracted as attribute information based on skeletal information regarding the skeleton of the person. The model generation unit 152 is trained to associate sentences with images, which are generated based on the person attribute information extracted by the attribute information extraction unit 151, and which indicate the content of the image, and embed them in a common space. A VSE model is generated.

By combining segmentation technology and posture estimation technology, the information processing apparatus 100 can accurately extract a region of a person included in an image, a region of clothing, and a posture of a person included in an image. Thereby, the information processing device 100 can hierarchically decompose the human region and the components of the human region (for example, joint position information of each joint, body part region, fashion item region, etc.) included in the image. can. Furthermore, the information processing apparatus 100 can appropriately extract person attribute information regarding attributes of a person from the person region and skeletal information based on the hierarchical relationship of the decomposed components. Further, the information processing apparatus 100 can appropriately generate a sentence indicating the content of the image based on the person attribute information appropriately extracted from the image. Therefore, the information processing device 100 according to the present invention can improve image search accuracy.

Further, the attribute information extraction unit 151 extracts posture information regarding the posture of the person and motion information regarding the movement of the person as the person attribute information based on the joint position information of each joint of the person as the skeletal information.

Thereby, the information processing apparatus 100 can appropriately extract information regarding the posture and motion of the person included in the image, and therefore can appropriately generate a sentence indicating the content of the image.

Further, the attribute information extraction unit 151 extracts items worn by the person as person attribute information based on item area information regarding item areas, which are divided areas including each fashion item worn by the person, as person area information. To extract item attribute information regarding the attributes of a fashion item.

Thereby, the information processing apparatus 100 can appropriately extract information regarding the fashion items of the person included in the image, and therefore can appropriately generate a sentence indicating the content of the image.

The attribute information extraction unit 151 also extracts body part area information regarding the attributes of the body parts of the person as the person attribute information based on body part area information regarding the body part area which is a divided area including each body part of the person. Extract part attribute information. For example, the attribute information extraction unit 151 extracts information about a person's hairstyle as body part attribute information based on information about a hair region of the person as body part area information. Further, the attribute information extraction unit 151 extracts information regarding a person's facial expression as body part attribute information based on information regarding the face area of the person as body part area information.

Thereby, the information processing apparatus 100 can appropriately extract information regarding the hairstyle and facial expression of the person included in the image, and can therefore appropriately generate a sentence indicating the content of the image.

Further, the information processing device 100 includes a reception section 153, a search section 155, and an extraction section 156. The reception unit 153 receives the first query sentence input by the user. The search unit 155 uses the VSE model generated by the model generation unit 152 to search for a first image related to the first query sentence. The extraction unit 156 extracts a concept related to the first image. The search unit 155 uses the VSE model to search again for the second image related to the second query sentence based on the concept extracted by the extraction unit 156.

Thereby, the information processing apparatus 100 can appropriately search for images related to the query sentence input by the user by using VSE. Further, the information processing apparatus 100 can extract a concept from an appropriately searched image and then search for images again based on the extracted concept. Therefore, the information processing device 100 can improve image search accuracy.

Further, the information processing device 100 further includes a generation unit 157. The generation unit 157 generates a second query sentence based on the concept extracted by the extraction unit 156. The search unit 155 uses the VSE model to search again for the second image related to the second query sentence generated by the generation unit 157.

Thereby, the information processing device 100 can generate an appropriate query sentence based on an appropriate concept. For example, the information processing device 100 can improve search accuracy by adding appropriate search keywords (or excluding inappropriate search keywords). Therefore, the information processing apparatus 100 can search for images again based on an appropriate query sentence, and thus can improve image search accuracy.

Furthermore, the information processing device 100 further includes an output unit 140 that outputs the search results by the search unit 155. The output unit 140 outputs the concept extracted by the extraction unit 156. The reception unit 153 receives a second query sentence based on the concept output by the output unit 140 from the user. The search unit 155 uses the VSE model to search again for the second image related to the second query sentence received by the reception unit 153.

Thereby, the information processing apparatus 100 can help the user generate an appropriate query sentence based on an appropriate concept. For example, the information processing device 100 allows the user to improve search accuracy by adding appropriate search keywords (or eliminating inappropriate search keywords). Therefore, the information processing apparatus 100 can search for images again based on an appropriate query sentence, and thus can improve image search accuracy.

Further, the information processing device 100 further includes an acquisition unit 154. The acquisition unit 154 acquires images that are a plurality of videos or each scene included in each of the plurality of videos. The search unit 155 inputs the set of the image acquired by the acquisition unit 154 and the first query sentence accepted by the reception unit 153 into the VSE model, and calculates the first similarity between the image and the first query sentence using the VSE model. , and a first image whose output first similarity exceeds a first threshold is searched.

Thereby, the information processing apparatus 100 can select an appropriate image through processing based on VSE.

Furthermore, the search unit 155 inputs the set of the image acquired by the acquisition unit 154 and the second query sentence based on the concept extracted by the extraction unit 156 into the VSE model, and The similarity is output from the VSE model, and a second image whose output second similarity exceeds the second threshold is searched again.

Thereby, the information processing apparatus 100 can select an appropriate concept by performing processing based on the VSE and processing based on the concept discriminator. For example, the information processing apparatus 100 can extract a concept related to content (for example, implicit content) that is not explicitly stated in the query sentence input by the user.

Further, the extraction unit 156 uses a concept classifier, which is a trained machine learning model, which is trained to output the concept similarity between the concept included in the image and the image when an image including a concept is input. Then, a concept related to the first image is extracted from the first image.

Thereby, the information processing apparatus 100 can extract an appropriate concept from an appropriately searched image by using the concept classifier. Further, the information processing apparatus 100 can extract an appropriate concept and then search for images again based on the appropriate concept. Therefore, the information processing device 100 can improve image search accuracy.

Further, the extraction unit 156 inputs the first image searched by the search unit 155 to the concept classifier, outputs the concept similarity between the concept included in the first image and the first image from the concept classifier, Concepts whose output concept similarity exceeds a concept threshold are extracted.

Thereby, the information processing apparatus 100 can extract an appropriate concept.

Furthermore, the extraction unit 156 extracts hidden concepts that are not included in the first query sentence from among the concepts whose output concept similarity exceeds the concept threshold.

Thereby, the information processing apparatus 100 can extract a concept related to content (for example, implicit content) that is not explicitly stated in the query sentence input by the user. Furthermore, the information processing apparatus 100 can re-search for images based on a concept (for example, an implicit concept) that is not explicitly stated in the query sentence input by the user. Therefore, the information processing device 100 can improve image search accuracy.
Further, the reception unit 153 receives a query image input by a user. The search unit 155 uses the VSE model to search for a character string related to the query image accepted by the reception unit 153, and searches for a third image related to a third query sentence based on the searched character string.

Thereby, the information processing apparatus 100 can extract a concept related to content (for example, implicit content) that is not explicitly stated in the query image input by the user.

Further, the acquisition unit 154 acquires a plurality of sentences or a character string that is each text included in each of the plurality of sentences. The search unit 155 inputs the combination of the character string acquired by the acquisition unit 154 and the query image accepted by the reception unit 153 into the VSE model, and outputs a third degree of similarity between the character string and the query image from the VSE model. Then, a character string whose output third similarity exceeds a third threshold is searched, and a third image related to a third query sentence based on the searched character string is searched.

Thereby, the information processing apparatus 100 can select an appropriate concept by performing processing based on the VSE and processing based on the concept discriminator.

Furthermore, the extraction unit 156 extracts a concept related to the third image. The search unit 155 uses the VSE model to search again for the fourth image related to the fourth query sentence based on the concept extracted by the extraction unit 156.

Thereby, the information processing apparatus 100 can extract a concept related to content (for example, implicit content) that is not explicitly stated in the query image input by the user. Further, the information processing apparatus 100 can re-search for images based on a concept (for example, an implicit concept) that is not explicitly stated in the query image input by the user. Therefore, the information processing device 100 can improve image search accuracy.

[6. Hardware configuration]
Further, the information processing apparatus 100 according to the embodiments described above is realized by, for example, a computer 1000 having a configuration as shown in FIG. FIG. 7 is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device 100. The computer 1000 includes a CPU 1100, a RAM 1200, a ROM 1300, an HDD 1400, a communication interface (I/F) 1500, an input/output interface (I/F) 1600, and a media interface (I/F) 1700.

CPU 1100 operates based on a program stored in ROM 1300 or HDD 1400, and controls each section. The ROM 1300 stores a boot program executed by the CPU 1100 when the computer 1000 is started, programs depending on the hardware of the computer 1000, and the like.

The HDD 1400 stores programs executed by the CPU 1100, data used by the programs, and the like. Communication interface 1500 receives data from other devices via a predetermined communication network and sends it to CPU 1100, and transmits data generated by CPU 1100 to other devices via a predetermined communication network.

The CPU 1100 controls output devices such as a display and a printer, and input devices such as a keyboard and mouse via an input/output interface 1600. CPU 1100 obtains data from an input device via input/output interface 1600. Further, CPU 1100 outputs the generated data to an output device via input/output interface 1600. Note that instead of the CPU 1100, an MPU (Micro Processing Unit) or a GPU (Graphics Processing Unit) may be used since it requires a large amount of calculation power.

Media interface 1700 reads programs or data stored in recording medium 1800 and provides them to CPU 1100 via RAM 1200. CPU 1100 loads this program from recording medium 1800 onto RAM 1200 via media interface 1700, and executes the loaded program. The recording medium 1800 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. etc.

For example, when the computer 1000 functions as the information processing device 100, the CPU 1100 of the computer 1000 realizes the functions of the control unit 150 by executing a program loaded onto the RAM 1200. The CPU 1100 of the computer 1000 reads these programs from the recording medium 1800 and executes them, but as another example, these programs may be acquired from another device via a predetermined communication network.

Some of the embodiments of the present application have been described above in detail based on the drawings, but these are merely examples, and various modifications and variations may be made based on the knowledge of those skilled in the art, including the embodiments described in the disclosure section of the invention. It is possible to carry out the invention in other forms with modifications.

[7. others〕
Furthermore, among the processes described in the above embodiments and modified examples, all or part of the processes described as being performed automatically can be performed manually, or may be described as being performed manually. All or part of this processing can also be performed automatically using known methods. In addition, information including the processing procedures, specific names, and various data and parameters shown in the above documents and drawings may be changed arbitrarily, unless otherwise specified. For example, the various information shown in each figure is not limited to the illustrated information.

Furthermore, each component of each device shown in the drawings is functionally conceptual, and does not necessarily need to be physically configured as shown in the drawings. In other words, the specific form of distributing and integrating each device is not limited to what is shown in the diagram, and all or part of the devices can be functionally or physically distributed or integrated in arbitrary units depending on various loads and usage conditions. Can be integrated and configured.

Furthermore, the above-described embodiments and modifications can be combined as appropriate within a range that does not conflict with the processing contents.

Further, the above-mentioned "section, module, unit" can be read as "means", "circuit", etc. For example, the search unit can be replaced with a search means or a search circuit.

100 Information processing device 110 Communication unit 120 Storage unit 130 Input unit 140 Output unit 150 Control unit 151 Attribute information extraction unit 152 Model generation unit 153 Reception unit 154 Acquisition unit 155 Search unit 156 Extraction unit 157 Generation unit

Claims (21)

In an image divided into regions using segmentation technology, region information regarding the object region, which is a divided region including an object having a structure, and structural information regarding the structure of the object estimated using pose estimation technology. an attribute information extraction unit that extracts attribute information regarding attributes of the object based on the object;
The text generated based on the attribute information extracted by the attribute information extraction unit is a VSE (Visual a model generation unit that generates a Semantic Embedding model;
An information processing device comprising: The attribute information extraction unit includes:
In the image, the area information is person area information regarding a person area that is a divided area including the object that is a person, and the structure information is the skeleton of the person estimated using the posture estimation technique. extracting person attribute information regarding attributes of the person as the attribute information based on skeletal information about the person;
The model generation unit is
The VSE model is a sentence generated based on the person attribute information extracted by the attribute information extraction unit, and is trained to associate a sentence indicating the content of the image with the image and embed it in a common space. generate,
The information processing device according to claim 1. The attribute information extraction unit includes:
extracting, as the skeletal information, posture information regarding the posture of the person and motion information regarding the motion of the person, as the person attribute information, based on joint position information of each joint of the person;
The information processing device according to claim 2. The attribute information extraction unit includes:
The person area information is based on item area information regarding an item area that is a divided area including each fashion item worn by the person, and the person attribute information is information about fashion items worn by the person. Extract item attribute information about attributes,
The information processing device according to claim 2 or 3. The attribute information extraction unit includes:
As the person area information, extract body part attribute information regarding attributes of the body parts of the person as the person attribute information based on body part area information regarding a body part area that is a divided area including each body part of the person. do,
The information processing device according to any one of claims 2 to 4. The attribute information extraction unit includes:
extracting information regarding the hairstyle of the person as the body part attribute information based on information regarding the hair area of the person as the body part area information;
The information processing device according to claim 5. The attribute information extraction unit includes:
extracting information regarding facial expressions of the person as the body part attribute information based on information regarding the face area of the person as the body part area information;
The information processing device according to claim 5 or 6. a reception unit that receives a first query sentence input by a user;
a search unit that searches for a first image related to the first query sentence using the VSE model generated by the model generation unit;
an extraction unit that extracts a first concept that is a detection target included in the first image;
Furthermore,
The search section includes:
re-searching for a second image related to a second query sentence based on the first concept extracted by the extraction unit using the VSE model;
The information processing device according to any one of claims 1 to 7. The detection target includes at least one of an object, a person, a scene, and an action.
The information processing device according to claim 8. further comprising a generation unit that generates the second query sentence based on the first concept extracted by the extraction unit,
The search section includes:
re-searching the second image related to the second query sentence generated by the generation unit using the VSE model;
The information processing device according to claim 8. further comprising an output unit that outputs search results by the search unit,
The output section is
outputting first concept information regarding the first concept extracted by the extraction unit;
The reception department is
receiving from the user the second query sentence based on the first concept information output by the output unit;
The search section includes:
re-searching the second image related to the second query sentence accepted by the reception unit using the VSE model;
The information processing device according to claim 8. further comprising an acquisition unit that acquires images that are a plurality of videos or each scene included in each of the plurality of videos,
The search section includes:
A set of the image acquired by the acquisition unit and the first query sentence accepted by the reception unit is input into the VSE model, and a first similarity between the image and the first query sentence is calculated from the VSE model. output, and searching for the first image for which the output first similarity exceeds a first threshold;
The information processing device according to any one of claims 8 to 11 . The search section includes:
A set of an image acquired by the acquisition unit and a second query sentence based on the first concept extracted by the extraction unit is input to the VSE model, and a second similarity between the image and the second query sentence is calculated. output from the VSE model, and re-search the second image for which the output second similarity exceeds a second threshold;
The information processing device according to claim 12 . The extraction section is
When an image is input, a concept classifier, which is a trained machine learning model, is trained to output the concept similarity between the image and a concept to be detected contained in the image. extracting the first concept from a first image;
The information processing device according to any one of claims 8 to 13 . The extraction section is
The first image searched by the search unit is input to the concept classifier, and the concept similarity between the first concept to be detected included in the first image and the first image is determined from the concept classifier. output, and extracting the first concept whose output concept similarity exceeds a concept threshold;
The information processing device according to claim 14 . The extraction section is
extracting a hidden concept that is the first concept corresponding to a character string not included in the first query sentence from among the first concepts for which the output concept similarity exceeds a concept threshold;
The information processing device according to claim 15 . The reception department is
accept a query image input by the user;
The search section includes:
using the VSE model to search for a character string related to the query image accepted by the reception unit, and search for a third image related to a third query sentence based on the searched string;
The information processing device according to any one of claims 8 to 16 . further comprising an acquisition unit that acquires a plurality of sentences or a character string that is each text included in each of the plurality of sentences,
The search section includes:
A set of the character string acquired by the acquisition unit and the query image accepted by the reception unit is input to the VSE model, and a third similarity between the character string and the query image is output from the VSE model. , searching for a character string for which the output third similarity exceeds a third threshold, and searching for the third image related to the third query sentence based on the searched character string;
The information processing device according to claim 17 . The extraction section is
extracting a third concept that is a detection target included in the third image;
The search section includes:
re -searching for a fourth image related to a fourth query sentence based on the third concept extracted by the extraction unit using the VSE model;
The information processing device according to claim 18 . An information processing method realized by a program executed by an information processing device, the method comprising:
In an image divided into regions using segmentation technology, region information regarding the object region, which is a divided region including an object having a structure, and structural information regarding the structure of the object estimated using pose estimation technology. an attribute information extraction step of extracting attribute information regarding attributes of the object based on the object;
The text generated based on the attribute information extracted in the attribute information extraction step is a VSE (Visual a model generation process that generates a Semantic Embedding model;
Information processing methods including In an image divided into regions using segmentation technology, region information regarding the object region, which is a divided region including an object having a structure, and structural information regarding the structure of the object estimated using pose estimation technology. an attribute information extraction procedure for extracting attribute information regarding attributes of the object based on the object;
The text generated based on the attribute information extracted by the attribute information extraction procedure is a VSE (Visual a model generation procedure for generating a Semantic Embedding) model;
An information processing program that causes a computer to execute.

Images