JP7224315B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and an information processing program.

Focusing on the difference in style due to the skeleton that humans are born with, we perform skeleton diagnosis in order to coordinate fashion that suits each style. In skeletal diagnosis, the skeletal structure can be classified into three types, for example, straight (S), wavy (W), and natural (N), based on physical characteristics.

There are two types of skeletal diagnosis: self-diagnosis and diagnosis by experts. In the self-diagnosis, for example, the person to be diagnosed decides the skeleton type by going through a Q&A flow chart regarding physical characteristics. On the other hand, in the diagnosis by an expert, for example, an expert who is certified by the Skeletal Diagnosis Analyst Association directly diagnoses the body of the person to be diagnosed and determines the skeleton type.

However, the self-diagnosis is not very reliable because the questions are detailed, such as ``I have muscle tone in XX,'' and there are many contents that are difficult for the subject to judge by himself/herself. On the other hand, diagnosis by an expert is highly reliable, but it is not easy for the person to be diagnosed because it is necessary to have the expert diagnose the patient directly.

Therefore, the present disclosure proposes an information processing device, an information processing method, and an information processing program that allow a user, who is a person to be diagnosed, to easily perform highly reliable skeleton diagnosis.

According to the present disclosure, an acquisition unit that acquires a plurality of first images of a person and a first skeleton type of the person, inputs the plurality of first images, and determines that the first skeleton type is the correct answer. An information processing apparatus is provided, comprising a learning unit that learns teacher data and generates a learning model.

Further, according to the present disclosure, an information processing device obtains a plurality of first images in which a person is captured and a first skeleton type of the person, inputs the plurality of first images, and acquires the first skeleton type is the correct answer, and a process of generating a learning model by learning teacher data is provided.

Further, according to the present disclosure, a plurality of first images showing a person and a first skeleton type of the person are acquired in the information processing apparatus, the plurality of first images are input, and the first skeleton type is obtained. is the correct answer, and a program is provided for executing processing for generating a learning model by learning teacher data.

According to the present disclosure, an acquisition unit that acquires a plurality of second images showing a user, a plurality of first images showing a person are input, and teacher data that corrects the first skeleton type of the person and a determination unit that determines the second skeleton type of the user based on the results output by inputting a plurality of second images to the learning model generated by learning An information processing device is provided.

Further, according to the present disclosure, the information processing device acquires a plurality of second images in which the user is photographed, inputs a plurality of first images, and learns teacher data that corrects the first skeleton type. determining a second skeleton type of the user based on results output from inputting a plurality of second images to the learning model generated by be.

Further, according to the present disclosure, the information processing apparatus acquires a plurality of second images in which the user is photographed, inputs a plurality of first images, and learns teacher data that corrects the first skeleton type. A program characterized by executing a process of determining a second skeleton type of a user based on a result output by inputting a plurality of second images to the learning model generated by be.

It is a figure which shows the structural example of the information processing system which concerns on this embodiment. 2 is a block diagram showing a functional configuration example of the information processing apparatus 100 according to the same embodiment; FIG. It is a figure which shows an example of the learning model which concerns on the same embodiment. It is a figure which shows an example of learning of the feature-value extraction layer 500 which concerns on the same embodiment. It is a figure which shows an example of learning of the fully connected layer 600 which concerns on the same embodiment. FIG. 10 is a diagram showing another example of learning of the feature quantity extraction layer 500 according to the same embodiment; FIG. 10 is a diagram showing another example of learning of the fully connected layer 600 according to the same embodiment; It is a figure which shows an example of the user interface which concerns on the same embodiment. It is a flow chart which shows a flow of learning model generation processing concerning the embodiment. It is a flowchart which shows the flow of the learning process which concerns on the same embodiment. 9 is a flowchart showing the flow of skeleton type determination processing according to the embodiment; 9 is a flowchart showing another flow of skeleton type determination processing according to the same embodiment. 2 is a block diagram showing a hardware configuration example of the information processing apparatus 100 according to the same embodiment; FIG.

The present embodiment will be described in detail below with reference to the drawings. In addition, in this specification and the drawings, substantially the same portions are denoted by the same reference numerals, thereby omitting redundant explanations.

In addition, description shall be performed in the following order.
1. Embodiment 1.1. System configuration example 1.2. Functional configuration example 1.3. Details of functions 1.4. Function flow 2 . Hardware configuration example 3 . summary

<1. embodiment>
<<1.1. System configuration example >>
First, a configuration example of an information processing system according to this embodiment will be described. FIG. 1 is a diagram showing a configuration example of an information processing system according to this embodiment. As shown in FIG. 1 , the information processing system includes an information processing device 100 , a user terminal 200 and a store terminal 300 . As shown in FIG. 1, an information processing apparatus 100, a user terminal 200 and a store terminal 300 are connected via a network N so as to be able to communicate with each other. The network N can employ various communication networks such as the Internet, regardless of whether they are wired or wireless.

The information processing device 100 is, for example, a server device managed by a service provider that provides a fashion coordination service based on skeletal diagnosis. The information processing apparatus 100 receives an image of the user (user image) from the user terminal 200 or the shop terminal 300, and inputs the user image into the learning model to determine the skeleton type of the user. The information processing apparatus 100 also searches for recommended coordination based on the determined skeleton type, and presents it to the user via the user terminal 200 or the store terminal 300 . The information processing device 100 may be a cloud server device, or may be a distributed computing system composed of a plurality of computers.

A user terminal 200 is a terminal owned and used by a user. Note that the user is, for example, a person to be diagnosed who undergoes a skeleton diagnosis in order to coordinate fashion that suits him/herself. The user terminal 200 may be a mobile terminal such as a smart phone or a tablet PC, or may be a stationary terminal installed in the user's home or the like.

The user terminal 200 is assumed to be used in such a way that the user receives a skeleton diagnosis at home. Therefore, the user uses the user terminal 200 or other camera device to photograph the user, for example, from the front, back, or side. The parts to be imaged are, for example, the whole body, half of the body, clavicle, chest, legs to waist, and hands. Also, the user uploads a captured user image to the information processing apparatus 100 via the user terminal 200 .

Further, the user terminal 200 receives from the information processing apparatus 100 the skeleton type of the user determined based on the photographed user image and recommended coordination, and presents them to the user. Uploading of user images and display of recommended coordination can be performed via a dedicated user interface (UI).

The store terminal 300 is, for example, a digital signage installed in a store that sells clothing and the like. However, the store terminal 300 may be a stationary terminal installed in a store, or may be a mobile terminal such as a smartphone or a tablet PC for store use handled by a store clerk.

The store terminal 300 is assumed to be used in such a way that a user undergoes a skeleton diagnosis at a store. Similarly to the user terminal 200, the store terminal 300 also uploads a photographed user image to the information processing device 100, receives the user's skeleton type and recommended coordination determined based on the photographed user image from the information processing device 100, present to the user.

Note that FIG. 1 shows the information processing system according to the present embodiment to include at least three devices: an information processing apparatus 100, a user terminal 200, and a store terminal 300. As shown in FIG. However, as described above, the information processing system according to the present embodiment may include only two devices, the information processing device 100 and the user terminal 200 or the information processing device 100 and the store terminal 300, depending on the usage pattern. . Alternatively, for example, only an application installed in one device of the user terminal 200 or the store terminal 300 can perform everything from taking a user image to skeletal diagnosis, and presenting a user's skeletal type and recommended coordination.

<<1.2. Functional configuration example >>
Next, a functional configuration example of the information processing apparatus 100 according to this embodiment will be described. FIG. 2 is a block diagram showing a functional configuration example of the information processing apparatus 100 according to this embodiment. As shown in FIG. 2, the information processing apparatus 100 according to the present embodiment includes a storage unit 110, an acquisition unit 120, a learning unit 130, a determination unit 140, a search unit 150, a transmission unit 160, an imaging unit 170, and a control unit 180. Prepare.

(storage unit 110)
The storage unit 110 according to this embodiment is a storage area for temporarily or permanently storing various programs and data. The storage unit 110 may store programs and data for the information processing apparatus 100 to execute various functions. As a specific example, the storage unit 110 stores programs and data for uploading user images and displaying recommended coordination, a learning model for determining a person's skeleton type, and various settings. management data and the like may be stored. Of course, the above is just an example, and the type of data stored in storage unit 110 is not particularly limited.

(Acquisition unit 120)
Since the acquisition unit 120 according to the present embodiment is used as teacher data for a learning model, an image of a person (corresponding to a “first image”) and a skeleton type of the person (corresponding to a “first skeleton type”) ). An image of a person is obtained by photographing the person from the front, back, or side, and from the whole body, the half body, the clavicle, the chest, the legs to the waist, and the hands. The skeletal type of a person is a skeletal type (for example, straight (S), wavy (W), and natural (N) three types) determined by an expert in diagnosing the person.

In addition, the acquisition unit 120 acquires an image of the user (user image, corresponding to a “second image”) uploaded via the user terminal 200 in order to diagnose the skeleton of the user. The user image is also a front, back, or side image of the user captured including parts such as the whole body, half body, clavicle, chest, legs to waist, and hands. That is, a user image showing the same part as the part used as teacher data for the learning model is acquired.

(Learning unit 130)
The learning unit 130 according to the present embodiment receives an image of a person acquired by the acquisition unit 120, learns teacher data that corrects the skeletal type of the person, and generates a learning model. In addition, images to be learned by the learning model can be input to the learning model as a set of a plurality of images taken for each part such as the whole body or half of the body as one set of feature data. The skeletal type used as the correct data is a skeletal type determined by an expert by diagnosing the person directly or by viewing an image. In addition, if the reliability is high, the correct data may include the skeletal type determined by the person to be diagnosed by following the Q&A flow chart regarding physical features.

Although the details will be described later, the learning method by the learning unit 130 can use, for example, a convolutional neural network (CNN) specialized for image learning. Furthermore, the learning unit 130 can transfer-learn a learning model using ImageNet, which is an image data set obtained by learning a huge number of images. Further, during this transfer learning, for example, the learning model can be separated into a feature amount extraction layer and a fully connected layer, and the feature amount output from the feature amount extraction layer can be used for learning of the fully connected layer. As a result, the learning time can be shortened and the load of the learning process can be reduced by performing the minimum necessary learning instead of learning all the layers of the learning model each time.

Note that the learning model of the present embodiment includes an input layer to which an image of a user is input, an output layer, and a first layer belonging to any layer from the input layer to the output layer other than the output layer. and a second element whose value is calculated based on the first element and the weight of the first element, and outputs the user's skeleton type from the output layer according to the image input to the input layer. Thus, the information processing apparatus 100 is made to function.

It should be noted that the generation device (for example, the information processing device 100 such as the server device) that generates the learning model of the present embodiment may generate the above-described learning model using any learning algorithm. For example, the generation device may generate the learning model of this embodiment using a learning algorithm such as a neural network (NN), a support vector machine (SVM), clustering, or reinforcement learning. As an example, assume that the generating device uses the NN to generate the learning model of this embodiment. In this case, the learning model may have an input layer containing one or more neurons, an intermediate layer containing one or more neurons, and an output layer containing one or more neurons.

Here, it is assumed that the learning model according to the present embodiment is realized by a regression model represented by "y= _a1 * _x1 + _a2 * _x2 +...+ _ai * _xi ". In this case, the first element included in the learning model corresponds to the input data (x _i ) such as x ₁ and x ₂ . Also, the weight of the first element corresponds to the coefficient a _i corresponding to x _i . Here, the regression model can be viewed as a simple perceptron with an input layer and an output layer. When each model is regarded as a simple perceptron, the first element can be regarded as a node of the input layer, and the second element can be regarded as a node of the output layer.

Also, it is assumed that the learning model according to this embodiment is realized by an NN having one or more intermediate layers, such as a DNN (Deep Neural Network). In this case, the first element included in the learning model corresponds to any node of the input layer or intermediate layer. Also, the second element corresponds to the next node, which is a node to which the value is transmitted from the node corresponding to the first element. Also, the weight of the first element corresponds to the connection coefficient, which is the weight considered for the value transmitted from the node corresponding to the first element to the node corresponding to the second element.

A user's skeleton type is calculated using a learning model having an arbitrary structure such as the regression model and NN described above. More specifically, in the learning model, coefficients are set so that when an image of a user is input, the skeleton type of the user is output. The learning model according to the present embodiment may be a model generated based on results obtained by repeating data input/output.

In the above example, the learning model according to the present embodiment is a model (referred to as model A) that outputs the user's skeleton type when an image of the user is input. However, the learning model according to the present embodiment may be a model generated based on results obtained by repeatedly inputting and outputting data to model A. For example, the learning model according to the present embodiment may be a learning model (model B) that receives an image of a user as an input and outputs the skeleton type of the user that is output by the model A. Alternatively, the learning model according to the present embodiment may be a learning model that receives an image of a user as an input and outputs the skeleton type of the user that the model B outputs.

(Determination unit 140)
The determining unit 140 according to the present embodiment determines the skeleton type of the user (corresponding to the “second skeleton type”) based on the result output by inputting the user image into the learning model. The result output from the learning model is, for example, "S: 0.8, W: 0.1, N: 0.1", with 1 being the maximum (100%), the range [0 to 1] Probability of each skeleton type indicated by a number.

(Search unit 150)
The search unit 150 according to the present embodiment searches for recommended coordination for the user based on the user's skeleton type determined by the determination unit 140 . Specifically, for example, the search unit 150 searches for recommended coordination stored in the storage unit 110 and associated with the skeleton type, using the skeleton type determined by the determination unit 140 as a search key. At this time, recommended coordination can be narrowed down based on user attributes such as height, weight, age, sex, and activity area of the user, the current season, and the like. Also, the recommended coordination can include not only clothing, furnishings, and accessories, but also materials and patterns that suit the user. Furthermore, the search unit 150 can also search for sales information (stores that carry the item, information on sales floors, online shops) associated with recommended clothing items and the like. In addition, the search unit 150 performs a user search based on the user information, and searches for various types of information such as the skeleton type and user attributes of users who have undergone skeleton diagnosis.

(transmitting unit 160)
The transmission unit 160 according to the present embodiment transmits the user's skeleton type determined by the determination unit 140 and the recommended coordination retrieved by the search unit 150 to the user terminal 200 and the store terminal 300 .

(Photographing unit 170)
As described above, the imaging unit 170 according to the present embodiment is a function included when the information processing system according to the present embodiment is realized by one device, and photographs the user. Therefore, the photographing unit 170 includes an imaging device, a focus ring, a zoom lens, and the like. A photograph taken by the photographing unit 170 is converted into digital data and stored in the storage unit 110 . If the image of the user is captured by another device, the information processing device 100 does not need to include the image capture unit 170 .

(control unit 180)
The control unit 180 according to the present embodiment is a processing unit that controls the entire information processing apparatus 100 and controls each configuration included in the information processing apparatus 100 . Details of the functions of the control unit 180 will be described later.

The functional configuration example of the information processing apparatus 100 according to the present embodiment has been described above. Note that the functional configuration described above with reference to FIG. 2 is merely an example, and the functional configuration of the information processing apparatus 100 according to the present embodiment is not limited to the example. For example, the information processing apparatus 100 does not necessarily have to include all of the configurations shown in FIG. The functional configuration of the information processing apparatus 100 according to this embodiment can be flexibly modified according to specifications and operations.

In addition, the function of each component is stored in a ROM (Read Only Memory) or a RAM (Random Access Memory) that stores a control program that describes the processing procedure for an arithmetic unit such as a CPU (Central Processing Unit) to realize these functions. A control program may be read out from a storage medium, and the program may be interpreted and executed. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at which the present embodiment is implemented. Also, an example of the hardware configuration of the information processing apparatus 100 will be described later.

<<1.3. Function details >>
Next, functions of the information processing apparatus 100 according to this embodiment will be described in detail. One of the characteristics of the control unit 180 of the information processing apparatus 100 according to the present embodiment is that it acquires a user image, determines the user's skeleton type using a learning model, and presents recommended coordination.

First, the learning model used by the information processing apparatus 100 will be described. FIG. 3 is a diagram showing an example of a learning model according to this embodiment. As shown in FIG. 3, the learning model according to this embodiment is, for example, CNN. When an image set 400, which is one image obtained by combining a plurality of images of a user, is input to the CNN, feature amount extraction layers (convolution layers) 500-1 to 500-5 (collectively, "feature amount extraction layers 500 ), the visual features of the image are structured into semantic features. Note that the feature quantity extraction layer 500 in FIG. 3 is an example, and the depth of the layers is not limited to five layers. Then, the feature amount output from the feature amount extraction layer 500 is input to the fully connected layer 600, and the resulting skeleton type of the user is output. In the example of FIG. 3, the output result indicates "S: 0.8, W: 0.1, N: 0.1", so the determining unit 140 selects the straight (S) with the highest probability from the user can be determined as the skeleton type of However, as shown in FIG. 3, an error may occur between the output result and the skeleton type correct data "S: 1.0, W: 0.0, N: 0.0". . Therefore, the learning unit 130 causes the fully connected layer 600 to learn the generated error by transfer learning so that such an error approaches 0 as much as possible.

In general transfer learning, as shown in FIG. 3, an image set 400 is input to a learning model, and the fully connected layer 600 learns the error between the output result and the correct data (the latter half of the fully connected layer 600 (There are also cases where learning is performed by including only a part of the feature amount extraction layer 500). However, it is extremely wasteful to repeat the processing of the feature quantity extraction layer 500 each time the fully connected layer 600 is learned. Therefore, in this embodiment, the learning model is separated into a feature amount extraction layer 500 and a fully connected layer 600, and the feature amount output from the feature amount extraction layer 500 is used for learning of the fully connected layer 600. FIG.

FIG. 4 is a diagram showing an example of processing of the feature quantity extraction layer 500 according to this embodiment. FIG. 4 shows processing of the separated feature quantity extraction layer 500 . As shown in FIG. 4, an image set 400 input to the feature quantity extraction layer 500 outputs a feature quantity 700 structured from visual features to semantic features. By using this feature quantity 700 during learning of the fully connected layer 600, the processing of the feature quantity extraction layer 500 can be performed only once, eliminating the need for repeated processing.

FIG. 5 is a diagram showing an example of learning processing of the fully connected layer 600 according to this embodiment. FIG. 5 shows the training process of the isolated fully connected layer 600. As shown in FIG. The learning process evaluates the error, adjusts the weight of the fully connected layer 600, and inputs the feature quantity 700 output from the feature quantity extraction layer 500 (shown in FIG. 4) to the fully connected layer 600. . This eliminates the need to repeat the processing of the feature quantity extraction layer 500 each time the fully connected layer 600 is learned.

In the learning process shown in FIGS. 3 to 5, since the image set 400, which is a single image obtained by combining a plurality of images of the user, is learned, the learning time is shortened and the learning process is faster than learning each of the plurality of images. It is possible to reduce the load. However, there is a possibility that image deterioration will occur due to the combination of a plurality of images. Therefore, as another example of the learning process according to the present embodiment, a method of making a learning model learn each of a plurality of images will be described.

FIG. 6 is a diagram showing another example of processing of the feature quantity extraction layer 500 according to this embodiment. In the example of FIG. 6, a front image 400-1, a back image 400-2, a side image 400-3, and a hand image 400-4 are input to the feature extraction layer 500 as multiple images of the user. do. Then, as shown in FIG. 6, the feature amount extraction layer 500 outputs feature amounts 700-1 to 700-4 for each input image. Since these feature amounts 700-1 to 700-4 are output using the plurality of images 400-1 to 400-4 of the user as they are, they are not affected by data deterioration due to image combination. .

FIG. 7 is a diagram showing another example of learning of the fully connected layer 600 according to this embodiment. Next, the feature quantities 700-1 to 700-4 output from the feature quantity extraction layer 500 are combined to generate one feature quantity 700. FIG. The combination of the feature quantities 700-1 to 700-4 is different from the combination of images, and is merely a combination of enumerated numerical data, so data deterioration does not occur. Then, by using this combined feature amount 700 when learning the fully connected layer 600, it is not necessary to repeat the processing of the feature amount extraction layer 500 each time the fully connected layer 600 is learned, and data deterioration due to image combination can be prevented. without being affected by

Next, the skeleton type of the user determined using the learning model and presentation of recommended coordination to the user based on the skeleton type will be described. The information processing device 100 transmits the user's skeleton type and recommended coordination to the user terminal 200 and the store terminal 300 . The user terminal 200 and the store terminal 300 receive the user's skeleton type and recommended coordination, and present them to the user via the UI.

FIG. 8 is a diagram showing an example of a user interface according to this embodiment. FIG. 8 is an example of a UI displayed on the user terminal 200 or the store terminal 300 for presenting the user's frame type and recommended coordination to the user. The UI may be an application downloaded from the information processing device 100 or the like and installed in the user terminal 200 or the store terminal 300, or may be a web application on the information processing device 100. FIG.

As shown in FIG. 8, the user terminal 200 and the store terminal 300 display the skeleton type of the user, commentary on the skeleton type, suitable materials and patterns, and recommended coordination suitable for the user's skeleton type as a result of skeleton diagnosis. be able to. User terminal 200 and store terminal 300 can also display sales information of clothing, furnishings, and accessories searched for recommended coordination. Furthermore, for each sales information, the user terminal 200 and the store terminal 300, for example, by being selected by a touch operation by the user, provide information on stores handling selected clothing items, sales floor guidance, and online shops. can be displayed. As a result, it is possible to promote sales of clothing, accessories, and accessories searched for recommended coordination.

<<1.4. Function Flow>>
Next, the procedure of the human learning model generation process according to the present embodiment will be described with reference to FIG. 9 . FIG. 9 is a flowchart showing the flow of learning model generation processing according to this embodiment. This process is executed by the information processing apparatus 100 to generate a learning model for determining a person's skeleton type.

First, as shown in FIG. 9, the acquisition unit 120 of the information processing apparatus 100 acquires an image of a person to be used as teacher data for a learning model (step S101). The images are, for example, a plurality of image sets 400 showing parts used for skeletal diagnosis, such as whole body, half body, and hands. Such images may be taken for learning processing, or existing images accumulated in ImageNet or the like can be used.

Next, the obtaining unit 120 obtains the skeleton type of the person appearing in the image obtained in step S101 (step S102). The skeletal type is a skeletal type determined in advance by an expert by diagnosing the person directly or by viewing an image. Note that the execution order of steps S101 and S102 may be reversed.

Next, the learning unit 130 of the information processing apparatus 100 receives the image of the person acquired by the acquisition unit 120, learns the teacher data that corrects the skeletal type of the person, and generates a learning model. (Step S103). At this time, in order to learn the learning model by transfer learning, the learning model is separated into the feature amount extraction layer 500 and the fully connected layer 600, and the feature amount 700 output from the feature amount extraction layer 500 is stored in the storage unit. 110 can be stored.

After step S103, steps S101 to S103 are repeated for the number of persons to be learned, and the process ends.

Next, the procedure of the person learning process according to this embodiment will be described with reference to FIG. 10 . FIG. 10 is a flowchart showing the flow of learning processing according to this embodiment. This process is executed by the information processing apparatus 100 to evaluate the error between the result output from the learning model for determining the skeleton type of the person and the correct answer, and feed back the error to learn the learning model. .

First, as shown in FIG. 10, the acquisition unit 120 of the information processing apparatus 100 acquires feature amount data (feature amount 700) output from the feature amount extraction layer 500 of the learning model generated in step S103 (step S201). The feature amount 700 is the same as long as the image input to the learning model does not change. Therefore, when repeated learning is performed using the error between the output result and the correct answer, the feature amount 700 is input to the fully connected layer 600. Therefore, the processing of the feature quantity extraction layer 500 does not have to be repeated.

An error between the output result of the learning model generated in step 103 and the correct data is obtained (step S202). For example, the result output from the learning model is "S: 0.8, W: 0.1, N: 0.1", and the correct data is "S: 1.0, W: 0.0, N: 0.0”, there is an error of −0.2 for S and −0.1 for W and N. In order to improve the accuracy of the learning model, learning is performed so that each error approaches zero.

Next, the learning unit 130 of the information processing device 100 learns the fully connected layer 600 of the learning model using the error and the feature amount 700 acquired by the acquisition unit 120 (step S203). More specifically, after evaluating the error and adjusting the weight of the fully connected layer 600, the feature quantity 700 is input to the learning model. After that, the result output from the learning model and the correct data are evaluated, the learning of the fully connected layer 600 is repeatedly performed, and this process ends.

Next, with reference to FIG. 11, a procedure for determining a user's skeleton type according to this embodiment will be described. FIG. 11 is a flowchart showing the flow of skeleton type determination processing according to the present embodiment. This process is, for example, a process on the assumption that the user uses the user terminal 200 to receive a skeleton diagnosis at home. This process is triggered by, for example, the user accessing, via the user terminal 200, an application for performing skeleton diagnosis installed in the user terminal 200 or a web application on the information processing apparatus 100. FIG.

First, as shown in FIG. 11, the user terminal 200 transmits user information to the information processing apparatus 100 so that the information processing apparatus 100 recognizes the user who has accessed (step S301). The user information is, for example, login information for each user.

If the information processing apparatus 100 has not received user information from the user terminal 200 (step S302: No), it waits for reception of user information. On the other hand, if user information has been received (step S302: Yes), the search unit 150 of the information processing apparatus 100 searches for the target user based on the user information (step S303).

Next, as a result of the user search in step S303, if the target user has undergone a skeleton diagnosis, the search unit 150 of the information processing apparatus 100 searches for the target user's skeleton type, and the process proceeds to step S310.

On the other hand, if the target user has not undergone skeleton diagnosis, the transmitting unit 160 of the information processing apparatus 100 transmits a request for a user image to the user terminal 200 in order to perform skeleton diagnosis (step S305).

If the user terminal 200 has not received a user image request from the information processing apparatus 100 (step S306: No), the user terminal 200 waits to receive a user image request. On the other hand, if a user image request has been received (step S306: Yes), the user terminal 200 prompts the user to upload a user image by displaying a message to that effect on the screen of the user terminal 200, for example.

Next, when the user shoots a user image using the user terminal 200 or the like or selects a shot user image, the user terminal 200 uploads the user image to the information processing apparatus 100 (step S307).

If the user image has not been received from the user terminal 200 (step S308: No), the information processing apparatus 100 waits for reception of the user image. On the other hand, when the user image is received (step S308: Yes), the learning unit 130 of the information processing device 100 inputs the user image into the learning model, and the determining unit 140, based on the result output from the learning model, The skeleton type of the target user is determined (step S309).

Next, the search unit 150 searches for recommended coordination based on the skeleton type of the target user (step S310). At this time, the search unit 150 searches for user attributes such as the height, weight, age, gender, and activity area of the target user, and various information such as the current season. You can also search for In addition, it is possible to search for sales information (such as information on stores and online shops) associated with clothing items searched for as recommended coordination.

Next, the transmission unit 160 transmits information such as the skeleton type of the target user determined by the determination unit 140 and the recommended coordination retrieved by the search unit 150 to the user terminal 200 (step S311).

If the user terminal 200 has not received information such as the skeleton type from the information processing apparatus 100 (step S312: No), it waits for reception of information such as the skeleton type. On the other hand, when information such as the skeleton type is received (step S312: Yes), the user terminal 200 displays the information such as the skeleton type on the screen of the user terminal 200 (step S313). After step S313, this process ends. However, for example, the user accesses an online shop or store that sells recommended clothing or the like based on the sales information displayed along with the recommended coordination in step S313, and performs shopping or confirms details of the product. be able to.

Next, another procedure of the user's skeleton type determination process according to the present embodiment will be described with reference to FIG. 12 . FIG. 12 is a flowchart showing another flow of skeleton type determination processing according to the present embodiment. This processing is processing, for example, assuming that the user visits a store that sells clothing and the like and receives a skeleton diagnosis at the store using the store terminal 300 . This process is triggered by, for example, the fact that the user is detected by the store terminal 300 through face authentication or the like.

First, as shown in FIG. 12, the store terminal 300 transmits detection information of the detected user to the information processing device 100 (step S401).

If the information processing apparatus 100 has not received the user detection information from the store terminal 300 (step S402: No), the information processing apparatus 100 waits for reception of the user detection information. On the other hand, when user detection information is received (step S402: Yes), the search unit 150 of the information processing apparatus 100 searches for a target user based on the user detection information (step S403).

Next, as a result of the user search in step S403, if the target user has undergone a skeleton diagnosis, the search unit 150 of the information processing apparatus 100 searches for the target user's skeleton type, and the process proceeds to step S410.

On the other hand, if the target user has not undergone a skeleton diagnosis, the transmission unit 160 of the information processing apparatus 100 transmits a user image capturing request to the store terminal 300 in order to perform skeleton diagnosis (step S405).

If the store terminal 300 has not received a user image shooting request from the information processing apparatus 100 (step S406: No), it waits for reception of a user image shooting request. On the other hand, when a user image capturing request is received (step S406: Yes), the shop terminal 300 captures the user and uploads the user image to the information processing device 100 (step S407).

If the user image has not been received from the store terminal 300 (step S408: No), the information processing apparatus 100 waits for reception of the user image. On the other hand, when the user image is received (step S408: Yes), the learning unit 130 of the information processing device 100 inputs the user image into the learning model, and the determining unit 140, based on the result output from the learning model, The skeleton type of the target user is determined (step S409).

Next, the search unit 150 searches for recommended coordination based on the skeleton type of the target user (step S410). At this time, as in step S310, the search unit 150 can also search for recommended coordination and sales information (such as sales floor guides) associated with clothing items searched for as recommended coordination.

Next, the transmission unit 160 transmits information such as the skeleton type of the target user determined by the determination unit 140 and the recommended coordination retrieved by the search unit 150 to the store terminal 300 (step S411).

If information such as the skeleton type has not been received from the information processing apparatus 100 (step S412: No), the store terminal 300 waits for reception of information such as the skeleton type. On the other hand, if information such as the skeleton type has been received (step S412: Yes), the shop terminal 300 displays the information such as the skeleton type on the screen of the shop terminal 300 (step S413). After step S413, this process ends. However, for example, the user can confirm sales floors selling recommended clothing and the like from the sales information displayed accompanying the recommended coordination in step S413, and can confirm details of shopping and products.

<2. Hardware configuration example>
Next, a hardware configuration example of the information processing apparatus 100 according to this embodiment will be described. FIG. 13 is a block diagram showing a hardware configuration example of the information processing apparatus 100 according to this embodiment. Referring to FIG. 13, the information processing apparatus 100 includes, for example, a processor 801, a ROM 802, a RAM 803, a host bus 804, a bridge 805, an external bus 806, an interface 807, an input device 808, and an output device 809. , a storage 810 , a drive 811 , a connection port 812 , and a communication device 813 . Note that the hardware configuration shown here is an example, and some of the components may be omitted. Moreover, it may further include components other than the components shown here.

(processor 801)
The processor 801 functions, for example, as an arithmetic processing device or a control device, and controls the overall operation or part of each component based on various programs recorded in the ROM 802, the RAM 803, the storage 810, or the removable recording medium 901. .

(ROM802, RAM803)
The ROM 802 is means for storing programs to be read into the processor 801 and data used for calculations. The RAM 803 temporarily or permanently stores, for example, a program to be read into the processor 801 and various parameters that appropriately change when the program is executed.

(Host Bus 804, Bridge 805, External Bus 806, Interface 807)
The processor 801, ROM 802, and RAM 803 are interconnected via, for example, a host bus 804 capable of high-speed data transmission. On the other hand, the host bus 804 is connected, for example, via a bridge 805 to an external bus 806 with a relatively low data transmission speed. Also, the external bus 806 is connected to various components via an interface 807 .

(Input device 808)
For example, a mouse, keyboard, touch panel, button, switch, and lever are used as the input device 808 . Furthermore, as the input device 808, a remote controller (hereinafter referred to as a remote controller) capable of transmitting control signals using infrared rays or other radio waves may be used. The input device 808 also includes a voice input device such as a microphone.

(output device 809)
The output device 809 is, for example, a display device such as a CRT (Cathode Ray Tube), an LCD, or an organic EL, an audio output device such as a speaker or headphone, a printer, a mobile phone, a facsimile, or the like, and outputs the acquired information to the user. It is a device that can be notified visually or audibly. Also, the output device 809 according to this embodiment includes various vibration devices capable of outputting tactile stimulation.

(storage 810)
Storage 810 is a device for storing various data. As the storage 810, for example, a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like is used.

(Drive 811)
The drive 811 is, for example, a device that reads information recorded on a removable recording medium 901 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, or writes information to the removable recording medium 901 .

(connection port 812)
The connection port 812 is, for example, a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface), an RS-232C port, or a port for connecting an external connection device 902 such as an optical audio terminal. be.

(Communication device 813)
The communication device 813 is a communication device for connecting to a network. subscriber line) or a modem for various communications.

(Removable recording medium 901)
The removable recording medium 901 is, for example, DVD (registered trademark) media, Blu-ray (registered trademark) media, HD DVD media, various semiconductor storage media, and the like. Of course, the removable recording medium 901 may be, for example, an IC card equipped with a contactless IC chip, an electronic device, or the like.

(External connection device 902)
The externally connected device 902 is, for example, a printer, portable music player, digital camera, digital video camera, or IC recorder.

Note that the storage unit 110 according to this embodiment is realized by the ROM 802, the RAM 803, and the storage 810. FIG. Further, the control unit 180 according to the present embodiment realized by the processor 801 loads each control program realizing the acquisition unit 120, the learning unit 130, the determination unit 140, the search unit 150, and the imaging unit 170 from the ROM 802, the RAM 803, and the like. Read and execute. Further, the transmission unit 160 according to this embodiment reads data from the ROM 802, the RAM 803, etc., transmits the data to the communication device 813 via the host bus 804, the bridge 805, the external bus 806, and the interface 807, and transmits the data to the external device. send.

<3. Summary>
As described above, the information processing apparatus 100 receives the acquisition unit 120 that acquires a plurality of second images of a user, a plurality of first images of a person, and a first skeleton of the person. A learning unit 130 that learns teacher data whose type is the correct answer and generates a learning model, and based on the result output by inputting a plurality of second images into the learning model, the user's second skeleton type and a determination unit that determines the

As a result, it is possible to provide a learning model that allows the user to easily self-diagnose without undergoing skeleton diagnosis by an expert. In addition, since the provided learning model learns the diagnostic results of experts as correct data, it is possible to perform highly reliable skeletal diagnosis. In addition, skeletal diagnosis using a learning model is based on semantic features structured from visual features of images, compared to skeletal diagnosis in which an expert determines the skeletal type visually and by palpation based on physical characteristics. It differs significantly in that it derives the probability of being of a particular skeleton type.

Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

Also, the effects described herein are merely illustrative or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the present technology can also take the following configuration.
(1) an acquisition unit 120 for acquiring a plurality of first images showing a person and a first skeleton type of the person; and a learning unit 130 that learns teacher data to generate a learning model.

As a result, the information processing apparatus 100 can provide a learning model that allows the user to easily self-diagnose without undergoing skeleton diagnosis by an expert. Furthermore, although this is a self-diagnosis, the provided learning model learns the diagnostic results of experts as correct data, so that highly reliable skeletal diagnosis can be performed.

(2) The learning unit 130 inputs, as the first image, one image obtained by combining the plurality of first images, and learns teacher data that corrects the first skeleton type. An information processing apparatus 100 characterized by generating a model.

As a result, the information processing apparatus 100 can shorten the learning time and reduce the load of the learning process by having the learning model learn each of the plurality of first images.

(3) The learning unit 130 further includes the feature amount output from the feature amount extraction layer of the learning model, the result output from the fully connected layer of the learning model, and the correct data of the first skeleton type. An information processing apparatus 100 that performs learning of the learning model using the error of .

As a result, the information processing apparatus 100 can shorten the learning time and reduce the load of the learning process by performing processing from the feature amount extraction layer of the learning model.

(4) The learning unit 130 inputs the plurality of first images as the first images, learns teacher data that corrects the first skeleton type, and generates the learning model. An information processing apparatus 100 characterized by:

As a result, the information processing apparatus 100 can generate a learning model without causing image deterioration that may occur due to combining a plurality of first images.

(5) The learning unit 130 further includes one feature value obtained by combining a plurality of feature values output from the feature value extraction layer of the learning model, a result output from the fully connected layer of the learning model, and the An information processing apparatus 100, characterized in that learning of the learning model is performed using errors from correct data of the first skeleton type.

As a result, the information processing apparatus 100 can shorten the learning time and reduce the load of the learning process by performing processing from the feature amount extraction layer of the learning model.

(6) The acquiring unit 120 acquires, as the plurality of first images, at least two of a first front image, a first rear image, and a first side image of the person. The information processing apparatus 100 according to any one of (1) to (5) above.

As a result, the information processing apparatus 100 can cause the learning model to learn an image showing a part of the body that is more likely to have features when skeletal diagnosis is performed. As a result, it is possible to provide a learning model that enables more accurate skeletal diagnosis.

(7) The obtaining unit 120 obtains, as the plurality of first images, at least two images of the person's clavicle, leg to waist, chest, and hand parts. The information processing apparatus 100 according to any one of (1) to (6) above.

As a result, the information processing apparatus 100 can cause the learning model to learn an image showing a part of the body that is more likely to have features when skeletal diagnosis is performed. Moreover, as a result, it is possible to provide a learning model capable of performing more accurate skeletal diagnosis.

(8) Acquisition unit 120 for acquiring a plurality of second images of a user, input of a plurality of first images of a person, and training data that corrects the first skeletal type of the person. and a determination unit 140 that determines the second skeleton type of the user based on the result output by inputting the plurality of second images into the learning model generated by learning. and the information processing apparatus 100.

As a result, the information processing apparatus 100 can provide a learning model that allows the user to easily self-diagnose without undergoing skeleton diagnosis by an expert. Furthermore, although this is a self-diagnosis, the provided learning model learns the diagnostic results of experts as correct data, so that highly reliable skeletal diagnosis can be performed.

(9) The acquisition unit 120 acquires at least two of a second front image, a second rear image, and a second side image of the user as the plurality of second images. The information processing apparatus 100 according to (8) above.

As a result, the information processing apparatus 100 can cause the learning model to learn an image showing a part of the body that is more likely to have features when skeletal diagnosis is performed. As a result, it is possible to provide a learning model capable of performing more accurate skeletal diagnosis.

(10) The obtaining unit 120 obtains, as the plurality of second images, at least two images of the user's clavicle, leg to waist, chest, and hand parts. The information processing apparatus 100 according to (8) or (9) above.

As a result, the information processing apparatus 100 can cause the learning model to learn an image showing a part of the body that is more likely to have features when skeletal diagnosis is performed. Moreover, as a result, it is possible to provide a learning model capable of performing more accurate skeletal diagnosis.

(11) The apparatus according to any one of (8) to (10) above, further comprising a search unit 150 that searches for recommended coordination for the user based on the second skeleton type. Information processing device 100 .

Thereby, the information processing apparatus 100 can present to the user recommended coordination suitable for the user's skeleton type.

(12) The search unit 150 further searches for sales information of at least one of clothing, furnishings, accessories, and materials and patterns that suit the user, which are searched as the recommended coordination. The information processing apparatus 100 according to (11).

As a result, the information processing apparatus 100 can promote sales of the clothing items, furnishings, and accessories searched for as recommended coordination.

(13) The information processing apparatus 100 acquires a plurality of first images showing a person and a first skeleton type of the person, inputs the plurality of first images, and obtains the first skeleton type. A method characterized by executing a process of learning correct teacher data and generating a learning model.

(14) acquiring a plurality of first images of a person and a first skeleton type of the person in the information processing apparatus 100, inputting the plurality of first images, and acquiring the first skeleton type; A program characterized by executing a process of learning correct teacher data and generating a learning model.

(15) The information processing apparatus 100 acquires a plurality of second images in which the user is photographed, inputs a plurality of first images in which a person is photographed, and the teacher determines that the first skeleton type of the person is correct. characterized by executing a process of determining the second skeleton type of the user based on a result output by inputting the plurality of second images into a learning model generated by learning data. how to.

(16) A teacher who obtains a plurality of second images of a user, inputs a plurality of first images of a person, and determines that the first skeleton type of the person is correct, to the information processing apparatus 100. characterized in that a process of determining the second skeleton type of the user is executed based on a result output by inputting the plurality of second images into a learning model generated by learning data. program to do.

100 information processing device 110 storage unit 120 acquisition unit 130 learning unit 140 determination unit 150 search unit 160 transmission unit 170 imaging unit 200 user terminal 300 store terminal

Claims (14)

An image of a person, which is learned using a plurality of first images of a first person as input data and teacher data having a correct label of the first skeleton type of the first person. When the plurality of first images are input to the learning model, the learning model is output from the feature amount extraction layer of the learning model, and each of the plurality of first images is output from the feature amount extraction layer. Using one feature value obtained by combining a plurality of feature values corresponding to each of the images of the learning model, the probability that it is the second skeleton type output from the fully connected layer of the learning model, and the first skeleton type inputting a plurality of second images of a user to the learning model learned by updating the fully connected layer with
An information processing apparatus, comprising: a control unit that determines the skeleton type of the user based on the probability of being each of the skeleton types output from the learning model. the plurality of first images includes at least two of a first front image, a first rear image, and a first side image of the first person;
The control unit controls, as the plurality of second images, a second front image, a second rear image, and a second rear image of the user in which the same part as the first image used as the teacher data is shown. 2. The information processing apparatus according to claim 1, wherein at least two of two side images are input to said learning model. The plurality of first images includes at least two images of a clavicle, legs to waist, chest, and hands of the first person;
The control unit selects, as the plurality of second images, at least images of the user's clavicle, legs to waist, chest, and hands showing the same part as the first image used as the teacher data. 2. The information processing apparatus according to claim 1, wherein two are input to said learning model. 4. The method according to any one of claims 1 to 3, wherein the control unit searches for recommended coordinates associated with the skeleton type, which are stored in advance, using the determined skeleton type as a search key. The information processing device according to any one of the items. The recommended coordination includes at least one of clothing, furnishings, accessories, and materials and patterns that suit the user,
5. The control unit according to claim 4, wherein the control unit further executes a process of searching sales information associated with the recommended coordination , which is stored in advance , using the searched recommended coordination as a search key. information processing equipment. 2. The method according to claim 1, wherein the control unit learns the learning model using the plurality of first images as input data and teacher data having the first skeleton type as a correct label. Information processing equipment. An image of a person, which is learned using a single image obtained by combining a plurality of first images of a first person as input data , and teacher data having the first skeleton type as a correct label. A learning model for determining a skeleton type from a feature output from a feature amount extraction layer of the learning model when one image obtained by combining the plurality of first images is input to the learning model to the learning model learned by updating the fully connected layer using the quantity, the probability of being the second skeleton type output from the fully connected layer of the learning model, and the first skeleton type , inputting one image obtained by combining a plurality of second images in which the user is captured;
determining the skeletal type of the user based on probabilities of being each of the skeletal types output from the learning model;
An information processing apparatus comprising a control section for executing processing . 8. The method according to claim 7 , wherein the control unit learns the learning model using each of the plurality of first images as input data and teacher data having the first skeleton type as a correct label. The information processing device described. the plurality of first images includes at least two of a first front image, a first rear image, and a first side image of the first person;
The plurality of second images are a second front image, a second rear image, and a second side image of the user in which the same part as the first image used as the training data is shown. 8. An information processing apparatus according to claim 7 , comprising at least two . The plurality of first images include at least two images of the first person's clavicle, leg to waist, chest, or hand part,
The plurality of second images include at least two images of the user's clavicle, legs to waist, chest, and hands, which show the same parts as the first images used as the training data. 8. The information processing apparatus according to claim 7 . The information processing device
An image of a person, which is learned using a plurality of first images of a first person as input data and teacher data having a correct label of the first skeleton type of the first person. When the plurality of first images are input to the learning model, the learning model is output from the feature amount extraction layer of the learning model, and each of the plurality of first images is output from the feature amount extraction layer. Using one feature value obtained by combining a plurality of feature values corresponding to each of the images of the learning model, the probability that it is the second skeleton type output from the fully connected layer of the learning model, and the first skeleton type inputting a plurality of second images of a user to the learning model learned by updating the fully connected layer with
determining the skeletal type of the user based on probabilities of being each of the skeletal types output from the learning model. The information processing device
12. The method according to claim 11 , wherein the learning model is trained using the plurality of first images as input data and teacher data having the first skeleton type as a correct label. . information processing equipment,
An image of a person, which is learned using a plurality of first images of a first person as input data and teacher data having a correct label of the first skeleton type of the first person. When the plurality of first images are input to the learning model, the learning model is output from the feature amount extraction layer of the learning model, and each of the plurality of first images is output from the feature amount extraction layer. Using one feature value obtained by combining a plurality of feature values corresponding to each of the images of the learning model, the probability that it is the second skeleton type output from the fully connected layer of the learning model, and the first skeleton type inputting a plurality of second images of a user to the learning model learned by updating the fully connected layer with
A program for determining the skeleton type of the user based on the probability of being each of the skeleton types output from the learning model. information processing equipment,
14. The program according to claim 13 , causing execution of a process of learning the learning model using the plurality of first images as input data and teacher data having the first skeleton type as a correct label. .

Images