JP7673544B2 - Skin condition analysis device and system - Google Patents

Description

The present invention relates to a device that analyzes the condition of a user's facial skin and provides the user with useful beauty and medical information.

2. Description of the Related Art Support systems that provide users with information aimed at promoting beauty are known.
This support system uses image processing technology and is configured to, for example, evaluate the condition of the user's skin and scalp from photographic data of the user and notify them of information regarding beauty promotion (see, for example, Patent Document 1).

JP 2018-97899 A

Incidentally, with regard to the application of systems using the above-mentioned image processing technology to medical use, their use in diagnosing skin diseases, such as skin cancer (melanoma), where it is particularly difficult to visually distinguish between diseased areas and normal areas such as blemishes and moles, has not yet been addressed.
It is said that approximately 320,000 people worldwide are affected by skin cancer each year. The mortality rate for melanoma is high at 18%, compared to 5% for non-melanoma. While it is difficult to realize that you have skin cancer, the disease progresses very quickly.

The present invention has been made in consideration of the above-mentioned circumstances, and the problem to be solved is to propose a new device and system for analyzing and diagnosing skin conditions, from the viewpoint of achieving both diagnostic accuracy and early detection of diseased areas, particularly as a preventive measure against skin cancer.

In order to solve the above problems, the present invention provides a skin condition analysis device to which a device for acquiring a face image of a user and a device for acquiring a skin image of the user are connected,
an image recording means for recording the face image;
a means for comparing the acquired face image with a past face image recorded in the image recording means to determine a skin condition of the entire face;
a means for generating beauty information such as a makeup method suitable for the user based on the result of the judgment and notifying the user of the information;
A specific skin disease analysis means for analyzing the possibility of occurrence of a specific skin disease based on the result of the determination;
a means for requesting a user to acquire a skin image of a local area of the face based on the analysis result;
a skin condition diagnosis means for diagnosing a skin image acquired by a user in response to the request;
and means for notifying a user of the diagnosis result by the skin condition diagnosis means.
In addition to the above-mentioned means, the skin condition diagnostic device is characterized by comprising a means for notifying a medical department of the diagnosis result by the skin condition diagnostic means and requesting a detailed diagnosis, and a means for notifying a user of the diagnosis result by the medical department.
The device is also characterized in that it is a device for diagnosing skin cancer as the specific skin disease.

According to the skin condition analysis device of the present invention, an image of the user's bare face before makeup is obtained and recorded daily as a face image, and by comparing the obtained face image with the recorded past face image, the user's current skin condition can be determined. From the results of this determination, beauty information such as the user's preferred makeup method can be provided to the user, facilitating daily beauty promotion.
Furthermore, based on the judgment results, the possibility that the user has a specific skin disease such as skin cancer is analyzed, and if there is a possibility, the user is prompted to take a skin image of a local area on the face, and a detailed skin condition is diagnosed from the taken skin image, making it possible to determine whether or not the user has a skin disease before he or she even realizes it.
In this case, the diagnosis result by the skin condition diagnosis means is notified to the medical department, and a detailed diagnosis is made by a specialist, so that an accurate diagnosis result can be obtained.
If a user is diagnosed with a disease, the user can be notified and early treatment can be provided, thereby contributing to the maintenance of the user's health.

In the skin condition analysis device having the above configuration, it is preferable that the beauty information generation means and the specific skin disease analysis means include a model generation means using deep learning by AI.
In other words, it is preferable that the beauty information generation means has a model generation means that uses actual measurements of a person's facial contour and the position and shape of the eyes, nose, mouth, and eyebrows as training data and generates an estimation model through deep learning using AI that will improve the appearance of the user's face when made up based on the skin condition of the user's entire face.
In addition, it is preferable that the specific skin disease analysis means has a model generation means that uses actual measured values such as the size, shape, color, and location of the affected area obtained from an image of the skin disease as teacher data and generates an estimated model of a skin image that is likely to be a skin disease by deep learning using AI.
Since the beauty information generation means and the specific skin disease analysis means have a model generation means using deep learning with AI, it is possible to automatically generate beauty information such as makeup methods suitable for the user and selection of makeup to use, and it is also possible to automatically perform a process of determining whether or not the user is suffering from a specific skin disease with high accuracy. By notifying the user of the generated beauty information and the diagnosis results of whether or not the user is suffering from a skin disease, it is possible to contribute to promoting the user's beauty and maintaining their health.

In the skin condition analyzing device having the above-described configuration, the face image acquiring means can use a smart mirror.
It is preferable to use a smart mirror that is large enough to capture a life-size image of the entire face or the entire body.
The skin image acquisition means may be a smartphone.

One of the purposes of the skin condition analysis device configured as above is to integrate the skin condition analysis and diagnostic results into the user's daily life without the user having to install new equipment, and it can be configured to be used in the user's private home as well as in commercial facilities such as beauty salons, fitness clubs, homes, cosmetics sales corners and restrooms in department stores. In this case, the smart mirror is installed in any of the above facilities, and the smart mirror is configured to acquire a facial image of the user, which is then sent to a processing device connected to the smart mirror via a communication line for processing.

1 is a diagram showing a configuration of a skin condition analyzing system according to an embodiment of the present invention. 2 is a diagram for explaining a schematic configuration of a data processing unit of the skin condition analyzing device. FIG. 1 shows face images captured by a smart mirror, where (A) is a face image captured and recorded in the past, and (B) is a face image captured recently. FIG. 13 is a diagram showing a skin image acquired by a smartphone.

A preferred embodiment of the present invention will be described below with reference to the drawings. Note that the present invention is not limited to the embodiment described below.

The present invention is intended to obtain an image of a user's face, analyze the condition of the facial skin based on the image, and further diagnose the occurrence of skin diseases, thereby providing the user with useful beauty information and disease diagnosis information.
As shown in Fig. 1, a skin condition analysis system according to one embodiment of the present invention (hereinafter, also referred to as "this system") includes a smart mirror 2, a smartphone 3 with a camera function, and a skin condition analysis device 4 which is a data processing server, and is configured by connecting the smart mirror 2 and smartphone 3 to the skin condition analysis device 4 via a communication line 5. In the figure, reference numeral 6 denotes an information processing terminal used by a specialist doctor in the medical department.

The smart mirror 2 is a means for acquiring a facial image of the user, and is equipped with a display unit having a function of displaying electronic information on the mirror surface reflecting the user, a camera unit for photographing the user facing the display unit, and a communication unit for connecting to a communication line 5. The camera unit photographs the user's face to acquire a facial image, which is then transmitted to the skin condition analysis device 4. The smart mirror 2 is also configured to be able to display and output information notified from the skin condition analysis device 4 as electronic information on the display unit.
The smart mirror 5 may be equipped with a microphone, a speaker, and a voice recognition means, and information processing such as obtaining a face image may be performed through voice recognition processing by conversation between the smart mirror 5 and the skin condition analyzer 4.

The smartphone 3 is a means for acquiring an image of the user's facial skin, and is configured to transmit the skin image captured by the built-in camera to the skin condition analysis device 4 via the communication line 5, and also to receive information notified from the skin condition analysis device 4 and display it on the display.
Communication between the smartphone 3 and the skin condition analysis device 4 may be performed by installing a dedicated application related to skin condition analysis processing on the smartphone 3 and executing this application, or by starting a web browser on the smartphone 3 to access the skin condition analysis device 4 via the communication line 5 and send and receive data.

The skin condition analysis device 4 is a computer having a function of receiving image data transmitted from the smart mirror 2 and smartphone 3 connected via a communication line 5, a function of generating information relating to the user's skin condition by recording and analyzing the received image data, and a function of outputting and providing the generated information, files, and other data to the recipient via the communication line 5, and is equipped with various processing units, such as a data input unit such as a keyboard and a display unit such as a monitor that displays data, a communication unit that transmits and receives data via the communication line 5, a memory unit that stores and keeps various data, files, and processing programs, an external storage medium reading unit that reads data, files, and programs stored in an external storage medium and inputs them into the storage unit, and a control unit that controls the various units in accordance with the processing program stored in the storage unit.
The skin condition analyzer 4 has a function as a web server that transmits web information to the communication line 5, and is configured to be able to send and receive files, data, etc. between the smartphone 3 with a web browsing function and the information processing terminal 6 in the medical department via a website set up on the communication line 5, or by sending and receiving data via e-mail or other text communication apps, or by sending and receiving information via a calling app.
The skin condition analysis device 4 is not limited to being a standalone computer or server, but can be configured as a collection of multiple distributed computers or servers, or multiple servers installed on a cloud, and can be configured so that each computer or server works together to perform processing to analyze the user's skin condition.

As shown in FIG. 2, the system 1 constitutes a platform for beauty information processing and a platform for medical information processing, and is set up so that the information is processed by controlling each processing unit of the skin condition analyzer 4, such as the data input/output unit, memory unit, and analysis unit, using a processing program.

In more detail, the hardware and software included in the skin condition analysis device 4 constitute an information processing means, specifically at least an image recording means for recording a face image acquired by the smart mirror 2 and a skin image acquired by the smartphone 3, a means for judging the skin condition of the entire face by comparing the acquired face image with a past face image recorded in the image recording means, a means for generating beauty information such as a makeup method suitable for the user based on the judgment result and notifying the user, a specific skin disease analysis means for analyzing the possibility of the occurrence of a specific skin disease based on the judgment result, a means for requesting the user to acquire a skin image of a local area of the face based on the analysis result, a skin condition diagnosis means for diagnosing the skin image acquired by the user upon request, a means for notifying a medical department of the diagnosis result by the skin condition diagnosis means and requesting a detailed diagnosis, and a means for notifying the user of the diagnosis result of the skin condition or the diagnosis result of the medical department, and these are used to process information related to beauty and medicine.

The service of providing beauty and medical information to users using this system 1 goes through the following procedures: acquiring a face image, determining skin condition, analyzing skin disease, acquiring a skin image, and reviewing the information in the medical department.
In the following, a case will be described in which the smart mirror 2 is installed in a sports club that has authorized its members to use the system, and the members use the smart mirror 2.

Acquiring face images
First, a smart mirror 2 installed in a sports club is connected to a skin condition analyzer 4, and the bare face of a member who is a user, before makeup is reflected in the smart mirror 2, photographed, and transmitted as a face image to the skin condition analyzer 4. The skin condition analyzer 4 records the received face image by linking it to the user's registration information, information on the date and time of photographing, etc.

<Skin condition assessment>
The skin condition of the member is judged by comparing face images acquired in the past and recorded in the skin condition analyzer 4 with a recently acquired face image.
The judgment process is performed automatically according to a request from a member who operates the information input interface of the smart mirror 2 when the face image is acquired, or according to a set period such as once a week, and the skin condition analyzer 4 reads out a face image 2P1 acquired in the past by the smart mirror 2 as shown in FIG. 3(A) from the image recording means, and compares it with a face image 2P2 acquired by the smart mirror 2 as shown in FIG. 3(B) to judge the skin condition of the entire face. Both images are subjected to image processing so that blemishes, wrinkles, pores, skin color, etc. on the skin surface are made clear, and the extent to which the current skin differs from the past skin is judged. The judgment is made by visualizing the change in the skin condition, for example, by expressing it as an image or quantifying it as a score, and recording this as the judgment result in the recording means in association with the member's registration information, and displaying and outputting it on the smart mirror 2 to notify the member who has just acquired the face image.
Additionally, beauty information such as makeup techniques suitable for the member is generated based on the judgment results, and is displayed and output on the smart mirror 2 to inform the member.
In addition, the beauty information generation means has a model generation means that generates an estimated model using deep learning with AI to generate an estimate model of how the member's face will look better when made up based on the member's overall facial skin condition, and beauty information is generated based on this estimate model.
The judgment results and beauty information may be notified to a smartphone 3 carried by the member.

Analysis of skin diseases
When the skin condition is judged to be poor or abnormal, a specific skin disease analyzing means analyzes whether or not a specific skin disease may occur based on the judgment.
For example, as shown in FIG. 3(B), if a large area of pigmentation that was not previously present is found above the eyebrows of a member's face (shown in the dashed rectangular frame in the figure), this area will be analyzed to determine whether it is affected by a disease such as skin cancer.
This specific skin disease analysis means has a model generation means that generates an estimated model of skin images that are likely to be skin diseases based on images of skin diseases registered in an external medical database, etc., using deep learning with AI, and analyzes whether areas such as the pigmentation are areas of skin diseases based on this.
If the analysis reveals that the member is not suffering from a skin disease and that there is no abnormality, the member's smartphone 3 or the like is notified of this fact.

Acquiring skin images
In the analysis of the specific skin disease, the analysis is performed based on the face image 2P2 in FIG. 3(B). However, when it is necessary to perform the analysis using a high-precision image because the image of the relevant area is unclear, the member is requested to obtain a magnified image of the area such as the pigmentation.
The request is made by a notification from the skin condition analysis device 4 to the member's smartphone 3, and upon receiving the notification, the member operates the smartphone 3 to take a picture of the area above the eyebrows on their face, as shown in Figure 4, and sends this to the skin condition analysis device 4 as a skin image 3P.
The skin condition analyzer 4 records the received skin image 3P in the image recording means in association with the member's registration information, and diagnoses whether or not the member has a specific skin disease using the skin condition diagnosis means. The skin condition diagnosis means is configured similarly to the specific skin disease analysis means.
When the diagnosis indicates that the member does not have a skin disease, the member's smartphone 3 is notified of this fact, and when the diagnosis indicates that the member is highly likely to have a skin disease, the member is encouraged to see a doctor for an early diagnosis.

<Considerations in the medical sector>
The diagnosis of whether or not the member has a specific skin disease may be made by having the member undergo a diagnosis by a specialist in the medical department upon receipt of the diagnosis result from the skin condition diagnosis means, or upon request of a member who desires a more accurate diagnosis.
In this case, the skin condition diagnosis device 4 sends a notification to the medical department's information processing terminal 6 requesting a diagnosis by a specialist, and the specialist diagnoses whether or not the member has a skin disease based on the image of the member's skin condition sent from the skin condition diagnosis device 4 to the information processing terminal 6 and the AI diagnosis results by the judgment, analysis and diagnosis means.
The diagnostic results are notified from the information processing terminal 6 to the skin condition diagnosis device 4, which then links them to the member's registration information, registers them, and notifies the member's smartphone 4. The member can then use the notified diagnostic results for subsequent treatment and disease prevention measures.

In the embodiment, a smart mirror is used as the face image acquisition means, and a smartphone is used as the skin image acquisition means, but other information processing terminals with the function of taking and recording images and transmitting and receiving them as digital signals may also be used. In addition, this system may be used for diagnosing diseases other than specific skin diseases, such as skin cancer.

1 Skin condition diagnosis system (this system), 2 Smart mirror, 2P1, 2P2 File image, 3 Smartphone, 3P Skin image, 4 Skin condition diagnosis device, 5 Communication line, 6 Information processing terminal in the medical department

Claims (9)

A skin condition analysis device in which a face image acquisition means for acquiring a face image of a user and a skin image acquisition means for acquiring a skin image of the user are connected,
an image recording means for recording the face image;
a means for comparing the acquired face image with a past face image recorded in the image recording means to determine a skin condition of the entire face;
a beauty information generating means for generating beauty information for the convenience of the user in promoting beauty based on the result of the judgment;
A means for notifying a user of the beauty information ;
A specific skin disease analysis means for analyzing the possibility of occurrence of a specific skin disease based on the result of the determination;
a means for requesting a user to acquire a skin image of a local area of the face based on the analysis result;
a skin condition diagnosis means for diagnosing a skin image acquired by a user in response to the request;
and means for notifying a user of a diagnosis result by said skin condition diagnosis means. A means for notifying a medical department of a diagnosis result by the skin condition diagnosis means and requesting a detailed diagnosis;
The skin condition analyzing device according to claim 1 , further comprising a means for notifying a user of the diagnosis results from the medical department. The skin condition analyzer according to claim 1 or 2, which diagnoses skin cancer as the specific skin disease. The skin condition analysis device according to any one of claims 1 to 3, characterized in that the beauty information generation means and the specific skin disease analysis means include a model generation means of deep learning using AI. A skin condition analysis device according to any one of claims 1 to 4, in which the face image acquisition means is a smart mirror. The skin condition analysis device according to claim 5, wherein the smart mirror is sized to be capable of capturing a life-size image of the entire face or body. A skin condition analysis device according to any one of claims 1 to 6, in which the skin image acquisition means is a smartphone. The skin condition analyzer according to any one of claims 1 to 7 can be used in beauty salons, fitness clubs, homes, and commercial facilities equipped with cosmetics sales corners and restrooms in department stores. A skin condition analysis system comprising a skin condition analysis device according to any one of claims 1 to 8, a smart mirror as a face image acquisition means, and a smartphone as a skin image acquisition means.

Images