JP7528555B2 - Medical diagnosis support device, medical diagnosis support method, and program - Google Patents

Description

The present invention relates to a medical diagnosis support device, a medical diagnosis support method, and a program.

Previously, technology has been proposed for gait analysis methods that effectively show subjects the relationship between their walking state and the condition of their knees.

For example, a known gait analysis method includes an acquisition step of acquiring acceleration data of the subject's lower back while walking, and an estimation step of estimating the subject's subjective evaluation score for the knee using statistical information on gait sample data measured in advance from a population and sample data of subjective evaluation scores for the knee obtained in advance from the population, and the lower back acceleration data acquired in the acquisition step (for example, Patent Document 1).

JP 2018-038753 A

The technology described in Patent Document 1 is a technology that effectively presents to a subject the relationship between their walking state and the condition of their knee. However, because the technology described in Patent Document 1 estimates a subject's subjective evaluation score regarding their knee, it is difficult to present candidates for the disease that the subject may suffer from in advance to the subject.

The present invention aims to provide a medical diagnostic support device, a medical diagnostic support method, and a program that can present candidates for diseases suffered by a subject patient in advance and calculate findings, thereby making it possible to grasp the patient's condition before the examination.

That is, the above-mentioned object of the present invention is achieved by the following configuration.
(1) an acquisition unit that acquires a plurality of images of a subject, the images including at least a moving image;
An extraction unit that extracts information about mutually different characteristics from each of the plurality of images acquired by the acquisition unit , and further extracts the information about the mutually different characteristics based on department information of the subject;
an estimation unit that estimates a disease candidate of the subject and a certainty or severity of the disease from information on the different features extracted by the extraction unit;
A control unit that outputs to an output unit the candidate disease of the subject estimated by the estimation unit and the certainty or severity of the disease,
applying a trained classifier to at least one of the extraction unit and the estimation unit;
Medical diagnostic support equipment.

(2) each of the plurality of images is taken from a different viewpoint of the subject;
A medical diagnosis support device as described in (1).

(3) the estimation unit estimates one or more disease candidates;
A medical diagnosis support device according to (1) or (2).

(4) The estimation unit
Predicting a candidate disease of the subject based on the information on the different features extracted by the extraction unit and department information of the subject.
A medical diagnosis support device according to any one of (1) to (3).

(5) The extraction unit
extracting information relating to a plurality of mutually distinct features from each of the plurality of images;
A medical diagnosis support device according to any one of (1) to (4).

(6) The trained classifier is
At least one of the following will be established for each region, hospital, and medical department.
A medical diagnosis support device according to any one of (1) to (5).

(7) Each of the plurality of images includes a moving image or/and a still image.
A medical diagnosis support device according to any one of (1) to (6).

(8) A storage unit that stores the identification information, the video, and the still image of the subject in association with each other,
The medical diagnosis support device further comprises any one of (1) to (7).

(9) The still image is
images of the same subject at different times and/or different locations;
A medical diagnosis support device according to (7) or (8).

(10) The output unit is configured by a terminal installed in the examination room.
A medical diagnosis support device according to any one of (1) to (9).

( 11 ) A step in which an acquisition unit acquires a plurality of images of a subject, the images including at least a moving image;
An extraction unit extracts information regarding different features from each of the acquired images;
The extraction unit further extracts information about the different characteristics based on department information of the subject;
A step of estimating a candidate disease of the subject and a certainty or severity of the disease from information on the extracted different features by an estimation unit;
The control unit outputs the estimated candidate disease of the subject and the certainty or severity of the disease to the output unit,
In at least one of the extraction unit and the estimation unit, inference is performed using a trained classifier.
A medical diagnosis support method executed by a medical diagnosis support device.

( 12 ) A computer
acquiring a plurality of images of a subject, the images including at least a motion picture;
Extracting information relating to distinct features from each of the captured images;
Further extracting information regarding the different characteristics based on department information of the subject;
A step of estimating a disease candidate of the subject and a certainty or severity of the disease from information on the extracted different features;
A step of outputting the estimated candidate disease of the subject and the certainty or severity of the disease to an output unit;
performing inference using a trained classifier in at least one of the extraction and the estimation;
A program to make it happen.

According to the present invention, by presenting possible diseases of the subject patient in advance and calculating the findings, it is possible to understand the patient's condition before the examination.

This will help narrow down the tests that need to be done after the medical interview and identify the disease, and also allow appropriate tests to be carried out early, enabling early detection and treatment of diseases.

1 is a diagram illustrating an example of the configuration of a medical diagnosis support device according to a first embodiment. 2 is a block diagram showing functions of a CPU of the medical diagnosis support device according to the first embodiment. FIG. 4 is a flowchart showing a candidate disease estimation process executed by the medical diagnosis support apparatus according to the first embodiment. 13 is a flowchart showing a feature information extraction process. An explanatory diagram showing the concept of a CPU extracting information regarding mutually different features from each of a plurality of images using a trained model in an extraction unit. This is an explanatory diagram showing the concept of a CPU estimating a subject's disease candidates and the certainty or severity of the disease using a learned model from information regarding different features extracted by the extraction unit in the estimation unit. 13 is an example of a display screen in which the CPU displays, on the display screen of the display unit, possible diseases of the subject and estimated results of the certainty and severity of the diseases. 11 is a flowchart showing the processing of a training method for a trained model used in an extraction unit of the medical diagnosis support device according to the first embodiment. FIG. 2 is an explanatory diagram showing the concept of a learning method for a trained model used in the extraction unit of the medical diagnosis support device according to the first embodiment. 11 is a flowchart showing the processing of a training method for a trained model used in an estimation unit of the medical diagnosis support device according to the first embodiment. FIG. 2 is an explanatory diagram showing the concept of a learning method for a trained model used in an estimation unit of the medical diagnosis support device according to the first embodiment. FIG. 11 is a block diagram showing functions of a CPU of an image processing apparatus according to a second embodiment. 13 is a flowchart showing a candidate disease estimation process of the medical diagnosis support device according to the second embodiment. 1 is a flowchart showing a process for estimating and identifying a subject's disease candidacy, probability, and severity. An explanatory diagram showing the concept of a CPU estimating the candidate disease, probability, and severity of a subject from each of multiple images using a trained model in an estimation unit. An explanatory diagram showing the concept of a CPU identifying specific information (subject's disease candidate, certainty, severity) from estimated information (subject's disease candidate, certainty, severity) in the identification unit. 13 is a flowchart showing the processing of a training method for a trained model used in an estimation unit of a medical diagnosis support device according to a second embodiment. FIG. 11 is an explanatory diagram showing the concept of a learning method for a trained model used in an estimation unit of a medical diagnosis support device according to a second embodiment.

The following describes in detail the form for carrying out the present invention. Note that the embodiment described below is one example for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiment. In addition, a configuration may be made by appropriately combining parts of each embodiment described below. Note that the same reference numerals are used for the same members, and the description will be omitted as appropriate.

First Embodiment
[Overall configuration of the image processing device]
1A is a diagram for explaining an example of the configuration of a medical diagnosis support device 100 according to the first embodiment. The medical diagnosis support device 100 according to the first embodiment is configured to include a CPU 101, an image acquisition unit 102, a trained model 103, an output unit 104, a display unit 105, a ROM (Read Only Memory) 106, a RAM (Random Access Memory) 107, an external storage device 108, and an information acquisition unit 110. The external storage device 108 is configured to include a disease identification program 109 (109A, 109B).

As shown in FIG. 1A, the CPU 101 is an arithmetic processing device that controls the entire medical diagnosis support device 100. The CPU 101 performs various controls by executing programs stored in the ROM 106 and the external storage device 108. For example, the CPU 101 executes a disease identification program 109A stored in the external storage device 108 to realize an acquisition unit 111, an extraction unit 112, an estimation unit 113, and a control unit 114 shown in FIG. 1B.

Figure 1B is a block diagram showing the functions of the CPU 101 of the medical diagnosis support device 100 according to the first embodiment.

The acquisition unit 111 has a function of acquiring a plurality of images of the subject, including at least moving images. The acquisition unit 111 acquires a plurality of images of the subject, including at least moving images, from the image acquisition unit 102, which will be described later. The plurality of images includes moving images and/or still images. The subject refers to a patient or a test subject.

The extraction unit 112 has a function of extracting information relating to mutually different characteristics from each of the multiple images acquired by the acquisition unit 111. For example, the extraction unit 112 extracts information relating to mutually different characteristics from each of the multiple images.

The estimation unit 113 has a function of estimating the subject's disease candidates and the certainty or severity of the disease from information on the different characteristics extracted by the extraction unit 112. The estimation unit 113 estimates one or more disease candidates. Note that the certainty includes the degree of certainty. Furthermore, the severity includes the degree of severity. Furthermore, the estimation unit 113 can estimate the subject's disease candidates based on the subject's department information. Here, the department information refers to information on the department in the hospital that the subject visits, and the department information includes, for example, examination history, treatment and surgery history, etc.

The control unit 114 has a function of outputting to the output unit 104 the candidate diseases of the subject estimated by the estimation unit 113 and the certainty or severity of the diseases.

In the first embodiment, the CPU 101 controls the entire medical diagnosis support device 100. However, instead of controlling the entire medical diagnosis support device 100, for example, multiple hardware devices may share the processing to control the entire medical diagnosis support device 100.

The image acquisition unit 102 acquires a plurality of images of the subject, including at least moving images. The image acquisition unit 102 acquires, for example, images (still images) taken by a camera installed in the waiting room or reception area, images (moving images) taken by a camera of the movement from the waiting room to the examination room, or images (moving images) of the situation in the waiting room. Note that the plurality of images includes moving images and/or still images. The image acquisition unit 102 sends the acquired plurality of images to the acquisition unit 111 of the CPU 101.

The trained model 103 includes a pre-trained classifier, and stores and saves various parameters. The trained model 103 is applied to at least one of the extraction unit 112 and the estimation unit 113 of the CPU 101. Any training model can be applied to the trained model 103, and for example, a different training model is provided for at least one of the region, hospital, and medical department. In addition, for example, when inference is made using a method other than the trained model 103, inference may be made using image processing or a subjective evaluation method by a human. Details of the trained model 103 will be described later.

The output unit 104 outputs candidate diseases of the subject and the certainty or severity of the disease by the control unit 114 of the CPU 101. The output unit 104 is configured, for example, by a terminal installed in an examination room and used by a doctor, nurse, etc.

The display unit 105 displays the disease candidates and the certainty or severity of the disease via the output unit 104. The display unit 105 is composed of a display device such as a liquid crystal display, an organic electroluminescence display, or a printer, or software for display (a viewer).

The ROM 106 stores, for example, a control program for controlling the medical diagnosis support apparatus 100 .
The RAM 107 is configured, for example, by a dynamic random access memory (DRAM), and functions as a working memory for temporarily storing data and image data required for the CPU 101 to execute the control program and the disease identification program 109 (109A, 109B).

The external storage device 108 is, for example, a hard disk drive (HDD) and stores the subject's identification information, videos, and still images in association with each other.

The information acquisition unit 110 acquires department information of the subject. The information acquisition unit 110, for example, receives input of patient information and patient biometric information of the subject, and acquires department information. The information acquisition unit 110 is composed of operation members such as a keyboard, a mouse, various switches, buttons, and a touch panel that accept various operations from the user.

[Processing of medical diagnosis support device]
Next, the candidate disease estimation process executed by the medical diagnosis support device 100 configured as described above will be described using a flowchart with reference to FIGS. 1A and 1B. FIG.

Figure 2 is a flowchart showing the disease candidate estimation process executed by the medical diagnosis support device 100 according to the first embodiment.

First, the CPU 101 of the medical diagnosis support device 100 acquires, in the acquisition unit 111, a plurality of images of the subject, including at least a moving image (step S001). In this case, the CPU 101 acquires a plurality of images from the image acquisition unit 102.

Next, the CPU 101 of the medical diagnosis support device 100 extracts information about different characteristics from each of the multiple images acquired by the acquisition unit 111 in the extraction unit 112 (step S003).

In this case, the CPU 101 executes a feature information extraction process (step S003) to extract information about the different features. Here, the feature information extraction process executed by the CPU 101 of the medical diagnosis support device 100 will be described with reference to the flowchart shown in FIG. 3.

FIG. 3 is a flowchart showing the feature information extraction process.
The CPU 101 starts this flowchart when it detects a subject from an image taken by a camera installed in the waiting room, reception area, etc., or from an image taken by a camera installed on the route from the waiting room to the examination room (e.g., a hallway).

From each of the multiple images, the CPU 101 determines the location where the image was captured and whether it is a video or a still image (step S101). For example, if the captured image is a still image of the subject captured by a camera installed in a waiting room, reception area, or the like, the CPU 101 extracts a facial image from the still image captured in the waiting room (step S103).

Here, the CPU 101 determines where the image of the subject was taken and whether it is a video or a still image. For example, the CPU 101 determines whether it is a still image in the waiting room, a video of the movement from the waiting room to the examination room, or a video in the waiting room.

In addition, if the captured image is a still image, images (face images) of the same subject at different times and/or different locations are obtained.

Then, the CPU 101 calculates the facial color of the subject from the acquired facial image (step S105). In this case, the CPU 101 calculates, for example, the blackness, blueness, etc., of the facial image of the subject patient. In other words, the CPU 101 extracts the color characteristics of the facial image of the subject at rest from a still image captured by a camera installed in the waiting room.

On the other hand, if the acquired image is a video of the subject moving from the waiting room to the examination room (step S101), the CPU 101 extracts a person from the video (step S107). The CPU 101 then calculates the amount of movement (walking distance) by tracking (step S109), and calculates the subject's walking time from the time it left the waiting room and the time it stood in front of the examination room (step S111). The CPU 101 then calculates the subject's walking speed (movement speed) from the walking distance and walking time (step S113). In other words, the CPU 101 extracts the characteristics of the subject's walking speed from the video captured by a camera installed on the movement route from the waiting room to the examination room.

On the other hand, if the acquired image is a video taken in a waiting room (step S101), the CPU 101 extracts the person's shoulders from the video (step S115). The CPU 101 then extracts the timing (breathing timing) of when the subject starts to inhale and stops to exhale (step S117). The CPU 101 then calculates the subject's breathing rate (step S119). In this case, the CPU 101 calculates how many times the breathing timing is repeated per minute as the subject's breathing rate at rest. In other words, the CPU 101 extracts the characteristics of the subject's breathing rate at rest from the video taken by a camera installed in the waiting room.

In this way, the CPU 101 extracts information about different characteristics from each of a plurality of images taken of the subject from different viewpoints in the extraction unit 112. Note that the CPU 101 may also extract information about different characteristics from each of a plurality of images taken of the subject from different viewpoints in the extraction unit 112.

Figure 4 is an explanatory diagram showing the concept of the CPU 101 extracting (calculating) information about different features from each of multiple images using the trained models 103 (trained models 103a to 103c) in the extraction unit 112.

As shown in FIG. 4, the CPU 101 extracts complexion information EI1 by applying the learned model 103a to the still image SI and information relating to the medical department (medical department information) in the extraction unit 112. Similarly, the CPU 101 extracts moving speed EI2 by applying the learned model 103b to the video MI1 and medical department information in the extraction unit 112. Similarly, the CPU 101 extracts breathing rate EI3 by applying the learned model 103c to the video MI2 and medical department information in the extraction unit 112.

The extraction unit 112 can narrow down (extract) the features (e.g., unique features) required for each medical department by applying the medical department information. For this reason, the CPU 101 can, for example, switch the trained model 103 for each medical department, or can apply a unique trained model 103 for each region or hospital.

Returning to FIG. 3, once the CPU 101 has calculated (extracted) the facial color information EI1, the movement speed EI2, and the breathing rate EI3, it ends the feature information extraction process and returns to the flowchart of FIG. 2 to continue processing.

The CPU 101 stores the calculated (extracted) facial color information EI1, movement speed EI2, and breathing rate EI3 (information relating to different characteristics) as extraction results in the external storage device 108 (step S005).

Next, the CPU 101 estimates multiple disease candidates for the subject from information on the mutually different characteristics extracted by the extraction unit 112 (step S007). In this case, the CPU 101 estimates one or more disease candidates in the estimation unit 113. The CPU 101 can estimate the disease candidates for the subject based on department information for the subject. Note that the department information is an optional component, and when estimating disease candidates in the estimation unit 113, the CPU 101 may make the estimation based on information other than the department information. For example, for epidemic diseases, disease candidates for the subject may be estimated based on information on the area in which the disease is prevalent.

The CPU 101 also estimates the certainty or severity of the disease from the information regarding the different characteristics extracted by the extraction unit 112 (step S009).

Figure 5 is an explanatory diagram showing the concept of the CPU 101 estimating, in the estimation unit 113, candidate diseases of a subject and the certainty or severity of the disease from information related to the different features extracted by the extraction unit 112, using the trained model 103d.

As shown in FIG. 5, the CPU 101, in the estimation unit 113, applies the learned model 103d from the complexion information EI1, the movement speed EI2, and the breathing rate EI3 (information on mutually different characteristics) to estimate the subject's disease candidates and the probability or severity of the disease. Specifically, from the complexion information EI1, the movement speed EI2, and the breathing rate EI3, the CPU 101 estimates: 1. The probability of COPD (chronic obstructive pulmonary disease) is 90% and the severity is 5; 2. The probability of pulmonary fibrosis is 85% and the severity is 4; and 3. The probability of asthma is 50% and the severity is 4.

Returning to the flowchart of FIG. 2, the CPU 101 causes the display unit 105 to display the candidate diseases of the subject and the estimated results of the disease certainty or severity (step S011), and ends the candidate disease estimation process.

Figure 6 shows an example of a display screen in which the CPU 101 displays, on the display screen DS of the display unit 105, possible diseases of the subject and estimated results of the certainty and severity of the diseases.

As shown in FIG. 6, the display screen DS of the display unit 105 displays: 1. The probability of COPD (chronic obstructive pulmonary disease) is 90% and the severity is 5; 2. The probability of pulmonary fibrosis is 95% and the severity is 4; and 3. The probability of asthma is 50% and the severity is 4.

The disease candidates displayed on the display screen DS may be all of the estimated disease candidates, or, for example, disease candidates with a disease certainty of 50% or more may be displayed. In the present embodiment, as an example, disease candidates with a disease certainty of 50% or more are displayed.

In this way, the CPU 101 of the medical diagnosis support device 100 presents possible diseases of the subject patient (subject) in advance and calculates the findings, allowing medical professionals such as doctors and nurses to understand the patient's condition before the examination.

Next, we will explain the trained model 103 (trained models 103a to 103c) used in the extraction unit 112.

[Generating a trained model]
The trained model 103 (trained models 103a to 103c) used in the extraction unit 112 uses previously prepared image information as input, and uses a large number of (e.g., several thousand to tens of thousands of) data sets as training sample data, with the output being an extraction result according to each purpose (e.g., a face region image in the case of calculating a facial color), and performs machine learning using the data. As the learning device (not shown), a standalone high-performance computer using a CPU and a GPU (Graphics Processing Unit) processor, or a cloud computer can be used.

The learning method using a neural network formed by combining perceptrons as a learning device (not shown) is described using a flowchart, but the present embodiment is not limited to this, and various methods can be used as long as they are supervised learning. For example, random forests, support vector machines (SVMs), boosting, Bayesian, network linear discrimination methods, nonlinear discrimination methods, etc. can be applied to the learning device (not shown).

[How to train a trained model]
FIG. 7 is a flowchart showing the processing of a learning method for the trained model 103 (trained models 103a to 103c) used in the extraction unit 112 of the medical diagnosis support device 100 according to the first embodiment.

The learning device (not shown) reads the learning sample data, which is the teacher data. For example, if it is the first time, the first set of learning sample data (image information, disease candidate, accuracy, severity) is read, and if it is the i-th time, the i-th set of learning sample data is read (step S201).

The learning device (not shown) inputs the input data (image information) from the loaded learning sample data into the neural network (step S203).

The learning device (not shown) compares the neural network's estimation results, i.e., the results according to the estimated purpose (e.g., a face area image in the case of calculating facial color), with the training data (step S205).

The learning device (not shown) adjusts the parameters based on the comparison results (step S207). For example, the learning device (not shown) adjusts and updates the parameters by performing a process called back-propagation (backpropagation method) so as to reduce the error in the comparison results.

When the learning device (not shown) has completed all processing of all data from the 1st to ith sets (YES in step S209), it proceeds to step S211. On the other hand, when the learning device (not shown) has not completed all processing of all data (NO in step S209), it returns to step 201, reads the next learning sample data, and repeats the processing from step S201 onwards.

The learning device (not shown) stores the trained model constructed in the processing up to step S207 (step S211), and ends the processing. In this way, trained model 103 (trained models 103a to 103c) performs machine learning using the training sample data.

As a result, the CPU 101 of the medical diagnosis support device 100 according to the first embodiment can apply the trained model 103 (trained models 103a to 103c) to the extraction unit 112.

Figure 8 is an explanatory diagram showing the concept of the learning method of the trained model 103 (trained models 103a to 103c) used in the extraction unit 112 of the medical diagnosis support device 100 according to the first embodiment. In Figure 8, one or more features are extracted (calculated) from one type of image (video). Furthermore, each feature is different from the features obtained from other types of images.

As shown in FIG. 8, the trained model 103a extracts and evaluates, for example, redness, blueness, yellowness, whiteness, blackness, etc., as facial color information of the subject from the still image SI. Specifically, the trained model 103a extracts from the still image SI the following values: redness is "0.20", blueness is "0.25", yellowness is "0", whiteness is "0.10", and blackness is "0.75". Each evaluation value is normalized between "0" and "1".

As a result, the trained model 103a can evaluate that the facial color of the subject in the still image SI is, for example, dark.

The trained model 103b also extracts the walking time of the subject from the video MI1 based on the time the subject left the waiting room and the time the subject stood in front of the examination room. The trained model 103b then extracts the walking speed (movement speed) of the subject from the walking distance and walking time. Specifically, the trained model 103b extracts that the walking speed of the subject from the waiting room to the examination room is 0.3 [m/s] and the walking distance of the subject is 4.0 [m].

As a result, the trained model 103b can evaluate that the walking speed of the subject in video MI1 is slow, for example, because the walking speed is below 0.8 m/s.

The trained model 103c also extracts the subject's degree of shortness of breath (respiration rate in the waiting room) from the video MI2. The respiration rate is the respiration rate, which indicates how many times a person breathes in one minute. Specifically, the trained model 103c extracts the subject's degree of shortness of breath to be 40 times/min.

As a result, the trained model 103c can evaluate, for example, that the subject in video MI2 has a large or fast breathing rate. Here, the trained model 103c evaluates, for example, a breathing rate of 12 to 20 breaths per minute as normal, and a breathing rate of 25 or more breaths per minute as tachypnea. In addition, a breathing rate of 40 or more breaths per minute may be evaluated as COPD.

Note that the expression of shortness of breath is not particularly limited, and can be evaluated using an appropriate expression depending on the degree of shortness of breath, such as not being able to breathe in properly, not being able to breathe out properly, or suffocating.

Next, we will explain the trained model 103 (trained model 103d) used by the estimation unit 113.

[Generating a trained model]
The trained model 103 (trained model 103d) used in the estimation unit 113 uses a large number of (e.g., several thousand to tens of thousands of) data sets as training sample data, in which numerical data for each feature prepared in advance is input and disease (disease candidate, probability, severity) is output, and performs machine learning using the data. As a learning device (not shown), a standalone high-performance computer using a CPU and GPU processor, or a cloud computer can be used.

The learning method using a neural network formed by combining perceptrons as a learning device (not shown) is described using a flowchart, but the present embodiment is not limited to this, and various methods can be used as long as they are supervised learning. For example, random forests, support vector machines (SVMs), boosting, Bayesian, network linear discrimination methods, nonlinear discrimination methods, etc. can be applied to the learning device (not shown).

[How to train a trained model]
9 is a flowchart showing the process of the learning method of the trained model 103 (trained model 103d) used in the estimation unit 113 of the medical diagnosis support device 100 according to the first embodiment. In the flowchart shown in Fig. 9, the same processes as those in the flowchart shown in Fig. 7 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

Note that the process steps that differ between the flowchart shown in FIG. 9 and the flowchart shown in FIG. 7 are steps S301 to S305, so only the process steps that differ from the flowchart shown in FIG. 7 will be described.

The learning device (not shown) reads the learning sample data, which is the teacher data. For example, if it is the first time, the first set of learning sample data (numerical data for each feature (feature data), disease candidates, accuracy, and severity) is read, and if it is the i-th time, the i-th set of learning sample data is read (step S301).

The learning device (not shown) inputs the input data (numerical data for each feature (feature data)) from the loaded learning sample data into the neural network (step S303).

The learning device (not shown) compares the neural network's estimation results, i.e., the estimated disease candidates, accuracy, and severity, with the training data (step S305).

The subsequent processing is the same as that in the flowchart of FIG. 7, and the learning device (not shown) stores the trained model constructed in the processing up to step S207 (step S211), and ends the processing. In this way, trained model 103 (trained model 103d) performs machine learning using the training sample data.

As a result, the CPU 101 of the medical diagnosis support device 100 according to the first embodiment can apply the trained model 103 (trained model 103d) to the estimation unit 113.

Figure 10 is an explanatory diagram showing the concept of the learning method of the trained model 103 (trained model 103d) used in the estimation unit 113 of the medical diagnosis support device 100 according to the first embodiment. In Figure 10, the correspondence between disease and severity is learned from each obtained feature, and the certainty and severity of the disease are estimated.

As shown in FIG. 10, the trained model 103d learns the correspondence between disease and severity from each of the obtained features using a pre-trained classifier. The trained model 103d estimates the likelihood and severity of COPD, the likelihood and severity of pulmonary fibrosis, the likelihood and severity of asthma, etc., from, for example, "redness of face," "blueness of face," "yellowness of face," "whiteness of face," "darker face," "walking speed [m/s]," "walking distance [m]," and "respiratory rate [breaths/min]," using a pre-trained classifier.

Specifically, the classifier has been trained to recognize, in advance, the following correct answer data for given input data: accuracy of COPD is "1" and severity of "5", accuracy of pulmonary fibrosis is "1" and severity of "4", accuracy of asthma is "0.4" and severity of "3", etc. Note that accuracy is a normalized value between 0 and 1, and severity is a value evaluated on a 5-point scale.

The trained model 103d acquires, as feature data, the redness of the face, "0.20", the blueness of the face, "0.25", the yellowness of the face, "0", the lightness of the face, "0.1", the darkness of the face, "0.75", the walking speed [m/s], "0.3", the walking distance [m], "4.0", and the respiration rate [breaths/min], "40", etc.

As a result, the trained model 103d can estimate, for example, that the accuracy of COPD is "0.9" and the severity is "5", that the accuracy of pulmonary fibrosis is "0.85" and the severity is "4", and that the accuracy of asthma is "0.5" and the severity is "4".

As described above, the CPU 101 of the medical diagnosis support device 100 in the first embodiment is configured to include an acquisition unit 111, an extraction unit 112, an estimation unit 113, and a control unit 114. The CPU 101 of the medical diagnosis support device 100 extracts, in the extraction unit 112, information relating to mutually different features from each of the multiple images acquired by the acquisition unit 111, and, in the estimation unit 113, estimates disease candidates of the subject and the certainty or severity of the disease from the extracted information relating to mutually different features.

As a result, the medical diagnosis support device 100 in the first embodiment can grasp the patient's condition before the examination by presenting in advance the candidate disease, certainty, or severity of the patient who is the subject.

In addition, the medical diagnosis support device 100 in the first embodiment can help narrow down the tests required after the interview and identify diseases, and can also perform appropriate tests early, thereby enabling early detection and early treatment of diseases.

Next, we will explain the medical diagnosis support device 100 according to the second embodiment.

Second Embodiment
[Overall configuration of the image processing device]
1A, but includes a CPU 101a instead of the CPU 101. The CPU 101a is an arithmetic processing device that controls the entire medical diagnosis support device 100, similar to the CPU 101 of the first embodiment.

The CPU 101a performs various controls by executing programs stored in the ROM 106 and the external storage device 108. For example, the CPU 101a executes a disease identification program 109B stored in the external storage device 108 to realize the acquisition unit 111, estimation unit 113a, identification unit 115, and control unit 114 shown in FIG. 11A.

Figure 11A is a block diagram showing the functions of the CPU 101a of the medical diagnosis support device 100 according to the second embodiment. Note that the same components as those of the CPU 101 of the medical diagnosis support device 100 according to the first embodiment shown in Figure 1B are given the same reference numerals, and descriptions thereof will be omitted as appropriate.

The acquisition unit 111 has a function of acquiring a plurality of images of a subject, including at least a moving image. The plurality of images includes a moving image and/or a still image.

The estimation unit 113a has a function of estimating a candidate disease of the subject and the certainty or severity of the disease from each of the multiple images acquired by the acquisition unit 111 using the trained model 103 (trained classifier). The estimation unit 113a estimates one or more candidate diseases. Furthermore, the estimation unit 113a can estimate a candidate disease of the subject based on the department information of the subject in addition to each of the multiple images acquired by the acquisition unit 111.

The identification unit 115 has a function of identifying the subject's disease candidates, the certainty or severity of the disease from the subject's disease candidates, the certainty or severity of the disease estimated by the estimation unit 113a.

The control unit 114 has a function of causing the output unit 104 to output the disease candidates, the certainty or severity of the disease identified by the identification unit 115.

The trained model 103 (trained models 103e to 103g) is configured to include a pre-trained classifier, and stores and saves various parameters. Any training model can be applied to the trained model 103 (trained models 103e to 103g), and for example, a different training model is provided for at least one of the following: region, hospital, and medical department. Details of the trained model 103 (trained models 103e to 103g) will be described later.

In the second embodiment, the CPU 101a controls the entire medical diagnosis support device 100. However, instead of controlling the entire medical diagnosis support device 100, for example, multiple hardware devices may share the processing to control the entire medical diagnosis support device 100.

[Processing of medical diagnosis support device]
Next, the candidate disease estimation process executed by the medical diagnosis support device 100 configured as described above will be described using a flowchart with reference to FIG. 11A.

Figure 11B is a flowchart showing a disease candidate estimation process executed by the medical diagnosis support device 100 according to the second embodiment. Note that the same processes as those in the flowchart of Figure 2 are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

First, the CPU 101a of the medical diagnosis support device 100 acquires, in the acquisition unit 111, a plurality of images of the subject, including at least moving images (step S001). In this case, the CPU 101 acquires a plurality of images from the image acquisition unit 102. Note that the plurality of images includes moving images and/or still images.

The CPU 101a of the medical diagnosis support device 100 executes "estimation and identification of disease candidates, accuracy, and severity" (step S401) in the estimation unit 113a. Here, the process of "estimation and identification of disease candidates, accuracy, and severity" executed by the CPU 101a of the medical diagnosis support device 100 will be described with reference to the flowchart shown in FIG. 12.

Figure 12 is a flowchart showing the "Process of estimating and identifying disease candidates, certainty, and severity." The CPU 101a, in the estimation unit 113a, estimates disease candidates, certainty, and severity from still images of the subject among multiple images (step S501). The estimation unit 113a estimates one or more disease candidates. The estimation unit 113a can estimate disease candidates of the subject based on department information of the subject. Note that department information is an optional component. The still images include images of the same subject at different times and/or different locations.

The CPU 101a also causes the estimation unit 113a to estimate disease candidates, accuracy, and severity from a video of the subject (e.g., video 1) among the multiple images (step S503). In this case, the estimation unit 113a similarly estimates one or more disease candidates. The estimation unit 113a can estimate disease candidates of the subject based on department information of the subject. Note that department information is an optional component.

The CPU 101a also causes the estimation unit 113a to estimate disease candidates, accuracy, and severity from a video of the subject (e.g., video 2) among the multiple images (step S505). In this case, the estimation unit 113a similarly estimates one or more disease candidates. The estimation unit 113a can estimate disease candidates of the subject based on department information of the subject. Note that department information is an optional component.

Figure 13 is an explanatory diagram showing the concept of the CPU 101a estimating disease candidates, probability, and severity from each of multiple images using the trained model 103 (trained models 103e to 103g) in the estimation unit 113a.

As shown in FIG. 13, the CPU 101a applies the learned model 103e to the still image SI of the subject and information about the medical department (medical department information) in the estimation unit 113a to estimate the candidate diseases as "COPD", "pulmonary fibrosis", and "cold" (estimated information IF1). In this case, the estimation unit 113a estimates, as estimated information IF1, the accuracy of COPD to be "90%" and the severity to be "5", the accuracy of pulmonary fibrosis to be "85%" and the severity to be "5", and the accuracy of cold to be "10%" and the severity to be "1".

The CPU 101a also applies the learned model 103f to the subject's video (video 1) MI1 and medical department information in the estimation unit 113a, and estimates the candidate diseases as "COPD", "pulmonary fibrosis", and "cold" (estimated information IF2). In this case, the estimation unit 113a estimates, as estimated information IF2, that the accuracy of COPD is "90%" and the severity is "5", that the accuracy of pulmonary fibrosis is "85%" and the severity is "4", and that the accuracy of cold is "10%" and the severity is "1".

The CPU 101a also applies the learned model 103g to the subject's video (video 2) MI2 and medical department information in the estimation unit 113a, and estimates the candidate diseases as "COPD", "pulmonary fibrosis", and "asthma" (estimated information IF3). In this case, the estimation unit 113a estimates, as estimated information IF3, the accuracy of COPD to be "90%" and the severity to be "5", the accuracy of pulmonary fibrosis to be "85%" and the severity to be "4", and the accuracy of asthma to be "80%" and the severity to be "4".

Returning to the flowchart of FIG. 12, the CPU 101a, in the identification unit 115, identifies the candidate disease, the certainty, or the severity of the subject's disease from the candidate disease, the certainty, or the severity of the subject's disease estimated by the estimation unit 113a (step S507). In this case, the identification unit 115 normalizes the candidate disease, the certainty, or the severity of the subject's disease.

Figure 14 is an explanatory diagram showing the concept of the CPU 101a identifying the subject's specific information (disease candidate, certainty, and severity) IF4 from the subject's estimated information (disease candidate, certainty, and severity) IF1 to IF3 in the identification unit 115.

As shown in FIG. 14, in the subject's estimated information IF1, the probability of COPD is estimated to be "90%" and the severity is "5", the probability of pulmonary fibrosis is estimated to be "85%" and the severity is "5", the probability of cold is estimated to be "10%" and the severity is "1". In the subject's estimated information IF2, the probability of COPD is estimated to be "90%" and the severity is "5", the probability of pulmonary fibrosis is estimated to be "85%" and the severity is "4", the probability of cold is estimated to be "10%" and the severity is "1". In the subject's estimated information IF3, the probability of COPD is estimated to be "90%" and the severity is "5", the probability of pulmonary fibrosis is estimated to be "85%" and the severity is "4", the probability of asthma is estimated to be "80%" and the severity is "4".

The CPU 101a, in the identification unit 115, identifies the candidate diseases of the subject from the estimated information IF1 to IF3 of the subject as "COPD", "pulmonary fibrosis", "asthma", and "cold", and identifies the probability of COPD as "90%" and the severity as "5", the probability of pulmonary fibrosis as "85%" and the severity as "4", the probability of asthma as "27%" and the severity as "4", and the probability of cold as "6%" and the severity as "1".

When calculating the accuracy of each disease candidate shown in FIG. 14, the CPU 101a calculates, for example, the average value in the estimated information IF1 to IF3 of the subject. Specifically, the average value is calculated by dividing the estimated accuracy by the number of occurrences in the estimated information IF1 to IF3 of the subject. Note that if the accuracy is not estimated in the estimated information IF1 to IF3, it is calculated as "0".

For example, the probability of the subject having COPD is calculated as 0.9 (90%) by (0.9 + 0.9 + 0.9) / 3. The probability of the subject having pulmonary fibrosis is calculated as 0.85 (85%) by (0.85 + 0.85 + 0.85) / 3. The probability of the subject having asthma is calculated as 0.27 (27%) by (0.8 / 3). The probability of the subject having a cold is calculated as 0.06 (6%) by (0.1 + 0.1) / 3. These disease candidates, probabilities, and severity are the subject's specific information IF4.

Returning to the flowchart of FIG. 12, once the CPU 101a has identified the disease candidates, their certainty, and their severity (step S507), it ends the process of "estimating and identifying disease candidates, their certainty, and their severity" and returns to the flowchart of FIG. 11B.

The CPU 101a displays the contents of the identification information IF4 of the identified subject on the display unit 105 as the identification result (step S009), and ends the disease candidate estimation process.

In this way, the CPU 101a of the medical diagnosis support device 100 can present the candidate disease, certainty, or severity of the patient subject in advance, allowing medical professionals such as doctors and nurses to understand the patient's condition before the examination.

Next, we will explain the trained model 103 (trained models 103e to 103g) used in the extraction unit 112.

[Generating a trained model]
The trained model 103 (trained models 103e to 103g) used in the estimation unit 113a uses prepared image information as input, and uses a large number of (for example, several thousand to tens of thousands of) data sets as training sample data, with the output being extraction results (disease candidates, accuracy, and severity) according to each purpose, and performs machine learning using the data. As a learning device (not shown), a standalone high-performance computer using a CPU and GPU processor, or a cloud computer can be used.

The learning method using a neural network formed by combining perceptrons as a learning device (not shown) is described using a flowchart, but the present embodiment is not limited to this, and various methods can be used as long as they are supervised learning. For example, random forests, support vector machines (SVMs), boosting, Bayesian, network linear discrimination methods, nonlinear discrimination methods, etc. can be applied to the learning device (not shown).

[How to train a trained model]
15 is a flowchart showing the process of the learning method of the trained model 103 (trained models 103e to 103g) used in the estimation unit 113a of the medical diagnosis support device 100 according to the second embodiment. Note that the same processes as those in the flowcharts of FIGS. 7 and 9 are denoted by the same reference numerals, and the description thereof will be omitted as appropriate.

The learning device (not shown) reads the learning sample data, which is the teacher data. For example, if it is the first time, the first set of learning sample data (image information, disease candidate, accuracy, severity) is read, and if it is the i-th time, the i-th set of learning sample data is read (step S601).

The learning device (not shown) inputs the input data (image information) from the loaded learning sample data into the neural network (step S603).

The learning device (not shown) compares the neural network's estimation results, i.e., the results according to the estimated objective (disease candidates, accuracy, and severity), with the training data (step S605).

The learning device (not shown) adjusts the parameters based on the comparison results (step S207). For example, the learning device (not shown) adjusts and updates the parameters by performing a process called back-propagation (backpropagation method) so as to reduce the error in the comparison results.

When the learning device (not shown) has completed all processing of all data from the 1st to ith sets (YES in step S209), it proceeds to step S211. On the other hand, when the learning device (not shown) has not completed all processing of all data (NO in step S209), it returns to step S601, reads the next learning sample data, and repeats the processing from step S601 onwards.

The learning device (not shown) stores the trained model constructed in the processing up to step S207 (step S211), and ends the processing. In this way, trained model 103 (trained models 103e to 103g) performs machine learning using the training sample data.

As a result, the CPU 101a of the medical diagnosis support device 100 according to the second embodiment can apply the trained model 103 (trained models 103e to 103g) to the estimation unit 113a.

Figure 16 is an explanatory diagram showing the concept of the learning method of the trained model 103 (trained models 103e to 103g) used in the estimation unit 113a of the medical diagnosis support device 100 according to the second embodiment. Figure 16 shows that one or more features are extracted (calculated) from each of multiple images to learn disease candidates, probability, and severity.

16, the trained model 103e estimates, for example, from a still image SI of the subject, the probability of COPD being "1", the severity being "5", the probability of pulmonary fibrosis being "1", the severity being "5", the cold syndrome being "0.1", the severity being "3", the probability of asthma being "0", the severity being "0", etc. Also, the trained model 103f estimates, for example, from a video (video 1) MI1 of the subject, the probability of COPD being "1", the severity being "5", the probability of pulmonary fibrosis being "1", the severity being "5", the cold syndrome being "0.1", the severity being "4", the probability of asthma being "0", the severity being "0", etc. Furthermore, the trained model 103g estimates, for example, from the subject's video (video 2) MI2, that the probability of COPD is "1" and the severity is "5", the probability of pulmonary fibrosis is "1" and the severity is "5", the cold syndrome is "0" and the severity is "0", and the probability of asthma is "0.8" and the severity is "3".

As described above, the CPU 101a of the medical diagnosis support device 100 in the second embodiment is configured to include an acquisition unit 111, an estimation unit 113a, an identification unit 115, and a control unit 114. The CPU 101a of the medical diagnosis support device 100 estimates the subject's disease candidates and the disease certainty or severity from each of the multiple images using the trained models 103 (trained models 103e to 103g) in the estimation unit 113a, and identifies the subject's disease candidates and the disease certainty or severity from the subject's disease candidates and disease certainty or severity estimated by the estimation unit 113a in the identification unit 115.

As a result, the medical diagnosis support device 100 in the second embodiment can grasp the patient's condition before the examination by presenting in advance the candidate disease, certainty, or severity of the patient who is the subject.

In addition, the medical diagnosis support device 100 in the second embodiment can help narrow down the tests required after the interview and identify diseases, and can also perform appropriate tests early, thereby enabling early detection and early treatment of diseases.

100 Medical diagnosis support device 101, 101a CPU
102 Image acquisition unit 103, 103a, 103b, 103c, 103d Trained model 103e, 103f, 103g Trained model 104 Output unit 105 Display unit 106 ROM
107 RAM
108 External storage device 109 Disease identification program 110 Information acquisition unit 111 Acquisition unit 112 Extraction unit 113, 113a Estimation unit 114 Control unit 115 Identification unit

Claims (12)

an acquisition unit that acquires a plurality of images of a subject, the images including at least a moving image;
An extraction unit that extracts information about mutually different characteristics from each of the plurality of images acquired by the acquisition unit, and further extracts the information about the mutually different characteristics based on department information of the subject;
an estimation unit that estimates a disease candidate of the subject and a certainty or severity of the disease from information on the different features extracted by the extraction unit;
A control unit that outputs to an output unit the candidate disease of the subject estimated by the estimation unit and the certainty or severity of the disease,
applying a trained classifier to at least one of the extraction unit and the estimation unit;
Medical diagnostic support equipment. Each of the plurality of images is taken from a different viewpoint of the subject.
The medical diagnosis support device according to claim 1 . The estimation unit estimates one or more disease candidates.
3. The medical diagnosis support device according to claim 1. The estimation unit is
Predicting a candidate disease of the subject based on the information on the different features extracted by the extraction unit and department information of the subject.
The medical diagnosis support device according to claim 1 . The extraction unit is
extracting information relating to a plurality of mutually distinct features from each of the plurality of images;
The medical diagnosis support device according to claim 1 . The trained classifier is
At least one of the following will be established for each region, hospital, and medical department.
The medical diagnosis support device according to claim 1 . Each of the plurality of images includes a video or/and a still image.
The medical diagnosis support device according to claim 1 . A storage unit that stores the identification information, the video, and the still image of the subject in association with each other,
The medical diagnosis support device according to claim 1 , further comprising: The still image is
images of the same subject at different times and/or different locations;
9. The medical diagnosis support device according to claim 7 or 8. The output unit is configured by a terminal installed in the examination room.
The medical diagnosis support device according to claim 1 . acquiring a plurality of images of a subject, the images including at least a moving image, by an acquisition unit;
An extraction unit extracts information regarding different features from each of the acquired images;
The extraction unit further extracts information about the different characteristics based on department information of the subject;
A step of estimating a candidate disease of the subject and a certainty or severity of the disease from information on the extracted different features by an estimation unit;
The control unit outputs the estimated candidate disease of the subject and the certainty or severity of the disease to the output unit,
In at least one of the extraction unit and the estimation unit, inference is performed using a trained classifier.
A medical diagnosis support method executed by a medical diagnosis support device. On the computer,
acquiring a plurality of images of a subject, the images including at least a motion picture;
Extracting information relating to distinct features from each of the captured images;
Further extracting information about the different characteristics based on department information of the subject;
A step of estimating a disease candidate of the subject and a certainty or severity of the disease from information on the extracted different features;
A step of outputting the estimated candidate disease of the subject and the certainty or severity of the disease to an output unit;
performing inference using a trained classifier in at least one of the extraction and the estimation;
A program for executing the above.

Images