JP7749536B2 - Information processing device, information processing method, and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

In the medical field, there is technology for search systems that search for patient cases. For example, Patent Document 1 discloses an image search device that inputs a patient's examination images into a neural network to classify them into multiple types of lesions, searches a case database for similar case images based on the classification results, and further accepts input specifying the position and name of key findings in the examination images, searching for similar case images based on the specified position and name of the key findings.

JP 2019-82881 A

However, the invention of Patent Document 1 requires the user to specify the location and name of the key finding that serves as the search key, which is not necessarily a convenient method for the user.

In one aspect, the objective is to provide an information processing device, etc. that can efficiently search for similar cases.

an information processing device according to one aspect of the present invention, the information processing device comprising: a storage unit that stores medical information of a first patient who has already undergone treatment, the medical information including progress information indicating whether or not complications have occurred after the treatment of the first patient, in association with features of the medical information; an acquisition unit that acquires medical information of a second patient who is the target of treatment; an extraction unit that extracts features of the medical information of the second patient; a calculation unit that calculates a similarity between the medical information of the first patient and the second patient based on the features of the medical information of the first patient and the features of the medical information of the second patient; a search unit that searches for multiple pieces of medical information of the first patient that are similar to the medical information of the second patient based on the calculated similarity; and an output unit that outputs the searched medical information of the multiple pieces of first patient in accordance with a search order determined according to the similarity, wherein the search unit lowers the search order of the medical information of the first patient if a complication has occurred in the searched first patient .

On the one hand, it allows for an efficient search for similar cases.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of a case search system. FIG. 2 is a block diagram illustrating an example of the configuration of a server. FIG. 2 is an explanatory diagram illustrating an example of a record layout of a case DB. FIG. 1 is an explanatory diagram showing an overview of a first embodiment. FIG. 10 is an explanatory diagram showing an example of a display screen of similar cases. 10 is a flowchart showing the steps of a learning model generation process. 10 is a flowchart showing a procedure for searching for similar cases. FIG. 10 is an explanatory diagram of a learning model according to the second embodiment. 10 is a flowchart showing the steps of a learning model generation process according to the second embodiment. 10 is a flowchart showing a procedure for searching for similar cases according to the second embodiment.

The present invention will be described in detail below with reference to the drawings showing embodiments thereof.
(Embodiment 1)
1 is an explanatory diagram showing an example of the configuration of a case retrieval system. In this embodiment, a case retrieval system is described that retrieves, from a database storing the medical information of a first patient who has already undergone vascular treatment, medical information of a first patient that is similar to the medical information of a second patient who will newly undergo vascular treatment. The case retrieval system includes an information processing device 1 and an imaging diagnostic device 2. The information processing device 1 and the imaging diagnostic device 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

In this embodiment, vascular treatment is described as an example, but the target hollow organs are not limited to blood vessels, and may be other hollow organs such as the bile duct, pancreatic duct, bronchi, and intestines.

The diagnostic imaging device 2 is a device unit that acquires medical images of a patient's tubular organs, such as an IVUS (Intravascular Ultrasound) device that performs ultrasound examinations using a catheter 21. The diagnostic imaging device 2 comprises the catheter 21, an image processing device 22, and a display device 23. The catheter 21 is a medical instrument inserted into the subject's blood vessels, and comprises a piezoelectric element that transmits ultrasound waves and receives reflected waves from within the blood vessels. The diagnostic imaging device 2 generates a tomographic image (medical image) of the blood vessels based on the reflected wave signal received by the catheter 21. The image processing device 22 is a processing device that processes the reflected wave data received by the catheter 21 to generate a tomographic image, displays the generated tomographic image on the display device 23, and also comprises an input interface for accepting input of various setting values used when conducting the examination.

In this embodiment, an IVUS device is used as an example of the diagnostic imaging device 2, and ultrasound tomographic images are generated as medical images. However, the diagnostic imaging device 2 may also generate optical coherence tomographic images (OCT images). The diagnostic imaging device 2 is not limited to an IVUS device, and may be, for example, an angiography device, a CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, etc. In other words, the medical images are not limited to ultrasound tomographic images, and may include optical coherence tomographic images, X-ray fluoroscopic images (angiography images, CT (Computed Tomography) images, etc.), magnetic resonance imaging (MRI) images, etc.

The information processing device 1 is an information processing device capable of various information processing and sending and receiving information, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. Note that server 1 may be a local server installed in the same facility (hospital, etc.) as the imaging diagnostic device 2, or a cloud server communicatively connected to the imaging diagnostic device 2 via the Internet, etc. The server 1 functions as a search device that searches for medical information of a first patient that is similar to the medical information of a second patient who is the target of treatment by the imaging diagnostic device 2, and outputs the search results to the imaging diagnostic device 2.

Specifically, as described below, server 1 performs machine learning in advance to learn from predetermined training data, and prepares a learning model 50 that inputs clinical information including medical images and outputs support information for supporting patient treatment (see Figure 4). The support information is information for supporting vascular treatment such as catheter surgery (e.g., PCI (Percutaneous Coronary Intervention)), and is, for example, a detection result indicating a lesion (e.g., plaque) in a blood vessel that appears in a medical image. Server 1 inputs the clinical information of a second patient into learning model 50, extracts features of the clinical information, and detects a lesion in the second patient's blood vessel based on the extracted features.

In this embodiment, the server 1 searches the database for medical information of the first patient that is similar to the medical information of the second patient as a similar case based on the features extracted by the learning model 50. The server 1 outputs the searched medical information of the first patient to the imaging diagnostic device 2 and presents it to the user (medical professional).

2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit .
The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc., and performs various information processing, control processing, etc. by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as a static random access memory (SRAM), a dynamic random access memory (DRAM), or a flash memory, and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside.

The auxiliary memory unit 14 is a non-volatile storage area such as a large-capacity memory or hard disk, and stores the program P and other data necessary for the control unit 11 to execute processing. The auxiliary memory unit 14 also stores a learning model 50 and a case DB 141. The learning model 50 is a machine learning model that has learned training data as described above, and is a model that inputs medical information and outputs support information for treatment support. The learning model 50 is expected to be used as a program module that constitutes artificial intelligence software. The case DB 141 is a database that stores patient cases, and stores the medical information of a first patient in association with the features of that medical information.

The auxiliary memory unit 14 may be an external storage device connected to the server 1. The server 1 may also be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed using software.

In addition, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operational input, a display unit that displays images, etc. The server 1 may also be equipped with a reading unit that reads portable storage media 1a such as a CD (Compact Disk), DVD (Digital Versatile Disc), or USB (Universal Serial Bus) memory, and may read and execute the program P from the portable storage medium 1a. Alternatively, the server 1 may read the program P from semiconductor memory 1b.

Figure 3 is an explanatory diagram showing an example of the record layout of case DB 141. Case DB 141 includes a case ID column, a patient information column, a treatment information column, an examination information column, an image information column, and a feature column. The case ID column stores a case ID for identifying the medical information of each first patient who is a case. The patient information column, treatment information column, examination information column, image information column, and feature column each store the patient information, treatment information, examination information, medical images, and medical information features of the first patient in association with the case ID.

Patient information is basic information about the first patient who underwent treatment and is the medical record of the first patient. Patient information includes, for example, the patient's age, gender, diagnosis, risk factors (presence or absence of lifestyle-related diseases, etc.), medical history, medication history, etc.

The treatment information indicates the details of the treatment performed on the first patient and is a record of vascular treatment such as PCI. The treatment information includes, for example, the date of treatment, the location of the treated lesion (hereinafter referred to as the "lesion site"), the characteristics of the lesion, the puncture site of the catheter 21, the amount of contrast agent administered, the time taken to capture the fluoroscopic image, the presence and content of additional procedures before stent placement, the presence and content of additional procedures after stent placement, the name, diameter, length, total number, total stent length of the placed stents, maximum balloon inflation diameter, maximum inflation pressure, treatment method for the bifurcation lesion, and post-treatment progress information (e.g., the presence or absence of complications). Note that, although the stent diameter is expressed as a diameter in this embodiment, it may also be expressed as a radius.

The examination information is the examination record of the first patient other than imaging diagnosis. The examination information includes, for example, blood test results, the number of affected blood vessels, left ventricular ejection fraction, and a history of cardiovascular-related sudden events (such as myocardial infarction).

Medical images are images of a patient's blood vessels, and as described above, include intravascular tomographic images (ultrasound tomographic images, optical coherence tomographic images), X-ray fluoroscopic images (angiography images), CT images, MRI images, etc. In this embodiment, we will mainly explain intravascular tomographic images (ultrasound tomographic images) acquired by the imaging diagnostic device 2 as an example of a medical image.

Figure 4 is an explanatory diagram showing an overview of embodiment 1. Figure 4 conceptually illustrates how a learning model 50 is used to search for medical information of a first patient that is similar to the medical information of a second patient. An overview of this embodiment will be explained based on Figure 4.

Learning model 50 is a machine learning model that takes a patient's medical information as input and outputs support information to assist the patient in vascular treatment. For example, learning model 50 is a neural network model generated by deep learning, and is a CNN (Convolutional Neural Network) that extracts features of input data using multiple convolutional layers. Learning model 50 has an intermediate layer (hidden layer) in which convolutional layers that convolve input data and pooling layers that map the convolved data are alternately connected, and extracts features of the input data.

In this embodiment, the learning model 50 is described as a CNN, but it may also be another neural network model such as an RNN (Recurrent Neural Network), or a model based on other learning algorithms such as a GAN (Generative Adversarial Network), an SVM (Support Vector Machine), or a decision tree.

In this embodiment, when the server 1 receives input of a medical image (tomographic image), which is one piece of medical information, it generates a learning model 50 that detects lesions in the medical image. For example, the server 1 generates a semantic segmentation model (such as U-net) or a Faster R-CNN (Region CNN) as the learning model 50.

Semantic segmentation is a type of CNN, a type of Encoder-Decoder model that generates output data from input data. In addition to a convolutional layer that compresses input image data, a semantic segmentation model also has a deconvolutional layer that maps (enlarges) the features obtained through compression to the original image size. The deconvolutional layer identifies which objects exist at which positions in the image on a pixel-by-pixel basis based on the features extracted by the convolutional layer, and generates a binarized label image that indicates which object each pixel corresponds to.

Faster R-CNN is a CNN primarily used for object detection, and in addition to an intermediate layer that extracts features from the input image, it also includes an RPN (Region Proposal Network) that estimates image regions where objects may exist. In Faster R-CNN, the features extracted from the image and the coordinate range of the image region estimated by the RPN are input to the output layer, ultimately detecting objects within the input image.

The server 1 generates these models as learning models 50 and uses them to detect lesions. Note that all of the above models are examples, and the learning model 50 may be any model that is capable of identifying the location and type of lesion within a medical image. In this embodiment, as an example, the learning model 50 will be described as a semantic segmentation model.

In Figure 4, the right side of the learning model 50 conceptually illustrates the lesions contained in the medical image with hatching. The learning model 50 receives input of intravascular tomographic images (medical images) acquired by the image diagnostic device 2, and detects image regions corresponding to lesions such as plaque. Note that plaque is an example of a lesion, and other parts such as calcified tissue, dissections (flaps) in the blood vessel wall, neointima, and other parts may also be detected.

Server 1 performs learning using training data in which image regions corresponding to lesions are labeled for training medical images. Specifically, in the training data, labels (metadata) indicating the coordinate range corresponding to the lesion and the type of lesion are assigned to the training medical images.

The training data is not limited to the data of the first patient stored in case DB 141, but may be data of other patients. Furthermore, the training data is not limited to data of actual patients, but may be virtual data inflated using a data generation method such as GAN.

The server 1 inputs the training tomographic image into the learning model 50 and obtains the detection results of the lesion as output. Specifically, it obtains as output a labeled image in which each pixel in the image area corresponding to the lesion is labeled with a value indicating the type of lesion.

Server 1 compares the detection results output from learning model 50 with the coordinate range of the correct image area indicated by the training data and the type of lesion, and optimizes parameters such as the weights between neurons so that the two are similar. In this way, server 1 generates learning model 50.

In addition to medical images, the server 1 may also use text data such as patient information and examination information as input to the learning model 50. In this case, for example, the server 1 may convert the text data using one-hot encoding or other means, and input it to the learning model 50 as a categorical variable representing the category of the medical image. This makes it possible to detect lesions using medical information other than images.

Note that it is not necessary to include text data such as patient information and examination information in the input to the learning model 50, and the medical information input to the learning model 50 may consist only of medical images.

Furthermore, while the above description has been given assuming that the learning model 50 detects lesions, the present embodiment is not limited to this. For example, the learning model 50 may be a model that takes medical information as input and predicts the correct treatment method (e.g., the length, diameter, or balloon expansion diameter of the stent to be used in vascular treatment). Alternatively, the learning model 50 may be a model that takes medical information as input and predicts post-treatment progress information (e.g., the probability of complications occurring). In this way, the learning model 50 may be any model that takes medical information as input and outputs support information for supporting treatment, and the content of the support information is not particularly limited.

In this embodiment, the server 1 uses the above-described learning model 50 to extract features of the second patient's medical information and uses these to search for the first patient's medical information. Specifically, the server 1 uses the features of a tomographic image in which a lesion is detected, among multiple tomographic images of the second patient's blood vessels, to search for the first patient's medical information.

For example, when treating a second patient, the server 1 acquires intravascular tomographic images of the second patient from the imaging diagnostic device 2. Specifically, the server 1 acquires from the imaging diagnostic device 2 a moving image consisting of multiple frame images captured along the longitudinal direction of the blood vessel in accordance with the scanning of the catheter 21. The server 1 sequentially inputs the tomographic images corresponding to each frame into the learning model 50 and detects the lesion.

The timing of performing this series of processes is not limited to when treatment is being performed; recorded video images can also be input into the learning model 50 after the fact to detect the lesion.

When a lesion is detected from the tomographic image, the server 1 identifies the feature extracted from the tomographic image in which the lesion is detected as the feature of the medical information of the second patient. The server 1 searches for medical information of the first patient that is similar to the medical information of the second patient based on the identified feature and the feature of the medical information of each first patient stored in the case DB 141.

Specifically, the server 1 calculates the similarity based on the feature quantities of both the documents and performs a search based on the calculated similarity. There are no particular limitations on the method for calculating the similarity, but for example, the server 1 calculates the cosine similarity of the feature quantities expressed as vectors.

For example, the server 1 determines whether the degree of similarity is equal to or greater than a predetermined threshold, and searches for the medical information of first patients determined to be equal to or greater than the threshold as similar cases. The server 1 determines a search rank for the medical information of each first patient according to the degree of similarity, and outputs the medical information of each first patient to the display device 23 according to the search rank.

When determining the search ranking for the first patient, it is preferable for the server 1 to change the search ranking according to the progress information of the first patient after treatment included in the medical information. Progress information is data indicating the patient's condition after treatment, such as data indicating whether or not a complication has occurred. For example, if the searched first patient has developed a complication, the server 1 will lower the search ranking. Note that the search ranking may also be changed based on not only whether or not a complication has occurred, but also the type of complication that has occurred.

In the above, the search ranking is changed according to the progress information, but this embodiment is not limited to this. For example, even if a complication has occurred, the server 1 may not change the search ranking, but may simply indicate that a complication has occurred on the display screen described below. This allows the user to decide whether or not to refer to the medical information of the first patient searched for as a similar case.

Figure 5 is an explanatory diagram showing an example of a display screen for similar cases. Figure 5 illustrates an example of a screen displaying medical information for the first patient searched for as a similar case. For example, the display device 23 displays a subject image 501, a case image 502, a case information column 503, a list of similar cases 504, a column for specifying an examiner 505, and a summary column 506.

Subject image 501 is a medical image of a second patient, and is an ultrasound tomographic image of a patient currently undergoing an ultrasound examination. Case image 502 is a medical image of a first patient found to be a similar case, and is an ultrasound tomographic image of the first patient's blood vessels that has been determined to be similar to the tomographic image of the second patient. Case information field 503 is a display field that displays medical information about the first patient other than the medical image displayed as case image 502, such as text data about the patient, examination information, treatment information, etc. The treatment information includes not only the surgical record of the treatment of the first patient, but also progress information about the first patient after treatment (e.g., complications). Server 1 reads the medical image and patient information of the first patient from case DB 141 and displays them on display device 23.

Note that in the example screen shown in Figure 5, only ultrasound tomographic images are displayed as the subject image 501 and the case image 502, but it goes without saying that other medical images, such as X-ray fluoroscopic images, may also be displayed.

The similar case list 504 is a display field that shows a list of search results for similar cases, and displays a summary of the medical information of each first patient found as a similar case, and the degree of similarity with the medical information of each first patient. For example, the display device 23 displays thumbnail images of reduced medical images of each first patient and the degree of similarity according to search ranking. The display device 23 accepts a selection input to select one of the thumbnail images from the similar case list 504. The display device 23 displays the medical image of the first patient corresponding to the selected case as a case image 502, and also displays patient information, etc. of the first patient in the case information field 503.

When displaying a list of medical information for each first patient in the similar case list 504, the display device 23 may, for example, change the display depending on the progress information (whether or not a complication has occurred). For example, if the first patient has developed a complication, the display device 23 displays a predetermined icon indicating the occurrence of the complication in association with the thumbnail image. This improves user convenience.

The practitioner designation field 505 is an input field for entering search conditions for similar cases, and is an input field for accepting input specifying the treatment provider (e.g., a doctor) of the first patient to be searched. For example, the display device 23 accepts input specifying the name of the practitioner in the practitioner designation field 505. When input specifying the practitioner is accepted in the practitioner designation field 505, the server 1 narrows the search to cases of the designated practitioner, i.e., the medical information of the first patient treated by that practitioner.

In this embodiment, the provider can be specified as a search condition, but other search conditions, such as the patient's age, diagnosis, medical history, etc., may also be specified in addition to the provider of treatment.

The tally column 506 is a display column that displays the tally results of tallying specific information items for the medical information of multiple first patients found to be similar cases. There are no particular limitations on the tally items, but for example, the server 1 tallys up the treatment methods of multiple first patients found to be similar cases and outputs the tally results to the display device 23. Specifically, the server 1 tallys up the number of cases by treatment method, such as catheter surgery, bypass surgery, or drug therapy, and displays the tally results.

6 is a flowchart showing the procedure of the process of generating the learning model 50. The process of generating the learning model 50 from training data will be described with reference to FIG.
The control unit 11 of the server 1 acquires training data in which correct values of support information are assigned to the training clinical information (step S11). For example, the control unit 11 acquires training data in which image regions corresponding to lesions in the medical images and label data indicating the types of lesions are assigned to the training medical images. As described above, the training input data may include text data such as patient information, and a learning model 50 may be generated that receives medical images and text data as inputs and outputs support information.

Based on the training data, the control unit 11 generates a learning model 50 that outputs support information for supporting patient treatment when medical information is input (step S12). Specifically, as described above, the control unit 11 generates a neural network such as a CNN as the learning model 50. The control unit 11 inputs training medical images into the learning model 50 and obtains the detection results of lesions in the medical images as output. The control unit 11 compares the detection results of lesions with correct label data and generates the learning model 50 by optimizing parameters such as the weights between neurons so that the two are similar. The control unit 11 then completes the series of processes.

7 is a flowchart showing the procedure for the search process for similar cases. The process for searching for medical information on a first patient, who is a similar case, will be described with reference to FIG.
The control unit 11 of the server 1 accepts input of a designated practitioner who performed treatment for the first patient for the medical information of the first patient to be searched (step S31). Note that if no practitioner is designated, the control unit 11 skips step S31.

The control unit 11 acquires medical information for the second patient to be treated (step S32). As described above, the medical information may include text data such as patient information and examination information in addition to the medical images acquired from the diagnostic imaging device 2.

The control unit 11 inputs the acquired medical information of the second patient into the learning model 50, extracts features, and outputs support information based on the extracted features (step S33). Specifically, the control unit 11 sequentially inputs multiple tomographic images captured consecutively along the longitudinal direction of the blood vessel into the learning model 50, and detects the lesion from each tomographic image.

The control unit 11 calculates the similarity between the medical information of the first patient and the second patient based on the feature values of the tomographic image in which the lesion was detected and the feature values of the medical information of each first patient stored in the case DB 141 (step S34). The control unit 11 searches for medical information of the first patient that is similar to the medical information of the second patient based on the calculated similarity (step S35). Note that if a designated practitioner input was received in step S31, the control unit 11 narrows the search to first patients treated by the designated practitioner.

The control unit 11 changes the search ranking of the medical information of each retrieved first patient according to the progress information of the first patient after treatment (step S36). For example, if a complication occurs in the first patient after treatment, the control unit 11 lowers the search ranking of the first patient.

The control unit 11 also tallys up the medical information of the retrieved multiple first patients (step S37). For example, the control unit 11 tallys up the number of cases of catheter surgery, bypass surgery, and drug therapy for the first patient's treatment methods. The control unit 11 outputs the medical information of the first patient retrieved from the case DB 141 to the imaging diagnostic device 2 (step S38), completing the series of processes.

As described above, according to this embodiment 1, it is possible to effectively search for similar cases that are similar to the second patient being treated.

In addition, according to this embodiment 1, by presenting the similarity along with the medical information of the first patient, the user can easily understand similar cases.

In addition, according to this embodiment 1, by presenting not only the records of the first patient's treatment but also progress information after treatment as the first patient's medical information, the validity of the first patient's treatment records can be determined.

In addition, according to this embodiment 1, by changing the search order of the medical information of the first patient depending on the progress information, the medical information of each first patient can be presented in a more appropriate order.

In addition, according to this embodiment 1, by using a learning model 50 that inputs medical information and outputs support information, it is possible to preferably extract features of the medical information of the second patient.

In addition, according to this embodiment 1, by using a model for lesion detection as the learning model 50, it is possible to detect the lesion of the second patient while simultaneously searching for similar cases.

In addition, according to this embodiment 1, it also has the function of specifying the practitioner and aggregating the treatment method (medical information), which increases convenience for the user.

(Embodiment 2)
In this embodiment, features are extracted from image data (medical images) and text data included in medical information, and the extracted features are used for case search. Note that the same reference numerals are used to designate the same parts as in the first embodiment, and the description thereof will be omitted.

Figure 8 is an explanatory diagram of a learning model 50 according to embodiment 2. Figure 8 illustrates a so-called multimodal model that includes a network (first extractor 51) for extracting features from image data and a network (second extractor 52) for extracting features from text data. An overview of this embodiment will be explained based on Figure 8.

The learning model 50 according to this embodiment comprises a first extractor 51, a second extractor 52, and a detector 53. The first extractor 51 is an extractor that extracts features of image data, and is, for example, a CNN network. The second extractor 52 is an extractor that extracts features of text data such as patient information and examination information, and is, for example, an RNN network. The first extractor 51 accepts input of a patient's medical image and extracts features of the medical image. The second extractor 52 accepts input of text data such as patient information and extracts features of the text data.

Detector 53 is a network that detects lesions using the features extracted by first extractor 51 and second extractor 52 as input. For example, detector 53 includes a combination layer that combines the features of the medical image and text data, and a decoder (deconvolution layer) that generates a labeled image by labeling image regions corresponding to lesions based on the combined features. Detector 53 combines the features of the medical image and text data, inputs them into the decoder, and outputs a labeled image that is the result of lesion detection.

When a lesion is detected, the server 1 searches the case DB 141 for the medical information of the first patient, who is a similar case, based on the features extracted by the first extractor 51 and the second extractor 52 at the time of detection. For example, the server 1 calculates the similarity separately based on the features of the medical images and the text data. That is, the server 1 calculates the similarity of the medical images based on the features of the medical images of the first patient and the features of the medical images of the second patient, and calculates the similarity of the text data (patient information, etc.) based on the features of the text data of the first patient and the features of the text data of the second patient.

The server 1 calculates a total similarity to comprehensively evaluate the similarity between the first patient and the second patient from the similarities between the medical images and the text data. For example, the server 1 calculates the total similarity by calculating the sum of the similarities between the medical images and the text data.

For example, the server 1 may use the average value of both data as the total similarity. Furthermore, for example, the server 1 may weight one of the data (e.g., medical images) to place more importance on the other data, and calculate the total similarity by weighting the similarity of the other data.

The server 1 determines whether the calculated total similarity is equal to or greater than a threshold value, and searches for the medical information of the first patient that is determined to be equal to or greater than the threshold value as a similar case. The server 1 then outputs the searched medical information of the first patient to the display device 23. In this case, the server 1 may display not only the total similarity, but also the similarity of each of the medical image and text data on the display device 23. This allows the user to understand whether the images or the text data are similar.

9 is a flowchart showing the procedure of the process of generating the learning model 50 according to embodiment 2. After acquiring training data for generating the learning model 50 (step S11), the server 1 executes the following process.
The control unit 11 of the server 1 generates a learning model 50 based on the training data (step S201). In this embodiment, as described above, a multimodal model is generated that extracts features from both medical images and text data. The control unit 11 inputs training medical images to a first extractor 51 to extract features, inputs training text data to a second extractor 52 to extract features, and inputs both features to a detector 53 to detect lesions. The control unit 11 compares the lesion detection results with the correct label data and generates the learning model 50 by optimizing parameters such as the weights between neurons so that the two are similar. The control unit 11 then completes the series of processes.

10 is a flowchart showing the procedure of the search process for similar cases according to Embodiment 2. After acquiring the medical information of the second patient (step S32), the server 1 executes the following process.
The control unit 11 of the server 1 inputs the medical images included in the medical information of the second patient to the first extractor 51 and the text data other than the medical images to the second extractor 52, extracts feature quantities from each of the medical images and the text data, and detects the lesion (step S221). Specifically, as described above, the control unit 11 inputs the feature quantities from both to the detector 53, combines the feature quantities, and detects the lesion based on the combined feature quantities.

If a lesion is detected, the control unit 11 calculates the similarity between the medical information of each first patient based on the features of the medical image extracted by the first extractor 51 and the features of the text data extracted by the second extractor 52 (step S222). Specifically, the control unit 11 calculates the similarity between the first patient and the second patient for each medical image and text data, and calculates the total similarity by, for example, summing the similarities of the medical image and the text data. The control unit 11 searches the case DB 141 for the medical information of the first patient based on the calculated similarity (step S223), and proceeds to step S36.

Note that, although a multimodal model was cited above as an example of learning model 50, it is also possible to prepare, for example, a model that outputs support information from medical images and a model that outputs support information from text data independently, and use each model to extract features of the medical images and text data, respectively. In other words, learning model 50 is not limited to a single model, but may be multiple models.

In addition, while the above description assumes that only one type of image is input to the learning model 50, multiple types of medical images may be input to the learning model 50, features may be extracted, and the extracted features may be used to search for similar cases. For example, the server 1 inputs X-ray fluoroscopic images in addition to ultrasound tomographic images to the learning model 50, and extracts features from each image. In this case, the learning model 50 may be provided with separate extractors for extracting features from ultrasound tomographic images and X-ray fluoroscopic images, and these may be connected in parallel to the detector 53. In this way, multiple types of medical images may be input to the learning model 50.

As described above, according to this second embodiment, by extracting the features of each of the medical images and text data contained in the medical information separately, the similarity between each of the medical images and text data can be evaluated separately, making it possible to search for similar cases more effectively.

The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the meaning described above, and is intended to include all modifications within the meaning and scope of the claims.

1. Server (information processing device)
11 Control unit 12 Main memory unit 13 Communication unit 14 Auxiliary memory unit 141 Case DB
1a Portable storage medium 1b Semiconductor memory 2 Diagnostic imaging device 21 Catheter 22 Image processing device 23 Display device 50 Learning model 501 Subject image 502 Case image 503 Case information field 504 List of similar cases 505 Physician designation field 506 Summary field 51 First extractor 52 Second extractor 53 Detector N Network P Program

Claims (12)

a storage unit that stores medical information of a first patient who has already been treated, the medical information including progress information indicating whether or not complications have occurred after the treatment of the first patient, in association with a feature amount of the medical information;
an acquisition unit that acquires medical information of a second patient to be treated;
an extraction unit that extracts features of the medical information of the second patient;
a calculation unit that calculates a similarity between the medical information of the first patient and the medical information of the second patient based on a feature amount of the medical information of the first patient and a feature amount of the medical information of the second patient;
a search unit that searches for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient based on the calculated similarity;
an output unit that outputs the searched medical information of the plurality of first patients in accordance with a search order determined according to the similarity;
The information processing device according to claim 1, wherein the search unit lowers the search ranking of medical information of the first patient when the first patient has developed a complication. The information processing apparatus according to claim 1 , wherein the output unit outputs a degree of similarity between the searched medical information of the first patient and the searched medical information of the second patient. 3. The information processing device according to claim 1, wherein the extraction unit extracts features of the medical information of the second patient using a trained model when the medical information is input, and outputs support information for medical support based on the extracted features. the model is a model that, when a medical image of a patient's hollow organ is input, extracts feature amounts from the medical image, detects a lesion in the medical image based on the extracted feature amounts, and outputs a detection result;
The extraction unit
inputting the medical image of the second patient into the model to detect the lesion;
The information processing apparatus according to claim 3 , wherein a feature amount of the medical image in which the lesion is detected is extracted as a feature amount of the medical information of the second patient. the medical image is a cross-sectional image of a patient's blood vessel;
5. The information processing device according to claim 4, wherein the extraction unit inputs a plurality of the tomographic images taken along a longitudinal direction of the blood vessel of the second patient into the model, and detects the lesion area from any of the plurality of tomographic images. the medical information of the first patient and the second patient includes medical images and text data;
the model includes a first extractor that extracts a feature amount of the medical image and a second extractor that extracts a feature amount of the text data;
The information processing device according to any one of claims 3 to 5, wherein the search unit searches for medical information of the first patient that is similar to medical information of the second patient based on feature amounts of the medical image and text data, respectively. The information processing device according to any one of claims 4 to 6, characterized in that the medical image includes at least one of an ultrasound tomographic image, an optical coherence tomographic image, an X-ray fluoroscopic image, a CT image, and an MRI image of the blood vessel of the first patient. a reception unit that receives an input of a designated practitioner who has performed medical care on the first patient;
The search unit searches for medical information of the first patient corresponding to the specified practitioner,
The medical information of the first patient includes information of the practitioner.
8. The information processing device according to claim 1, wherein the information processing device is a computer. a counting unit that counts the medical information of the searched first patients,
the search unit searches for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient;
The information processing device according to any one of claims 1 to 8, wherein the output unit outputs a compilation result of the medical information of the first patient. The information processing device according to any one of claims 1 to 9, wherein the medical information of the first patient is information about an intravascular treatment performed on the first patient. Obtain medical information of a second patient to be treated;
extracting features of medical information of the second patient;
calculating a similarity between the medical information of the first patient and the second patient based on a feature amount of the medical information of the first patient who has already been treated and a feature amount of the medical information of the second patient;
based on the calculated similarity, searching for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient from a storage unit that stores medical information including progress information indicating whether or not complications have occurred after treatment of the first patient in association with feature amounts of the medical information;
outputting the searched medical information of the plurality of first patients in accordance with a search order determined according to the similarity;
an information processing method, wherein a computer executes a process of lowering the search ranking of medical information of the first patient when the searched first patient has developed a complication; Obtaining medical information of a second patient to be treated;
extracting features of medical information of the second patient;
calculating a similarity between the medical information of the first patient and the second patient based on a feature amount of the medical information of the first patient who has already been treated and a feature amount of the medical information of the second patient;
based on the calculated similarity, searching for a plurality of pieces of medical information of the first patient that are similar to the medical information of the second patient from a storage unit that stores medical information including progress information indicating whether or not complications have occurred after treatment of the first patient in association with feature amounts of the medical information;
outputting the searched medical information of the plurality of first patients in accordance with a search order determined according to the degree of similarity;
A program causing a computer to execute a process of lowering the search ranking of medical information of a searched first patient if the searched first patient has developed a complication.