JP7536862B2 - PROGRAM, INFORMATION PROCESSING METHOD, INFORMATION PROCESSING APPARATUS AND MODEL GENERATION METHOD - Google Patents

Description

The present invention relates to a program, an information processing method, an information processing device, and a model generation method.

Various methods have been proposed to support treatment based on medical images that visualize the inside of the human body, such as ultrasound images, optical coherence tomography (OCT) images, and X-ray images. For example, Patent Document 1 discloses a treatment device selection support system that accepts designation of the range of a diseased area in an intravascular image from an operator, sets the shape of a treatment device appropriate for the diseased area in the designated range, and searches a database for information on treatment devices having the set shape and displays it.

JP 2007-75141 A

However, reading information such as diseased areas from medical images and reflecting it in treatment is not easy and requires extensive knowledge and clinical experience. The invention in Patent Document 1 only searches a database for treatment devices suitable for diseased areas within the range specified by the operator, and does not adequately support treatment based on medical images.

On one aspect, the aim is to provide a program, etc. that can effectively support treatment based on medical images.

A program relating to one aspect causes a computer to execute a process of acquiring a medical image of a patient's blood vessels, inputting the acquired medical image into a first model that has been trained to detect a lesion in the blood vessel when the medical image is input, detecting the lesion, generating a second medical image in which an image area corresponding to the detected lesion is processed so that it can be identified, and inputting the generated second medical image into a second model that has been trained to output balloon information related to a balloon to be used in the patient's endovascular treatment when the second medical image is input, and outputting the balloon information.

In one aspect, it can be used to effectively support medical image-based treatment.

FIG. 1 is an explanatory diagram showing an example of the configuration of a treatment support system. FIG. 2 is a block diagram showing an example of the configuration of a server. FIG. 2 is an explanatory diagram showing an example of a record layout of a medical treatment DB. FIG. 2 is an explanatory diagram showing an overview of a learning model. FIG. 2 is an explanatory diagram showing details of a learning model. FIG. 2 is an explanatory diagram showing an example of a display screen of a display device. 13 is a flowchart showing the steps of a learning model generation process. 13 is a flowchart showing a procedure for outputting treatment information. 13 is a flowchart showing a procedure of an output process of treatment information according to the first modified example. FIG. 11 is an explanatory diagram of a learning model according to the second modified example. FIG. 11 is an explanatory diagram of a learning model according to the second embodiment; FIG. 11 is an explanatory diagram showing an example of a display of a second medical image according to the second embodiment; FIG. 11 is an explanatory diagram showing an example of a display of a second medical image according to the second embodiment; FIG. 11 is an explanatory diagram showing an example of a display of a second medical image according to the second embodiment; FIG. 11 is an explanatory diagram showing an example of a display of a second medical image according to the second embodiment; 13 is a flowchart showing the steps of a process for generating a learning model according to the second embodiment. 13 is a flowchart showing a procedure of an output process of treatment information according to the second embodiment. 13 is a flowchart showing the procedure of a treatment information output process according to Modification 3.

The present invention will now be described in detail with reference to the drawings showing embodiments thereof.
(Embodiment 1)
1 is an explanatory diagram showing an example of the configuration of a treatment support system. In this embodiment, a treatment support system that presents treatment information for supporting a patient's treatment to a user (medical worker) based on patient information on a patient undergoing endovascular treatment and medical images of the patient's blood vessels will be described. The treatment support system includes an information processing device 1 and an image diagnostic system 2. The information processing device 1 and the image diagnostic system 2 are communicatively connected to a network N such as a LAN (Local Area Network) or the Internet.

In this embodiment, intravascular treatment is described as an example, however, the target tubular organ is not limited to blood vessels, but may be other tubular organs such as the bile duct, pancreatic duct, bronchi, intestines, etc.

The imaging diagnostic system 2 has an intravascular imaging device 21, a fluoroscopic imaging device 22, and a display device 23. The intravascular imaging device 21 is a device for imaging intravascular tomographic images of a patient, and is, for example, an IVUS (Intravascular Ultrasound) device that performs ultrasound examinations using a catheter 211. The catheter 211 is a medical instrument inserted into the patient's blood vessels, and is equipped with a piezoelectric element that transmits ultrasound waves and receives reflected waves from within the blood vessels. The intravascular imaging device 21 generates an ultrasound tomographic image based on the signal of the reflected wave received by the catheter 211, and displays it on the display device 23.

In this embodiment, the intravascular imaging diagnostic device 21 generates an ultrasound tomographic image, but it may also generate, for example, an optical coherence tomographic image (OCT image).

The fluoroscopic imaging device 22 is a device unit for taking fluoroscopic images through the inside of a patient's body, for example an angiography device for performing angiography examinations. The fluoroscopic imaging device 22 is equipped with an X-ray source 221 and an X-ray sensor 222, and takes an X-ray fluoroscopic image of the patient by the X-ray sensor 222 receiving X-rays irradiated from the X-ray source 221. For example, an X-ray opaque marker is attached to the tip of the catheter 211, and the position of the catheter 211 is visualized in the fluoroscopic image. The fluoroscopic imaging device 22 displays the fluoroscopic image that visualizes the position of the catheter 211 on the display device 23, and presents it together with an intravascular tomographic image.

Although ultrasound tomographic images, optical coherence tomographic images, and angiography images have been given above as examples of medical images, medical images may also be computed tomography (CT) images, magnetic resonance imaging (MRI) images, etc.

The information processing device 1 is an information processing device capable of various information processing and sending and receiving information, such as a server computer, a personal computer, etc. In this embodiment, the information processing device 1 is a server computer, and will be referred to as server 1 for simplicity in the following. The server 1 may be a local server installed in the same facility (hospital, etc.) as the image diagnostic system 2, or a cloud server connected via the Internet or the like. The server 1 functions as a treatment support device that outputs treatment information for supporting intravascular treatment of a patient based on patient information on a patient to be treated and medical images (tomographic images and fluoroscopic images) of the patient. Specifically, as described later, the server 1 performs machine learning to learn predetermined training data, and has a learning model 50 (see FIG. 4) that uses patient information and medical images as inputs and outputs treatment information prepared in advance. The server 1 inputs patient information and medical images of the patient to be treated into the learning model 50, and acquires treatment information from the learning model 50. The server 1 outputs the treatment information to the image diagnostic system 2 and displays it on the display device 23.

The treatment information is information for supporting the treatment of a patient and is information for assisting a user (medical worker) who treats a patient. In this embodiment, endovascular treatment is taken as an example, and information for supporting PCI (Percutaneous Coronary Intervention) using a catheter 211 is output as treatment information. For example, the treatment information includes the position and characteristics of a lesion such as plaque, a treatment device such as a stent or balloon to be used, a recommended dose of contrast agent and imaging time when taking a fluoroscopic image, and progress information after treatment (e.g., risk of complications). The server 1 estimates this information using the learning model 50 and displays it on the display device 23. The server 1 also detects objects such as a lesion or a treatment device from within a medical image, generates a second medical image in which the detected object is processed so as to be identifiable, and displays it on the display device 23.

In this embodiment, the treatment information generation process is performed in the server 1 using the learning model 50, but the learning model 50 constructed by the server 1 may be installed in the image diagnosis system 2 and the treatment information generation process may be performed locally.

2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit .
The control unit 11 has one or more arithmetic processing devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an AI chip (AI semiconductor), etc., and performs various information processing by reading and executing a program P stored in the auxiliary memory unit 14. The main memory unit 12 is a temporary storage area such as an SRAM (Static Random Access Memory), a DRAM (Dynamic Random Access Memory), a flash memory, etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing processing related to communication, and transmits and receives information to and from the outside.

The auxiliary memory unit 14 is a non-volatile storage area such as a large capacity memory or a hard disk, and stores the program P and other data necessary for the control unit 11 to execute processing. The auxiliary memory unit 14 also stores a learning model 50 and a medical DB 141. The learning model 50 is a machine learning model that has learned training data as described above, and is a model that uses patient information and medical images as input and outputs treatment information to support patient treatment. The learning model 50 is expected to be used as a program module that constitutes artificial intelligence software. The medical DB 141 is a database that stores patient medical data.

The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

In addition, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. The server 1 may also include a reading unit that reads portable storage medium 1a such as a CD (Compact Disk), a DVD (Digital Versatile Disc), a USB (Universal Serial Bus) memory, or an external hard disk, and may read and execute the program P from the portable storage medium 1a. Alternatively, the server 1 may read the program P from the semiconductor memory 1b.

Figure 3 is an explanatory diagram showing an example of a record layout of medical DB 141. Medical DB 141 includes a medical ID column, a patient information column, a treatment information column, and an image information column. The medical ID column stores a medical ID for identifying each medical data. The patient information column, treatment information column, and image information column each store, in correspondence with a medical ID, patient information on the patient being treated, treatment information on the treatment administered to the patient, and medical images obtained by examining the patient.

Patient information is information about patients who have received treatment, and is the patient's medical records. Treatment information includes, for example, the patient's age, sex, diagnosis, risk factors (presence or absence of lifestyle-related diseases, etc.), medical history, as well as medical history, concomitant medications, blood test results, number of affected blood vessels, left ventricular ejection fraction, and history of sudden cardiovascular events (such as myocardial infarction).

The treatment information is information about the intravascular treatment performed on the patient, such as a record of PCI. The treatment information includes, for example, the date of PCI, the location of the treated lesion (hereinafter referred to as the "lesion site"), the nature of the lesion, the puncture site of the catheter 211, the total amount of contrast agent used, as well as the time of fluoroscopic image capture, the presence or absence and content of additional procedures before stent placement, the presence or absence and content of additional procedures after stent placement, the type of stent placed (e.g., product name), diameter, length, total number, type and length of balloons, maximum expansion diameter, maximum expansion pressure, expansion time, decay time, treatment method for bifurcation lesions, and progress information after treatment (e.g., the presence or absence of complications). In this embodiment, the diameters of the stent and balloon are expressed as diameters, but it goes without saying that they may be radii.

Medical images are images of a patient's blood vessels, and as described above, include cross-sectional images of the inside of blood vessels (ultrasound tomography, optical coherence tomography, etc.), fluoroscopic images that visualize the inside of the patient's body using X-rays (angiography images, computed tomography images, etc.), and magnetic resonance images. In addition, the image information also includes physiological functional evaluation results such as FFR (Fractional Flow Reserve). In this embodiment, as an example, ultrasound tomography images and angiography images are used as medical images to input the learning model 50.

Figure 4 is an explanatory diagram showing an overview of the learning model 50. The learning model 50 is a machine learning model that takes patient information and medical images as input and outputs treatment information to assist in the treatment of the patient. The server 1 performs machine learning to learn from predetermined training data to generate the learning model 50 in advance. The server 1 then inputs the medical images of the patient acquired from the image diagnostic system 2 and patient information related to the patient into the learning model 50 to generate treatment information.

Specifically, as shown in Fig. 4, the learning model 50 receives as input patient information stored in the medical DB 141, tomographic images captured by the intravascular imaging diagnostic device 21, and fluoroscopic images captured by the fluoroscopic imaging device 22. The learning model 50 then outputs the estimation results for each item shown in the upper right corner of Fig. 4 as treatment information.

For example, the treatment information includes information about a lesion in a blood vessel. Specifically, the information about the lesion includes the site of the lesion (e.g., the type of blood vessel in which the lesion is located) and the characteristics of the lesion. Note that the information about the lesion may include not only the position and characteristics, but also the length of the lesion in the longitudinal direction of the blood vessel, the size (area) of the lesion in the cross section of the blood vessel, etc.

The treatment information also includes information about a treatment device to be inserted into a blood vessel. The treatment device is, for example, a stent or a balloon for expanding a blood vessel. The treatment device is not limited to a stent, and may include, for example, a rotorablate for resecting a lesion. The learning model 50 outputs stent information related to the stent, such as the type of stent to be used (e.g., product name), shape (diameter and length), total number of stents, presence or absence and content of additional procedures before stent placement, and presence or absence and content of additional procedures after stent placement. The learning model 50 also outputs balloon information related to the balloon, such as the type of balloon to be used (e.g., product name), shape (length), and expansion conditions (maximum expansion diameter, maximum expansion pressure, expansion time, and attenuation time required for the balloon to deflate after expansion).

The information regarding the therapeutic devices may also include the order in which each therapeutic device will be used. This can provide convenient assistance in selecting a therapeutic device when the user selects each therapeutic device and performs treatment.

The treatment information also includes information on the imaging conditions of the fluoroscopic images. The information on the imaging conditions includes, for example, the amount of contrast agent administered and the imaging time of the fluoroscopic images ("fluoroscopic time" in FIG. 4). The server 1 trains the learning model 50 to learn the amount of contrast agent administered, the imaging time, etc. when treatment is performed on the patient, and is capable of outputting the recommended amount of contrast agent administered, the imaging time, etc.

The treatment information also includes progress information that estimates the progress of the patient after treatment. The progress information is, for example, an estimation result that estimates the risk of complications. The server 1 allows the learning model 50 to learn complications that have developed after treatment for patients who have already undergone treatment, and is capable of outputting complications that are likely to develop after treatment as progress information. As described below, it is preferable to output a probability value that evaluates the degree of development of each complication for each of multiple complications that may develop after treatment (see Figure 6).

As an example, Figure 4 illustrates a case where a tomographic image of plaque occurring in a patient's blood vessel and a fluoroscopic image showing the position of catheter 211 at the time the tomographic image was generated are input to learning model 50 along with patient information. In this case, learning model 50 outputs the position and characteristics of the lesion corresponding to plaque, the stent to be used and its expansion diameter and expansion pressure, progress information after treatment, etc.

The learning model 50 outputs the above-mentioned various information as treatment information, and also generates a second medical image by processing the input medical image so that a specific object in the image can be identified, and outputs this as one of the treatment information. The second medical image is an image in which an image area corresponding to an object such as a lesion is processed as described above, and the image area of the object is displayed in a different display mode from other image areas. For example, the learning model 50 generates a second medical image in which an image area corresponding to an object is displayed in a color other than black and white in a tomographic image displayed in black and white. The server 1 outputs the second medical image to the display device 23 together with information such as the lesion site, and displays it.

In addition, the object targeted by the learning model 50 is not limited to the lesion, and may be, for example, a treatment device to be inserted into the patient's blood vessel (for example, a stent already placed in the patient's blood vessel). The learning model 50 only needs to be capable of generating a second medical image that can identify a specific object present in the blood vessel.

For simplicity, in the following explanation, the image region corresponding to an object in a medical image will be referred to as the "object region."

Figure 5 is an explanatory diagram showing details of the learning model 50. In Figure 5, a model consisting of a first model 51 and a second model 52 is illustrated as an example of the learning model 50. The details of the learning model 50 will be explained based on Figure 5.

The first model 51 is a machine learning model that receives medical images (tomographic images and fluoroscopic images) as input and outputs detection results of objects detected within the images. For example, the first model 51 is a neural network model generated by deep learning, and is a Convolutional Neural Network (CNN) that extracts features of the input image using multiple convolution layers. The first model 51 includes an intermediate layer (hidden layer) in which a convolution layer that convolves pixel information of the input image and a pooling layer that maps the convolved pixel information are alternately connected, and extracts features (feature map) of the input image.

In this embodiment, the first model 51 is described as a CNN, but it may also be a model based on other learning algorithms, such as a Generative Adversarial Network (GAN), a Recurrent Neural Network (RNN), a Support Vector Machine (SVM), a decision tree, etc.

In this embodiment, the server 1 generates a first model 51 that identifies, on a pixel-by-pixel basis, whether each pixel in an input medical image corresponds to an object region. For example, the server 1 generates a semantic segmentation model (such as U-net) or a MASK R-CNN (Region CNN) as the first model 51.

The semantic segmentation model is a type of CNN, and a type of Encoder-Decoder model that generates output data from input data. In addition to the convolution layer that compresses the input image data, the semantic segmentation model also has a deconvolution layer that maps (enlarges) the features obtained by compression to the original image size. The deconvolution layer identifies which object is located at which position in the image on a pixel-by-pixel basis based on the features extracted in the convolution layer, and generates a binarized label image that indicates which object each pixel corresponds to.

MASK R-CNN is a variation of Faster R-CNN, which is mainly used for object detection, and is a model that connects a deconvolution layer to Faster R-CNN. In MASK R-CNN, the image features extracted by CNN and the coordinate range of the target object extracted by RPN (Region Proposal Network) are input to the deconvolution layer, and a mask image that finally masks the image area of the object in the input image is generated.

The server 1 generates these models as the first model 51 and uses them to detect objects. Note that all of the above models are merely examples, and the first model 51 may be any model capable of identifying the position and shape of an object in a medical image. In this embodiment, as an example, the first model 51 will be described as a semantic segmentation model.

On the right side of Figure 4, object regions in the intravascular tomographic image and fluoroscopic image are conceptually illustrated with hatching. Objects detected by the first model 51 are, for example, lesions such as plaque, or therapeutic devices such as placed stents. The server 1 inputs medical images into the first model 51 and detects various objects.

Plaque is an example of a lesion, and other parts such as narrowed parts of blood vessels, calcified tissue, dissections (flaps) in the blood vessel wall, neointima, and the like may be detected. A stent is an example of a therapeutic device, and other devices may be detected. Lesions and therapeutic devices are examples of objects, and other objects may be detected.

Returning to Figure 5 for further explanation, the server 1 performs learning using training data in which training medical images are labeled with data indicating object regions as exemplified in Figure 4. Specifically, in the training data, labels (metadata) indicating the coordinate ranges corresponding to the object regions and the types of objects are assigned to the training medical images.

The server 1 inputs a training medical image into the first model 51 and obtains the detection result of the detected object as an output. Specifically, it obtains a labeled image as an output in which each pixel in the object region is labeled with a value indicating the type of object.

The server 1 compares the detection result output from the first model 51 with the coordinate range of the correct object area and the type of object indicated by the training data, and optimizes parameters such as the weights between neurons so that the two are close to each other. In this way, the server 1 generates the first model 51.

In this embodiment, two types of images, a tomographic image and a fluoroscopic image, are processed. For example, the server 1 may prepare two first models 51 corresponding to each image, or the first model 51 may be configured as a multimodal model having two networks for processing the tomographic image and the fluoroscopic image, respectively. This makes it possible to detect objects from each image.

In this embodiment, the first model 51 receives as input a plurality of frames of medical images (video) that are successive in time series, and detects objects from the medical images of each frame. Specifically, the first model 51 receives as input a plurality of frames of medical images (e.g., tomographic images) that are successive along the longitudinal direction of a blood vessel in accordance with the scanning of the catheter 211. The first model 51 detects objects from the medical images of each successive frame along the time axis.

For convenience in the following explanation, each successive frame of medical image will simply be referred to as a "frame image."

The server 1 may input multiple frame images one by one into the first model 51 and detect objects from each frame image individually, but it is preferable to input multiple consecutive frame images simultaneously so that object areas can be detected from multiple frame images simultaneously. For example, the server 1 sets the first model 51 as a 3D-CNN (e.g., 3D U-net) that handles three-dimensional input data. The server 1 handles the first model 51 as three-dimensional data with the coordinates of the two-dimensional frame images as the two axes and the time at which each frame image was acquired (the time at which the frame image was generated) as the one axis. The server 1 inputs multiple frame images (e.g., 16 frames) for a predetermined unit time into the first model 51 as one set and simultaneously outputs label images in which the object area is labeled for each of the multiple frame images. This makes it possible to detect objects taking into account the frame images before and after the frame images that are consecutive in time series, thereby improving detection accuracy.

In the above, the time series of frame images can be processed by treating the data as three-dimensional data including the time axis, but the present embodiment is not limited to this. For example, the server 1 may detect an object from a plurality of consecutive frame images by making the first model 51 a model that combines a CNN and an RNN. In this case, for example, a LSTM (Long-Short Term Memory) layer is inserted after the intermediate layer related to the CNN, and the object is detected by referring to the feature amounts extracted from the previous and next frame images. Even in this case, processing can be performed taking into account the previous and next frame images in the same manner as above, and the detection accuracy can be improved.

Next, the second model 52 will be described. The second model 52 is a machine learning model that receives patient information and medical images as input, and outputs treatment information to support the patient's endovascular treatment. For example, the second model 52 is a CNN that includes an intermediate layer in which convolutional layers and pooling layers are alternately connected, and extracts features of the input data.

In this embodiment, the second model 52 is described as being a CNN, but it may also be a model based on other learning algorithms such as GAN, RNN, SVM, decision tree, etc.

The server 1 learns using training data in which correct treatment information has been added to the training patient information and medical images. The training patient information and medical images are patient information and medical images of patients who have already undergone treatment, and are the medical records of the patients, and medical images such as tomographic images and fluoroscopic images obtained during treatment. The correct treatment information is a record of the treatment (PCI) performed on the patient. The server 1 provides the patient information and medical images stored in the medical DB 141, and the treatment information, as training data to the second model 52, and learns.

Specifically, the server 1 inputs data such as the patient's age, sex, and diagnosis contained in the patient information as categorical variables indicating attributes of the medical image into the second model 52. The server 1 inputs the patient information as categorical variables into the second model 52 together with the tomographic image and the fluoroscopic image, and obtains treatment information for each item shown in the upper right corner of Figure 4 as output.

As shown in the upper right of Figure 4, the output items of the second model 52 include items that are shown as classification results, such as "lesion site" and "lesion characteristics," and items that are expressed as continuous values, such as "total contrast time" and "fluoroscopy time." In this way, the second model 52 simultaneously handles so-called classification problems and regression problems, and can, for example, treat all items as regression problems and output continuous values, and convert items to be shown as classification results from continuous values to binary values. Alternatively, a separate output layer corresponding to each item can be provided, and the features extracted in the intermediate layer can be input to each output layer, so that estimation of each item can be performed separately. Alternatively, the second model 52 itself can be prepared for each item, and estimation can be performed separately.

The server 1 may directly use the medical image of the patient's blood vessels as input to the second model 52, but in this embodiment, a second medical image is generated in which the object area detected in the first model 51 is processed so as to be identifiable, and input to the second model 52. The second medical image is an image in which the object area is displayed in a different display mode depending on the type of object, for example, an image in which a label image output from the first model 51 is superimposed on the original medical image.

For example, the server 1 processes the label image output from the first model 51 into a semi-transparent mask and superimposes it on the original medical image. In this case, the server 1 makes the display mode of each object region different depending on the type of object, for example by changing the display color of the mask depending on the type of object. In this way, the server 1 generates a medical image in which the object region is displayed in a different display mode from the other regions, and inputs the medical image to the second model 52.

The server 1 inputs the training patient information and the medical image with the processed object region into the second model 52, and obtains the treatment information as output. The server 1 compares the output treatment information with the correct treatment information, and optimizes parameters such as the weights between neurons so that the two are close to each other. In this way, the server 1 generates the second model 52.

In this embodiment, two types of models, a first model 51 for object detection and a second model 52 for generating treatment information, are combined, but object detection may be performed on a rule basis. In other words, it is not essential that the learning model 50 includes the first model 51. Objects may be detected on a rule basis as a preprocessing step for input to the learning model 50, and the detection results may be input to the learning model 50.

When generating treatment information for a patient who is to undergo new treatment, the server 1 acquires medical images of the patient from the image diagnostic system 2, inputs the images to the learning model 50, and generates treatment information including the second medical image. This processing may be performed in real time when the images are acquired, or recorded medical images (videos) may be acquired together and processed afterwards.

The server 1 outputs the generated treatment information to the display device 23 for display. Note that in this embodiment, the treatment information is described as being output to the image diagnostic system 2, but it goes without saying that the treatment information may be output to a device other than the image diagnostic system 2 (e.g., a personal computer) and displayed thereon.

Figure 6 is an explanatory diagram showing an example of a display screen of the display device 23. The display device 23 displays each piece of treatment information output from the server 1 and presents it to the user. For example, the display device 23 displays a list 71, a progress information column 72, a tomographic image 73, and an fluoroscopic image 74.

The list 71 is a table showing a list of the estimation results for each item output from the second model 52. The list 71 includes, for example, information on the lesion, information on the treatment device, information on the fluoroscopic image capturing conditions, and other information. The display device 23 presents this treatment information to the user in the list 71 to support the endovascular treatment.

The progress information column 72 is a display column showing the estimated results of estimating the progress of the user after treatment, and as described above, is the estimated result regarding the onset of a complication. The server 1 uses the learning model 50 to calculate the probability value of onset for each of multiple complications, and displays the probability values in a list in the progress information column 72 in association with the complication name.

The display device 23 also displays a tomographic image 73 generated by the intravascular imaging diagnostic device 21 and a fluoroscopic image 74 captured by the fluoroscopic image capturing device 22. In this case, the display device 23 displays a second medical image obtained by processing the object region of the original medical image as the tomographic image 73 and the fluoroscopic image 74.

Fig. 7 is a flowchart showing the procedure of the process of generating the learning model 50. The process of generating the learning model 50 by learning the training data will be described with reference to Fig. 7.
The control unit 11 of the server 1 acquires training data in which correct treatment information is assigned to the training patient information and medical images (step S11). The training patient information and medical images are patient information and medical images of patients who have already undergone treatment. The treatment information is information on the treatment performed on the patient, and includes label data indicating the object area in addition to the treatment records stored in the medical DB 141.

Based on the training data, the control unit 11 generates a first model 51 that detects objects when a medical image is input (step S12). For example, as described above, the control unit 11 generates a CNN related to semantic segmentation as the first model 51. The control unit 11 inputs a medical image for training to the first model and obtains the detection result of detecting the object region as an output. The control unit 11 compares the detected object region with the correct label data, and optimizes parameters such as the weights between neurons so that the two are similar to each other, thereby generating the first model 51.

Furthermore, the control unit 11 generates a second model 52 that outputs treatment information to support the treatment of a patient when patient information and a medical image are input based on the training data (step S13). Specifically, the control unit 11 generates the second model 52 that outputs treatment information by inputting the patient information and the second medical image in which the object region is processed based on the detection result in the first model 51. The control unit 11 inputs the training patient information and the second medical image in which the object region is processed based on the correct label data to the second model, and obtains the treatment information as an output. The control unit 11 compares the output treatment information with the correct treatment information, and optimizes parameters such as the weights between neurons so that the two are similar to each other, thereby generating the second model 52. The control unit 11 ends the series of processes.

8 is a flowchart showing the procedure of the treatment information output process. The process content when the treatment information of the patient to be treated is output using the learning model 50 will be described with reference to FIG.
The control unit 11 acquires patient information on a patient to be treated and medical images of the blood vessels of the patient (step S31). The control unit 11 inputs the acquired medical images to the first model 51 and detects objects (step S32). Specifically, the control unit 11 inputs tomographic images and fluoroscopic images captured by the intravascular imaging device 21 and the fluoroscopic imaging device 22 to the first model 51 and detects objects in the tomographic images and fluoroscopic images.

The control unit 11 inputs the second medical image with the object region processed based on the detection result in step S32 and the patient information into the second model 52 to generate treatment information (step S33). The control unit 11 outputs the generated treatment information to the display device 23 and displays it (step S34). Specifically, the control unit 11 outputs the second medical image with the object region processed as treatment information, in addition to information such as the lesion in the blood vessel, the treatment device to be used, and the shooting conditions of the fluoroscopic image, and displays it on the display device 23. The control unit 11 ends the series of processes.

As described above, according to this embodiment 1, by using a learning model 50 that has already learned training data, treatment based on medical images can be appropriately supported.

Furthermore, according to this embodiment 1, the position, characteristics, etc. of a lesion in a blood vessel (hollow organ) can be estimated and presented to the user.

In addition, according to this embodiment 1, the type, shape, number of uses, order of use, etc. of the therapeutic device to be used can be estimated and presented to the user.

Furthermore, according to this embodiment 1, it is possible to estimate and present suitable shooting conditions for fluoroscopic images.

In addition, according to this embodiment 1, it is possible to estimate progress information after treatment, such as the onset of complications, and present it to the user.

Furthermore, according to this embodiment 1, by combining a first model 51 that detects objects from medical images with a second model 52 that generates treatment information based on the detection results by the first model 51, the correct position, shape, etc. of the object can be provided to the second model 52, thereby improving the estimation accuracy of the treatment information.

Furthermore, according to this embodiment 1, by generating a second medical image in which the object area has been processed, it is possible to effectively assist in identifying lesions, etc.

Furthermore, according to this embodiment 1, the accuracy of estimating treatment information can be improved by inputting multiple frame images generated in time series into the learning model 50.

In addition, according to this embodiment 1, by using not only cross-sectional images of blood vessels (hollow organs) but also fluoroscopic images as medical images, not only local images but also images of the entire blood vessels can be provided to the learning model 50, thereby improving the estimation accuracy.

(Variation 1)
In the first embodiment, a form has been described in which treatment information is output using the learning model 50. In this modified example, a form will be described in which a correction input for correcting the output treatment information is received from a user, and re-learning is performed based on the corrected treatment information.

9 is a flowchart showing the procedure of the treatment information output process according to Modification 1. After outputting the treatment information (step S34), the server 1 executes the following process.
The control unit 11 of the server 1 accepts, from the user, input for correcting the treatment information output to the display device 23 (step S35). For example, the control unit 11 accepts input for correcting information of each item displayed in the list 71 on the display screen exemplified in Fig. 6. Furthermore, the server 1 accepts input of the correct coordinate range, type, etc., for the second medical image displayed as the tomographic image 73, when the coordinate range of the object region, the type of the object, etc. differ from the actual ones.

When input of corrections to the treatment information is received, the control unit 11 performs re-learning using the patient information and medical images input to the learning model 50 and the corrected treatment information as training data, and updates the learning model 50 (step S36). That is, the control unit 11 optimizes parameters such as the weights between neurons so that the treatment information output from the learning model 50 approximates the corrected treatment information, and regenerates the learning model 50. The control unit 11 ends the series of processes.

As described above, according to this variant example 1, the learning model 50 can be optimized through the operation of this system.

(Variation 2)
In the first embodiment, a form has been described in which an object is detected by the first model 51, and a second medical image in which the object region has been processed is input to the second model 52. In this modified example, a form will be described in which the type, dimensions, etc. of the object are identified from the detection result, and object information indicating the type, dimensions, etc. of the identified object is used as input to the second model 52.

Figure 10 is an explanatory diagram of the learning model 50 according to variant example 2. As in embodiment 1, the learning model 50 according to this variant example also includes a first model 51 and a second model 52. In this variant example, the server 1 performs image analysis of the object area detected by the first model 51, and identifies the type, dimensions, etc. of the object.

For example, the server 1 identifies the type, dimensions, etc. of a stent that has already been placed in the patient's blood vessel. Note that in this modified example, the object to be identified is a stent, but it may be another object such as a lesion. The server 1 performs image analysis of the object area detected as a stent, identifies the name of the placed stent, and identifies the diameter, length, etc. of the stent. The server 1 inputs data on the identified stent type, dimensions, etc. into the second model 52 as object information, and generates treatment information.

In this case, the server 1 may input not only the object information (text data) but also the original medical image input to the first model 51 to the second model 52. Alternatively, the server 1 may input the second medical image with the object region processed to the second model 52, as in the first embodiment.

As described above, in this modified example, image analysis is performed as preprocessing of the second model 52, and object information is identified and input to the second model 52. This provides data such as the type and dimensions of the object to the second model 52, thereby improving the estimation accuracy.

This variant is the same as embodiment 1 except that image analysis is performed as pre-processing of the second model 52, so detailed explanations of the flowchart and other details will be omitted.

(Embodiment 2)
In this embodiment, the present invention relates to intravascular treatment, and in particular to a form for assisting balloon manipulation. Note that the same reference numerals are used to designate the same parts as in the first embodiment, and the description thereof will be omitted.

Figure 11 is an explanatory diagram of a learning model 50 according to embodiment 2. As in embodiment 1, the learning model 50 according to this embodiment also includes a first model 51 and a second model 52, and generates a second medical image in which the object region has been processed, and other treatment information. As shown in the upper right of Figure 11, the treatment information includes balloon information regarding a balloon that expands the patient's blood vessels, and outputs the type of balloon to be used, the expansion conditions (maximum expansion diameter, maximum expansion pressure, etc.), etc.

In this embodiment, the learning model 50 further includes a third model 53. The third model 53 is a machine learning model that estimates a target position in a blood vessel to be expanded by a balloon and a target expansion diameter when a pre-treatment medical image is input. Specifically, the third model 53 receives a medical image as input and detects an image area to be expanded by a balloon in the image. In addition to the balloon information generated by the second model 52, the server 1 detects an image area to be expanded by a balloon using the third model 53, generates a second medical image showing the detected area, and displays it on the display device 23 as one piece of balloon information.

For convenience, in the following explanation, the area to be expanded by the balloon will be referred to as the "target area."

For example, the server 1 uses Mask R-CNN as the third model 53. As described above, Mask R-CNN is a CNN that detects a target image area from an input image, and is a model that can identify the target image area on a pixel-by-pixel basis. For example, the server 1 generates the third model 53 using as training data data in which a label indicating the coordinate range within the balloon-expanded image is added to a fluoroscopic image of a patient who has already undergone treatment.

Note that the third model 53 is not limited to Mask R-CNN, but may be other CNNs such as U-net, or other machine learning models such as GAN.

For example, the third model 53 detects the target area as a bounding box in the shape of a sandbag, as shown in Fig. 11. The target medical image may be not only a fluoroscopic image but also a tomographic image. The target area may be any area that correctly indicates the area to be expanded by the balloon, and may be rectangular in shape, for example.

The server 1 may input the fluoroscopic image acquired from the fluoroscopic image capture device 22 directly to the third model 53, but it is preferable to input the second medical image in which an object area such as a lesion has been processed by the first model 51, as shown in Fig. 11. This allows the third model 53 to determine the target area taking into account the position, shape, etc. of the lesion to be treated.

As in variant example 2, the type and dimensions of an object such as a lesion may be identified from the second medical image, and the identified object information may be provided to the third model 53.

For example, the server 1 generates a second medical image that simultaneously shows the lesion and the area to be expanded by the balloon by further superimposing a bounding box indicating the target area detected by the third model 53 on the second medical image in which the object area corresponding to the lesion has been processed. The server 1 may also generate an image that only surrounds the target area as the second medical image. The server 1 outputs the generated second medical image to the display device 23 as a fluoroscopic image 74.

In this embodiment, a first model 51 for detecting objects such as lesions and a third model 53 for detecting the target area to be expanded by a balloon are prepared separately, but both may be the same model and object detection and target area detection may be performed simultaneously.

Figures 12 and 13 are explanatory diagrams showing an example of the display of the second medical image according to embodiment 2. In this embodiment, the display device 23 displays the second medical image showing the target area to be expanded with the balloon, and also displays the balloon currently inserted in the blood vessel in a identifiable manner in the second medical image, thereby assisting in balloon operation. Figures 12A to 13B chronologically illustrate how the balloon is inserted into the blood vessel, reaches the target position, and is expanded.

12A and 12B show the balloon being inserted into the target area. The display device 23 displays the object area 741 and the target area 742 in the fluoroscopic image 74 in a distinguishable manner, for example by color display. As described above, the object area 741 corresponds to the lesion, and the target area 742 corresponds to the area in the blood vessel where the balloon 743 is to be placed.

Furthermore, the display device 23 displays a balloon 743 inserted into the blood vessel and a rectangular balloon area 744 representing the current position of the balloon 743. The server 1 detects at least the current position and current expansion diameter of the balloon (hereinafter referred to as the "current diameter") from the fluoroscopic image acquired from the fluoroscopic image capturing device 22, and displays them identifiably using a method such as color display.

The detection of the balloon 743 may be performed using the first model 51 or by image recognition using pattern matching.

Furthermore, the server 1 calculates a difference value between the detected current position of the balloon 743 and the target position of the balloon 743 indicated by the target area 742, and displays it in the upper left of the fluoroscopic image 74. The current position of the balloon 743 may be, for example, the midpoint of the elongated balloon 743, or the tip of the balloon 743, and any point of the balloon 743 may be detected as the current position. The target position of the balloon 743 may be, for example, the center of gravity of the punch-bag-shaped target area 742, or the midpoint of the arc of the target area 742 located on the opposite side of the balloon 743, and any point within the target area 742 may be set as the target position. The server 1 sequentially detects the balloon 743 from the fluoroscopic image, calculates a difference value between the current position and the target position, and displays it on the display device 23.

In addition to the position of the balloon 743, the server 1 may also calculate the difference between the target expansion diameter and the current diameter of the balloon and display it on the display device 23.

The server 1 determines whether the balloon 743 has reached the target position based on the difference between the current position and the target position. If it is determined that the balloon 743 has reached the target position, the server 1 calculates the expansion rate by dividing the current diameter by the target expansion diameter based on the current diameter of the balloon 743 and the target expansion diameter of the balloon 743 indicated by the target region 742, and displays it in the upper left of the fluoroscopic image 74. Note that the target expansion diameter is the width of the target region 742 in the direction perpendicular to the longitudinal direction of the blood vessel, and is the distance between both long sides of the punch-bag-shaped target region 742.

13A and 13B show the process from when the balloon 743 reaches the target position until the expansion is completed. When the balloon 743 reaches the target position, the display device 23 switches the display in the upper left corner of the fluoroscopic image 74 to the expansion rate. The server 1 sequentially detects the current diameter of the balloon 743, calculates the expansion rate, and displays it on the display device 23.

As described above, guidance is displayed according to the current position and current diameter of balloon 743 to assist in balloon operation. Note that the guidance displayed is not limited to information according to the position and expansion diameter of balloon 743, and other information may be displayed. For example, when balloon 743 has completed expansion and started to contract, display device 23 displays the remaining time required for balloon 743 to contract based on the decay time estimated by second model 52. This allows for more optimal assistance in balloon operation.

FIG. 14 is a flowchart showing the steps of a process for generating a learning model 50 according to the second embodiment.
The control unit 11 of the server 1 acquires training data for generating the learning model 50 (step S201). The training data according to the present embodiment includes medical images of patients who have undergone treatment, label data indicating object regions such as lesions, and label data indicating image regions (target regions) to be expanded by a balloon. The control unit 11 transitions the process to step S12.

After executing the process of step S13, the control unit 11 generates a third model 53 that detects the target area to be expanded by a balloon when a medical image is input (step S202). Specifically, as described above, the control unit 11 generates Mask R-CNN as the third model 53. For example, the control unit 11 inputs a second medical image in which the object area of the training medical image has been processed according to the label data into the third model 53, and obtains the detection result of the target area detection as an output. The control unit 11 compares the obtained target area with the correct label data, and optimizes parameters such as the weights between neurons so that the two are close to each other, thereby generating the third model 53. The control unit 11 ends the series of processes.

15 is a flowchart showing the procedure of the treatment information output process according to embodiment 2. After generating the treatment information using the second model 52 (step S33), the server 1 executes the following process.
The control unit 11 inputs the pre-treatment medical image of the patient to the third model 53 and detects the target area to be expanded by the balloon (step S221). Specifically, as described above, the control unit 11 inputs the second medical image (fluoroscopic image) generated using the first model 51 to the third model 53 and detects the target area indicating the target position and the target expansion diameter of the balloon.

The control unit 11 detects the balloon inserted into the patient's blood vessel from the medical image (step S222). The control unit 11 generates a second medical image showing the detected balloon and the target area, and outputs the second medical image together with the treatment information generated in step S33 to the display device 23 (step S223).

The control unit 11 calculates the difference between the current position of the balloon and the target position of the balloon indicated by the target area, and outputs it to the display device 23 (step S224). The control unit 11 determines whether the balloon has reached the target position based on the difference between the current position and the target position (step S226). If it is determined that the balloon has not reached the target position (S225: NO), the control unit 11 returns the process to step S224. If it is determined that the balloon has reached the target position (S225: YES), the control unit 11 calculates the expansion rate from the current diameter of the balloon and the target expansion diameter, and outputs it to the display device 23 (step S226).

The control unit 11 determines whether the expansion to the target expansion diameter is complete (step S227). If the control unit 11 determines that the expansion is not complete (S227: NO), the control unit 11 returns the process to step S226. If the control unit 11 determines that the expansion is complete (S227: YES), the control unit 11 ends the series of processes.

In the above, the type and shape of the balloon to be used in the learning model 50 are automatically predicted, and the target position and expansion conditions of the balloon are predicted, but the present embodiment is not limited to this. For example, the server 1 may receive a specification input from the user specifying the type and shape of the balloon to be used in the endovascular treatment, input information on the specified balloon (first balloon information) to the learning model 50, and predict information on the target position and expansion conditions of the balloon (second balloon information). For example, the server 1 inputs the first balloon information specified by the user together with the medical image into the third model 53 to detect the target area when the balloon specified by the user is used. This makes it possible to provide support for balloon operation according to the user's request.

As described above, according to this embodiment 2, it is possible to effectively assist in balloon operation during endovascular treatment.

(Variation 3)
In the first modification, a form in which re-learning is performed by accepting correction input after outputting treatment information has been described. Similarly, in the second embodiment, re-learning may be performed by accepting correction input after outputting balloon information.

16 is a flowchart showing the procedure of the treatment information output process according to the modified example 3. After outputting the treatment information including the balloon information (step S223), the server 1 executes the following process.
The control unit 11 of the server 1 accepts input to modify the output balloon information (step S241). For example, the control unit 11 accepts input to modify the type of balloon displayed as the balloon information, the expansion conditions, etc. The control unit 11 also accepts input to modify the coordinate range of the target region for the second medical image. The control unit 11 transitions the process to step S224.

If it is determined that the balloon expansion is complete (S227: YES), the control unit 11 performs re-learning using the original medical image input to the learning model 50 and the corrected balloon information as training data, and updates the learning model 50 (step S242). That is, the control unit 11 optimizes parameters such as the weights between neurons so that the balloon information output from the learning model 50 approximates the corrected balloon information, and regenerates the learning model 50. The control unit 11 ends the series of processes.

As described above, according to this variant example 3, the learning model 50 can be optimized through the operation of this system.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

1 Server (information processing device)
11 Control unit 12 Main memory unit 13 Communication unit 14 Auxiliary memory unit 141 Medical treatment DB
REFERENCE SIGNS LIST 1a Portable storage medium 1b Semiconductor memory 2 Diagnostic imaging system 21 Intravascular diagnostic imaging device 211 Catheter 221 X-ray source 222 X-ray sensor 22 Fluoroscopic imaging device 23 Display device 50 Learning model 51 First model 52 Second model 53 Third model 71 List 72 Progress information column 73 Tomographic image 74 Fluoroscopic image 741 Object region 742 Target region 743 Balloon 744 Balloon region N Network P Program

Claims (13)

Obtaining medical images of the patient's blood vessels,
inputting the acquired medical image into a first model that has been trained to detect a lesion in the blood vessel when the medical image is input, and detecting the lesion;
generating a second medical image by processing an image area corresponding to the detected lesion so as to be identifiable;
A program that causes a computer to execute a process of inputting the generated second medical image into a second model that has been trained to output balloon information related to a balloon to be used for intravascular treatment of the patient when the second medical image is input, and outputting the balloon information. The program according to claim 1 , further comprising: outputting the balloon information indicating a shape or a type of the balloon to be used for intravascular treatment. The program according to claim 1 or 2, further comprising: outputting the balloon information indicating a target position in the blood vessel to be dilated by the balloon or a dilation condition. The generated second medical image is input to a third model that has been trained to detect a target area to be expanded by the balloon when the second medical image is input, and the target area is detected;
The program according to claim 3 , further comprising: outputting a third medical image showing the detected target region. Detecting a balloon inserted into the blood vessel from the current medical image;
The program according to claim 4 , further comprising: outputting the third medical image showing the detected balloon and the target region. Calculating a difference between the detected current position or current expansion diameter of the balloon and the target position or target expansion diameter of the balloon indicated by the target region;
The program according to claim 5 , further comprising: outputting the calculated difference value. determining whether the balloon has reached the target position based on a difference value between the current position and the target position;
The program according to claim 6 , further comprising: a calculation unit for calculating an expansion ratio by dividing the current expansion diameter by the target expansion diameter when it is determined that the target position has been reached; The program according to any one of claims 1 to 7 , further comprising: outputting the balloon information indicating a decay time to be ensured after the inflation of the balloon. The program according to any one of claims 1 to 8 , further comprising: outputting progress information predicting a risk of developing complications after expanding the balloon as the balloon information. The program according to any one of claims 1 to 9 , wherein the medical image includes at least one of an ultrasonic tomographic image, an optical coherence tomographic image, a fluoroscopic image, and a magnetic resonance image of the blood vessel. Obtaining medical images of the patient's blood vessels,
inputting the acquired medical image into a first model that has been trained to detect a lesion in the blood vessel when the medical image is input, and detecting the lesion;
generating a second medical image by processing an image area corresponding to the detected lesion so as to be identifiable;
An information processing method in which a computer executes a process of inputting the generated second medical image into a second model that has been trained to output balloon information related to a balloon to be used for intravascular treatment of the patient when the second medical image is input, and outputting the balloon information. an acquisition unit that acquires medical images of blood vessels of a patient;
a detection unit that detects a lesion by inputting the acquired medical image into a first model that has been trained to detect the lesion when the medical image is input;
a generating unit that generates a second medical image by processing an image area corresponding to the detected lesion area so that the image area can be identified;
an output unit that inputs the generated second medical image into a second model that has been trained to output balloon information related to a balloon to be used for intravascular treatment of the patient when the second medical image is input, and outputs the balloon information. Obtaining training data including a medical image of a patient's blood vessel, label data indicating an image area corresponding to a lesion in the medical image, and balloon information related to a balloon used in an intravascular treatment of the patient;
generating a first model based on the medical image and the label data, the first model detecting the lesion when the medical image is input;
A model generation method in which a computer executes a process of generating a second model that outputs the balloon information when the second medical image is input, based on a second medical image in which an image region corresponding to the lesion can be identified based on the label data and the balloon information.

Images