JP7597626B2 - PROGRAM, IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS AND MODEL GENERATION METHOD - Google Patents

Description

The present invention relates to a program, an image processing method, an image processing device, and a model generation method.

Intravascular treatment is performed by inserting a catheter into a patient's blood vessel and expanding the narrowed portion of the blood vessel using a balloon or stent. In intravascular treatment, an optical coherence tomography (OCT) is used to capture a tomographic image of the blood vessel from inside the blood vessel, and the blood vessel is observed using the resulting tomographic image. When taking an image using OCT, diffuse reflection and attenuation of light can occur due to blood containing blood cell components such as red blood cells, so a flush operation is performed in which a contrast agent, low molecular weight dextran, saline, or the like is injected into the blood vessel to temporarily create a state where there is no blood. Patent Document 1 discloses a flush liquid used in the flush operation to properly capture a tomographic image to be used in image diagnosis.

Patent No. 6562902

In the flushing operation, it is necessary to appropriately set conditions such as the type and dilution rate of the flushing solution, the injection amount, and the flow rate during injection. If the flushing operation is not performed under appropriate conditions, the flushing operation will have to be repeated and the tomographic images will have to be retaken, which will extend the procedure time. The conditions for the flushing operation must be set taking into account the state of the blood vessels to be treated and the size of the catheter to be used, and it is not easy to select appropriate conditions. Patent Document 1 does not mention the selection of various conditions for the flushing operation.

In one aspect, the objective is to provide a program etc. capable of proposing conditions for flash operation (operation information related to flash operation).

A program according to one aspect causes a computer to execute a process of acquiring an ultrasonic tomographic image of a cross section of a blood vessel using an ultrasonic signal, calculating information about the lumen of the blood vessel based on the acquired ultrasonic tomographic image, and, based on the calculated information about the lumen, identifying operation information about a flash operation when acquiring an image of the cross section of the blood vessel using an optical signal.

In one aspect, conditions for flash operation (operation information regarding flash operation) can be suggested.

FIG. 1 is an explanatory diagram illustrating an example of the configuration of an image diagnostic system. FIG. 1 is an explanatory diagram illustrating an overview of a guiding catheter. FIG. 1 is a block diagram showing an example of the configuration of an image processing device. FIG. 2 is an explanatory diagram showing an example of the configuration of a medical treatment DB; FIG. 1 is an explanatory diagram showing an overview of a learning model. FIG. 13 is a schematic diagram showing an example of a screen of flash information. 13 is a flowchart illustrating an example of a process for generating a second learning model. 13 is a flowchart showing an example of a procedure for a flash information presentation process. FIG. 13 is an explanatory diagram showing a modified example of the learning model.

The program, image processing method, image processing device, and model generation method disclosed herein will be described in detail below with reference to drawings showing embodiments thereof. In the following embodiments, cardiac catheterization, which is an intravascular treatment, will be described as an example.

1 is an explanatory diagram showing an example of the configuration of an image diagnostic system. The image diagnostic system of this embodiment includes an intravascular inspection device 1, a fluoroscopic image capture device 2, an image processing device 3, a display device 4, and an input device 5. The intravascular inspection device 1 is a device for imaging a tomographic image of a patient's blood vessel, and captures a tomographic image of a cross section of the blood vessel from inside the blood vessel. The intravascular inspection device 1 of this embodiment is an image diagnostic device using a dual-type imaging catheter 10 having both functions of intravascular ultrasound (IVUS) and optical coherence tomography (OCT). The dual-type imaging catheter 10 is provided with an IVUS mode for acquiring an ultrasonic tomographic image only by IVUS, an OCT mode for acquiring an optical coherence tomographic image only by OCT, and a dual mode for acquiring both tomographic images by IVUS and OCT, and these modes can be switched for use. The intravascular inspection device 1 of this embodiment is not limited to a configuration using a dual-type imaging catheter 10, and may be configured using an imaging catheter equipped with only IVUS or only OCT. Hereinafter, ultrasound tomographic images and optical coherence tomographic images will be referred to as IVUS images and OCT images, respectively, as appropriate.

The imaging catheter 10 is a medical instrument inserted into a patient's blood vessel, and the intravascular examination device 1 captures IVUS and OCT images of the blood vessel from within the blood vessel using the imaging catheter 10 inserted into the blood vessel. When capturing IVUS or OCT images using the imaging catheter 10, a medical professional first inserts a guiding catheter 60 (see FIG. 2) described below into the patient's blood vessel, and then inserts the imaging catheter 10 into the guiding catheter 60 inserted into the blood vessel. The imaging catheter 10 has an imaging diagnostic probe 11 (see FIG. 2), which includes an ultrasonic transceiver that transmits ultrasonic waves (ultrasonic signals) and receives reflected waves from within the blood vessel, and an optical transceiver that transmits near-infrared light (optical signals) and receives reflected light from within the blood vessel. The intravascular examination device 1 generates an IVUS image based on the signal of the reflected ultrasonic waves received by the ultrasonic transceiver of the imaging catheter 10, and generates an OCT image based on the signal of the reflected light received by the optical transceiver. The IVUS images and OCT images captured by the intravascular inspection device 1 using the imaging catheter 10 are displayed on the display device 4 via the image processing device 3.

When taking an OCT image, the medical staff performs a flush operation in which a flush liquid made of a contrast agent, low molecular weight dextran, saline, or the like is injected into the blood vessel to temporarily create a state in which there is no blood (a state in which the blood is replaced with the flush liquid). In the flush operation, the medical staff injects the flush liquid into the blood vessel via a guiding catheter 60 inserted into the blood vessel.

The fluoroscopic imaging device 2 is a device for taking a fluoroscopic image of the inside of a patient's body. For example, it is an angiography device for taking an image of the blood vessels from outside the patient's body using X-rays while injecting a contrast agent into the blood vessels of the patient, and obtaining an angiographic image, which is a fluoroscopic image of the blood vessels. The fluoroscopic imaging device 2 is equipped with an X-ray source and an X-ray sensor, and the X-ray sensor receives X-rays irradiated from the X-ray source to take an X-ray fluoroscopic image of the patient. Note that X-ray opaque markers that do not transmit X-rays are attached to the tips of the imaging catheter 10 and the guiding catheter 60, and the positions of the imaging catheter 10 and the guiding catheter 60 (X-ray opaque markers) are visualized in the fluoroscopic image. The fluoroscopic image taken by the fluoroscopic imaging device 2 is displayed on the display device 4 via the image processing device 3. In this embodiment, the image diagnostic system is configured to include the fluoroscopic imaging device 2 for taking an angiographic image, but is not particularly limited as long as it is an apparatus for taking images of the patient's blood vessels and the catheters 10 and 60 from multiple directions outside the body.

The display device 4 and the input device 5 are connected to the image processing device 3. The display device 4 is, for example, a liquid crystal display or an organic EL display, and displays medical images such as IVUS images and OCT images taken by the intravascular inspection device 1 and angio images taken by the fluoroscopic image capture device 2. The input device 5 is, for example, a keyboard, mouse, trackball, or microphone, and accepts various operations by the surgeon. The display device 4 and the input device 5 may be stacked together to form a touch panel. The input device 5 and the image processing device 3 may also be configured as one unit. Furthermore, the input device 5 may be a sensor that accepts gesture input, gaze input, or the like.

The image processing device 3 outputs the IVUS and OCT images captured by the intravascular inspection device 1 and the fluoroscopic images captured by the fluoroscopic image capture device 2 to the display device 4 for display. The image processing device 3 also identifies the size (diameter and length in the direction of travel of the blood vessel) of the blood vessel to be captured based on the IVUS image captured by the intravascular inspection device 1, and generates operation information (hereinafter referred to as flash information) regarding the flash operation to be performed when capturing the OCT image based on the identified blood vessel size, etc. Therefore, in this embodiment, by capturing an IVUS image before capturing an OCT image, the image processing device 3 can generate flash information based on the IVUS image and present it to the surgeon.

2 is an explanatory diagram for explaining the outline of the guiding catheter 60. The guiding catheter 60 has a sheath 61 and a branch hub 62 attached to the end of the sheath 61. In the following description, the side of the guiding catheter 60 far from the branch hub 62 is described as the tip side, and the branch hub 62 side is described as the base end side. The sheath 61 forms a continuous tube portion from the tip to the opening 64 of the branch hub 62, and the opening 64 of the branch hub 62 is used as an insertion port of the imaging catheter 10 (imaging diagnostic probe 11) connected to the intravascular inspection device 1. In addition, the branch hub 62 is provided with a side port 63, which is used as a connection port for a syringe 70 filled with a flushing liquid 72. The guiding catheter 60 receives a guide wire previously inserted into a blood vessel, and is guided to the vicinity of the entrance of the coronary artery by the guide wire. Inside the guiding catheter 60, the diagnostic imaging probe 11 connected to the connector portion 13 is inserted through the opening 64 of the branch hub 62.

The imaging catheter 10 has a diagnostic imaging probe 11 and a connector section 13 arranged at the end (base end side) of the diagnostic imaging probe 11. The imaging catheter 10 is connected to an MDU (Motor Drive Unit) (not shown) of the intravascular inspection device 1 via the connector section 13. The diagnostic imaging probe 11 has a sheath (not shown) inserted into the guiding catheter 60, a shaft inserted into the sheath, and a sensor section 12 provided at the tip of the shaft. The tip of the sheath is provided with a guidewire insertion section (not shown) through which a guidewire can be inserted. The guidewire insertion section is used to receive a guidewire previously inserted into a blood vessel and to guide the diagnostic imaging probe 11 to an observation site or an affected area by the guidewire. The sensor section 12 has an ultrasonic transmission/reception section and an optical transmission/reception section. An electric signal cable (not shown) connected to the ultrasonic transmission/reception section and an optical fiber cable (not shown) connected to the optical transmission/reception section are inserted into the shaft, and the electric signal cable and the optical fiber cable are connected to the connector section 13. The diagnostic imaging probe 11 can be advanced and retreated inside the guiding catheter 60, and the shaft and the sensor unit 12 can be advanced and retreated inside the sheath of the diagnostic imaging probe 11 and can also rotate in the circumferential direction. The intravascular inspection device 1 has a drive unit (MDU) to which the imaging catheter 10 is detachably attached by the connector unit 13, and controls the operation of the diagnostic imaging probe 11 inserted into the blood vessel 100 by driving a built-in motor in response to the operation of a medical professional. For example, the MDU performs a pull-back operation to rotate the diagnostic imaging probe 11 in the circumferential direction while pulling it at a constant speed. The diagnostic imaging probe 11 rotates while moving from the tip side to the base end by the pull-back operation, and scans the inside of the blood vessel 100 continuously at a predetermined time interval, thereby continuously taking multiple IVUS images or OCT images that are approximately perpendicular to the diagnostic imaging probe 11.

When performing the flush operation, the medical staff attaches the syringe 70 filled with flush liquid 72 to the side port 63 of the branch hub 62, positions the tip of the diagnostic imaging probe 11 near the observation position, and then pushes in the plunger 71 of the syringe 70. This causes the flush liquid 72 in the syringe 70 to pass through the guiding catheter 60 and be ejected from the tip of the guiding catheter 60 into the blood vessel 100. The ejected flush liquid 72 flushes out the blood in the area, filling it with flush liquid 72. In this state, an OCT image can be taken by the diagnostic imaging probe 11 (sensor unit 12), thereby obtaining an OCT image that is not affected by blood. The plunger 71 is pushed in using, for example, an autoinjector.

Figure 3 is a block diagram showing an example of the configuration of the image processing device 3. The image processing device 3 is a computer, and includes a control unit 31, a main memory unit 32, an input/output I/F 33, an auxiliary memory unit 34, and a reading unit 35. The control unit 31 is configured using one or more arithmetic processing devices such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-purpose computing on graphics processing units), a TPU (Tensor Processing Unit), etc. The control unit 31 is connected to each of the hardware components constituting the image processing device 3 via a bus.

The main memory unit 32 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory, and temporarily stores data required for the control unit 31 to execute arithmetic processing.

The input/output I/F 33 is an interface to which the intravascular inspection device 1, the fluoroscopic imaging device 2, the display device 4, and the input device 5 are connected. The control unit 31 acquires IVUS images or OCT images from the intravascular inspection device 1 and acquires angio images from the fluoroscopic imaging device 2 via the input/output I/F 33. The control unit 31 also displays medical images on the display device 4 by outputting medical image signals of the IVUS images, OCT images, or angio images to the display device 4 via the input/output I/F 33. Furthermore, the control unit 31 accepts information input to the input device 5 via the input/output I/F 33.

The auxiliary storage unit 34 is a storage device such as a hard disk, an EEPROM (Electrically Erasable Programmable ROM), or a flash memory. The auxiliary storage unit 34 stores the computer program P executed by the control unit 31 and various data necessary for the processing of the control unit 31. The auxiliary storage unit 34 also stores the first learning model M1, the second learning model M2, and the medical DB 34a described below. The first learning model M1 is a machine learning model that has learned predetermined training data, and is a model that takes an IVUS image as an input and outputs the area of the vascular lumen in the input IVUS image. The second learning model M2 (second learning model) is a machine learning model that has learned predetermined training data, and is a model that takes vascular information, treatment information, and patient information as input and outputs flash information related to flash operation. The first learning model M1 and the second learning model M2 are expected to be used as program modules that constitute artificial intelligence software. The medical DB 34a is a database that stores the medical data of patients. The auxiliary storage unit 34 may be an external storage device connected to the image processing device 3. The computer program P may be written to the auxiliary storage unit 34 during the manufacturing stage of the image processing device 3, or the image processing device 3 may receive the program via communication from a remote server device and store it in the auxiliary storage unit 34.

The reading unit 35 reads data stored on a recording medium 30 such as a CD (Compact Disk), a DVD (Digital Versatile Disc), or a USB (Universal Serial Bus) memory. The computer program P may be recorded in a readable manner on the recording medium 30, or the reading unit 35 may read the computer program P from the recording medium 30 and store it in the auxiliary storage unit 34. The computer program P may also be recorded in a semiconductor memory, and the control unit 31 may read the computer program P from the semiconductor memory and execute it.

The image processing device 3 may be a multi-computer including multiple computers. The image processing device 3 may also be a server-client system, a cloud server, or a virtual machine virtually constructed by software. In the following explanation, the image processing device 3 is described as being a single computer.

In the image processing device 3 of this embodiment, the control unit 31 executes a process of detecting the size of the lumen of the captured blood vessel based on the IVUS image captured by the intravascular inspection device 1 by reading and executing the computer program P stored in the auxiliary storage unit 34. The control unit 31 also executes a process of generating flash information regarding the flash operation performed when capturing an OCT image based on the detected size of the blood vessel lumen (blood vessel information), information regarding the treatment device used for intravascular treatment by the intravascular inspection device 1 (treatment information), and patient information (patient information). Therefore, the image processing device 3 of this embodiment can present the flash information regarding the flash operation when capturing an OCT image to the surgeon. In addition, in the image processing device 3 of this embodiment, the first learning model M1 is used when detecting the blood vessel lumen in the IVUS image, and the second learning model M2 is used when generating operation information for the flash operation from the blood vessel information, treatment information, and patient information.

Figure 4 is an explanatory diagram showing an example of the configuration of the medical DB 34a. The medical DB 34a includes a medical ID column, a patient information column, a treatment information column, and an image information column. The medical ID column stores a medical ID for identifying each medical data. The patient information column, the treatment information column, and the image information column store, in association with the medical ID, patient information on the patient to be treated, treatment information on the treatment performed on the patient and the treatment to be performed, and medical images obtained by examining the patient. The patient information is information on the patient who has been treated or is scheduled to be treated, and is the medical record of the patient. The patient information includes the patient's age, sex, weight, height, renal function level (e.g., a value obtained by evaluating the Cr value on a five-point scale), diagnosis, medical history, etc. The patient information may also include a medical history, a history of taking medicine, blood test results, etc. The treatment information is information on the intravascular treatment performed on the patient or the intravascular treatment to be performed. The treatment information includes, for example, the date of treatment or the date of planned treatment, the position of the treated lesion (lesion site), the size (diameter, length, volume) and shape of the vascular lumen at the lesion, the type (e.g., product name), diameter, length, etc. of the guiding catheter 60 used or the guiding catheter 60 to be used, the type, diameter, length, etc. of the imaging catheter 10 used or the imaging catheter 10 to be used. The treatment information also includes flash information such as the type, dilution rate, dosage (flash amount), and flow rate (flash flow rate) of the flash liquid (e.g., contrast agent) used in the flash operation when taking the OCT image. The treatment information also includes, in association with the flash information, the success probability when the flash operation is performed under each condition included in the flash information. The success probability is a value registered by the surgeon who performed the flash operation to take the OCT image, and is registered by the surgeon after taking the OCT image, for example. The success of the flash operation means that the surgeon judges that an OCT image that can be sufficiently used for image diagnosis has been taken, and the success probability indicates the satisfaction level of the surgeon on a scale of 100 points, for example. The treatment information also includes the pullback speed and travel distance when capturing an IVUS image or an OCT image, the type of contrast agent used when capturing an angio image with the fluoroscopic image capturing device 2, the dilution rate, and the dosage, etc. The treatment information may also include the characteristics of the lesion, the puncture site of the guiding catheter 60, the capture time of the fluoroscopic image, the type, diameter, length, and total number of stents placed, the type, length, and maximum expansion diameter of the balloon, the maximum expansion pressure, the expansion time, and the decay time, and progress information after treatment (e.g., the presence or absence of complications), etc. The medical images are images of the patient's blood vessels, and include angio images, IVUS images, and OCT images. The medical images may also include computed tomography images (CT images), magnetic resonance images (MRI images), etc.

Figure 5 is an explanatory diagram showing an overview of the learning models M1 and M2. Figure 5A shows the first learning model M1, and Figure 5B shows the second learning model M2. The first learning model M1 is a model that recognizes a specific object contained in an IVUS image. The first learning model M1 is a model that can classify objects in an image on a pixel-by-pixel basis, for example, by semantic segmentation. The first learning model M1 of this embodiment is a machine learning model that inputs one IVUS image and has been trained to recognize the vascular lumen contained in the IVUS image, and outputs the recognition result. Specifically, the first learning model M1 classifies each pixel of the input IVUS image into a region of the vascular lumen and other regions, and outputs a classified IVUS image (hereinafter referred to as a label image) in which each pixel is associated with a label for each region. The first learning model M1 can be configured, for example, by U-Net, FCN (Fully Convolutional Network), SegNet, etc.

The first learning model M1 has an input layer, an intermediate layer, and an output layer (not shown), and the intermediate layer has a convolution layer, a pooling layer, and a deconvolution layer. The convolution layer extracts image features from the pixel information of the image input through the input layer to generate a feature map, and the pooling layer compresses the generated feature map. The deconvolution layer enlarges (maps) the feature map generated by the convolution layer and the pooling layer to the original image size. The deconvolution layer identifies which object exists in which position in the image on a pixel basis based on the features extracted by the convolution layer, and generates a label image indicating which object each pixel corresponds to. As shown on the right side of FIG. 5A, the label image output from the first learning model M1 is an image in which each pixel of the IVUS image is classified into a region of the blood vessel lumen and a region other than the region, and a pixel value corresponding to each region is assigned. In FIG. 5A, the pixels classified into the region of the blood vessel lumen are shown by hatching.

The first learning model M1 having the above-mentioned configuration can be generated by preparing training data including a training IVUS image and a label image in which data indicating an object to be discriminated (here, the vascular lumen) is labeled for each pixel in the IVUS image as shown on the right side of FIG. 5A, and using this training data to machine-learn an unlearned learning model. In the training label image, a coordinate range corresponding to the area of each object and a label indicating the type of each object are assigned to the training IVUS image. The first learning model M1 learns to output the label image included in the training data when an IVUS image included in the training data is input. Specifically, the first learning model M1 performs calculations in the intermediate layer based on the input IVUS image, and obtains a detection result in which each object (here, the vascular lumen) in the IVUS image is detected. More specifically, the first learning model M1 obtains as an output a label image in which a value indicating the type of classified object is labeled for each pixel in the IVUS image. The first learning model M1 then compares the obtained detection result (label image) with the coordinate range of the correct object area and the object type indicated by the training data, and optimizes parameters such as the weights (coupling coefficients) between neurons so that the two are similar. There are no particular limitations on the method of optimizing the parameters, but the steepest descent method, backpropagation method, etc. can be used. This results in a first learning model M1 that, when an IVUS image is input, outputs a label image indicating the area of the vascular lumen in the IVUS image.

The image processing device 3 prepares such a first learning model M1 in advance and uses it to detect the vascular lumen in the IVUS image. The first learning model M1 only needs to be able to identify the position and shape of the vascular lumen in the IVUS image. In this embodiment, the first learning model M1 only needs to be able to recognize the vascular lumen in the IVUS image, but it may also be configured to recognize the vascular wall (tunica media and plaque) and the catheter area in addition to the vascular lumen. The first learning model M1 may be learned by another learning device. The learned first learning model M1 generated by learning by the other learning device is downloaded from the learning device to the image processing device 3, for example, via a network or via the recording medium 30, and stored in the auxiliary storage unit 34.

The second learning model M2 receives as input vascular information on the vascular lumen detected from the IVUS image using the first learning model M1, treatment information on the endovascular treatment to be performed on the patient, and patient information stored in the medical DB 34a. If any of the items of vascular information, treatment information, and patient information cannot be prepared, default information set in advance for each item may be input to the second learning model M2. The second learning model M2 then outputs the estimation results estimated for each item as shown on the right side of FIG. 5B as flash information. The vascular information includes, for example, the diameter, length, and volume of the lesion (treatment target area) in the blood vessel. The vascular information may also include the position of the tip of the guiding catheter 60 relative to the coronary artery, and the engagement state between the tip of the guiding catheter 60 and the coronary artery. For example, the length of the lesion can be the pullback distance in the pullback operation when the intravascular inspection device 1 captures the IVUS image. In addition, the position of the tip of the guiding catheter 60 and the engagement state with the coronary artery can be detected based on, for example, an angio image taken by the fluoroscopic image capture device 2. The pullback distance can be calculated based on the number of IVUS images taken by one pullback operation and the pullback speed.

The treatment information includes, for example, the position of the lesion (for example, the type of blood vessel). The treatment information also includes catheter information related to the catheter, such as the type (for example, product name) and shape (diameter and length) of the guiding catheter 60 to be used, the position where the tip of the guiding catheter 60 is to be placed during the flush operation, the type and shape (diameter and length) of the imaging catheter 10 to be used, and the position where the tip of the imaging catheter 10 is to be placed during the flush operation. The positions of the tips of the guiding catheter 60 and the imaging catheter 10 can be positions identified based on the angio image taken by the fluoroscopic image taking device 2. The treatment information also includes contrast agent information related to the contrast agent, such as the type of contrast agent to be used, the dilution rate, and the internal pressure limit of the syringe 70 used to administer the contrast agent. The treatment information also includes pullback information related to the pullback operation, such as the pullback speed and pullback distance in the pullback operation performed when taking the IVUS image. The treatment information may also include an angio image of the patient. Furthermore, if an intravascular treatment is being performed, the treatment information may include the type, dilution rate, and dosage of contrast agent that has already been used, and the type, dilution rate, and dosage of contrast agent that will be used in the future. The patient information includes, for example, the patient's age (age group), weight, and renal function level. The treatment information and patient information described above can use the information recorded in the medical DB 34a.

The flush information output by the second learning model M2 includes the type and dilution rate of the contrast agent used in the flush solution, the flow rate (flush flow rate) and dosage (flush flow rate) when administering the flush solution, and the probability of success when the flush operation is performed under these conditions. The second learning model M2 is configured, for example, by a CNN (Convolutional Neural Network), but may also be configured using algorithms such as decision trees, random forests, SVMs (Support Vector Machines), and RNNs (Recurrent Neural Networks), or may be configured by combining multiple algorithms.

The second learning model M2 can be generated by preparing training data in which training vascular information, treatment information, and patient information are associated with correct flash information, and using this training data to perform machine learning on an untrained learning model. The training vascular information, treatment information, and patient information can be medical data of patients who have undergone endovascular treatment, and for example, each piece of information stored in the medical DB 34a can be used. The correct flash information can also be flash information included in the patient's medical data. The image processing device 3 inputs the vascular information, treatment information, and patient information included in the training data into the second learning model M2, and trains the second learning model M2 to output the correct flash information included in the training data.

Specifically, the image processing device 3 inputs training vascular information, treatment information, and patient information into the second learning model M2, and obtains flash information for each item illustrated in FIG. 5B as output from the second learning model M2. The image processing device 3 compares the output flash information with correct flash information included in the training data, and optimizes parameters such as the weights (coupling coefficients) between neurons so that the two are close to each other. There are no particular limitations on the method of optimizing the parameters, but the steepest descent method, backpropagation method, etc. can be used. This results in a second learning model M2 that outputs flash information when the patient's vascular information, treatment information, and patient information are input.

Note that, among the items shown in FIG. 5B, "type of contrast agent" is an item shown in the classification result, but the other items are items expressed as continuous values. In this way, the second learning model M2 simultaneously handles so-called classification problems and regression problems, but for example, all items can be regarded as regression problems and continuous values can be output, and items to be shown as classification results can be converted from continuous values to binary values. Alternatively, a separate output layer corresponding to each item can be provided, and the features extracted in the intermediate layer can be input to each output layer, so that estimation of each item can be performed separately. Alternatively, the second learning model M2 itself can be prepared for each item, and estimation can be performed separately.

The image processing device 3 prepares such a second learning model M2 in advance and uses it to generate flash information for a patient who will undergo new treatment. When generating flash information for a patient who will undergo new treatment, the image processing device 3 takes an IVUS image of the patient using the intravascular examination device 1 and detects the size of the vascular lumen from the obtained IVUS image using the first learning model M1. The image processing device 3 also generates flash information by adding the contents of the patient's planned treatment (treatment information) and patient information to the detected size of the vascular lumen (vascular information) and inputting the result into the second learning model M2. The image processing device 3 outputs the generated flash information to the display device 4 for display.

The second learning model M2 may be learned by another learning device. When learning is performed by another learning device, the learned second learning model M2 is downloaded from the learning device to the image processing device 3, for example, via a network or via the recording medium 30, and stored in the auxiliary storage unit 34.

Figure 6 is a schematic diagram showing an example of a screen of flash information. The screen shown in Figure 6 displays two patterns of flash information to be presented to the surgeon: one that emphasizes reducing the dosage (flash amount) of the flush liquid (e.g., contrast agent), and the other that emphasizes reducing the flow rate (flash flow rate) of the flush liquid. The flash information shown in Figure 6 includes the flush flow rate, the flush amount (total amount of contrast agent), and the success rate, and further includes warnings and advice regarding this flash information. The warnings and advice are stored in advance in the auxiliary storage unit 34, for example, depending on whether the flash information emphasizes reducing the flush amount or the flush flow rate. Therefore, the control unit 31 can present the warnings and advice to the surgeon along with the flash information by reading out the warnings and advice according to the contents of the generated flash information from the auxiliary storage unit 34 and displaying them on the display screen.

As shown in the advice of pattern A in FIG. 6, by aligning the tip of the guiding catheter 60 to an appropriate position relative to the blood vessel during the flush operation, the amount of flush liquid flowing into other blood vessels can be reduced, and as a result, the amount of flushing can be reduced. As shown in the advice of pattern B in FIG. 6, by increasing the dilution rate of the contrast agent used in the flush liquid, the viscosity of the flush liquid decreases, making it possible to reduce the load on the blood vessel. In this way, advice that can further reduce the amount of flushing or the flush flow rate can be presented to the surgeon. The type of contrast agent and the dilution rate may be included in the flash information and displayed. The image processing device 3 may also be configured to identify optimal flash information from patient information and display only the identified flash information.

The surgeon identifies the optimal flash information based on each piece of information and precautions for each presented pattern of flash information. For example, the surgeon identifies the optimal pattern of flash information taking into account the patient's physical condition. The surgeon also identifies the optimal flash information taking into account the advice presented along with the flash information and the surgeon's own procedural skills. The surgeon then performs a flash operation with the identified flash information (flash conditions). Here, the surgeon sets the identified flash conditions (flash flow rate, flash volume, etc.) for, for example, an autoinjector, and executes injection of the flash liquid by the autoinjector. After executing the flash operation, the surgeon then executes capture of an OCT image.

When the flash information emphasizes reducing the amount of flash, the precautions and advice presented to the surgeon together with the flash information may be, for example, information advising the surgeon to reduce the total amount of flash by increasing the pullback speed. Note that when the pullback speed is increased, the time required for the pullback operation is shortened, making it possible to perform the flush operation using a smaller amount of flush liquid.

The process of learning the training data and generating the second learning model M2 is described below. FIG. 7 is a flowchart showing an example of the process procedure for generating the second learning model M2. The following process is performed by the control unit 31 of the image processing device 3 according to a computer program P stored in the auxiliary storage unit 34, but may also be performed by another learning device.

The control unit 31 of the image processing device 3 acquires training data that associates vascular information, treatment information, and patient information for one patient with flash information regarding the flash operation performed on this patient (S11). The flash information includes the success probability of the flash operation under each condition included in the flash information. The vascular information, treatment information, and patient information for the training data and the flash information for the training data can be each piece of information stored in the medical treatment DB 34a. Note that the training data may be registered in a training data DB (not shown) by preparing vascular information, treatment information, and patient information for the training data, and flash information in advance. In this case, the control unit 31 may acquire the training data from the training data DB.

The control unit 31 performs a learning process for the second learning model M2 using the acquired training data (S12). Here, the control unit 31 inputs the vascular information, treatment information, and patient information contained in the training data to the second learning model M2, and acquires output values for each item of flash information. The control unit 31 compares the output values for each output item with the information for each correct item, and optimizes parameters such as the weights between neurons so that the two are close to each other.

The control unit 31 determines whether there is unprocessed data (S13). For example, if training data has been registered in advance in the training data DB, the control unit 31 determines whether there is unprocessed training data in the training data stored in the training data DB. If it is determined that there is unprocessed data (S13: YES), the control unit 31 returns to the process of step S11, and the learning process performs the processes of steps S11 to S12 based on the unprocessed training data. If it is determined that there is no unprocessed data (S13: NO), the control unit 31 ends the series of processes.

By the above-mentioned processing, a second learning model M2 is obtained that is trained to output flash information when the patient's vascular information, treatment information, and patient information are input. The second learning model M2 can be further optimized by repeatedly performing the learning processing using the training data as described above. In addition, a second learning model M2 that has already been trained can also be retrained by performing the above-mentioned processing, in which case a second learning model M2 with higher discrimination accuracy can be obtained.

Figure 8 is a flowchart showing an example of a procedure for presenting flash information. The following process is performed by the control unit 31 of the image processing device 3 according to the computer program P stored in the auxiliary storage unit 34. In the image diagnostic system of this embodiment, when a tomographic image of a blood vessel is captured during intravascular treatment, the intravascular inspection device 1 is first used in IVUS mode to capture an IVUS image. Then, if necessary, the intravascular inspection device 1 is used in OCT mode to capture an OCT image. At that time, the image processing device 3 generates conditions (flash information) for the flash operation to be performed when capturing an OCT image based on the previously captured IVUS image, and presents the conditions to the surgeon. The surgeon can select the content of the flash operation to be performed (flash information) taking into account the presented flash information.

The control unit 31 (acquisition unit) of the image processing device 3 acquires an IVUS image of the patient's blood vessel taken from inside the blood vessel from the intravascular inspection device 1 (S21). The control unit 31 then generates flash information based on the acquired IVUS image. Note that after the processing of step S21, the control unit 31 may execute a process of generating flash information based on the IVUS image when, for example, an instruction to capture an OCT image is received from the surgeon via the input device 5.

When generating flash information, the control unit 31 first inputs an IVUS image into the first learning model M1, and detects the vascular lumen in the IVUS image based on the output image from the first learning model M1 (S22). The control unit 31 (calculation unit) then calculates the diameter of the vascular lumen based on the detected vascular lumen (S23). The diameter of the vascular lumen is, for example, the minimum and maximum values of the vascular inner diameter passing through the center of the vascular lumen, and the control unit 31 calculates the diameter based on, for example, the number of pixels on the vascular inner diameter in the IVUS image. The control unit 31 (calculation unit) also calculates the length of the vascular lumen photographed by IVUS based on the detected vascular lumen (S24). The length of the vascular lumen is the length in the running direction (longitudinal direction) of the blood vessel, and is, for example, the pullback distance due to the pullback operation performed when photographing the IVUS image. The pullback distance can be calculated based on the number of IVUS images photographed by one pullback operation and the pullback speed. Furthermore, the control unit 31 (calculation unit) calculates the volume of the vascular lumen based on the detected vascular lumen (S25). The volume of the vascular lumen is a volume calculated from the cross-sectional area and length of the vascular lumen. The cross-sectional area of the vascular lumen may be calculated based on the calculated diameter of the vascular lumen, or may be calculated based on the number of pixels in the vascular lumen in the IVUS image. For example, the control unit 31 calculates the cross-sectional area of the vascular lumen in one IVUS image, and calculates the volume of the vascular lumen in one IVUS image based on the distance between adjacent IVUS images (the distance between the shooting positions of the IVUS images). Then, the control unit 31 calculates the volume of the vascular lumen photographed by IVUS by adding up the volumes of the vascular lumen in each IVUS image. Through such processing, the control unit 31 calculates vascular information related to the vascular lumen, such as the diameter, length, and volume of the vascular lumen photographed by the IVUS image. The calculation process for each of these pieces of information can be performed by the calculation process performed by the conventional intravascular inspection device 1 or image processing device 3. The control unit 31 also detects the position of the tip of the guiding catheter 60 and the tip of the diagnostic imaging probe 11 at this point in time based on an angio image captured by, for example, the fluoroscopic image capture device 2, and uses the detected positions as vascular information.

The control unit 31 (identification unit) reads out the patient's treatment information and patient information from the medical DB 34a, inputs the calculated vascular information and the read out patient's treatment information and patient information into the second learning model M2, and generates flush information based on the output information from the second learning model M2 (S26). The control unit 31 outputs the generated flush information to the display device 4, and displays the flush information on a screen such as that shown in FIG. 6 (S27). For example, multiple patterns of flush information are displayed as shown in FIG. 6, and the surgeon selects the flush information to be executed from the displayed flush information. Based on the selected flush information, the surgeon performs a flush operation by, for example, mounting a syringe 70 filled with the selected flush liquid 72 on an autoinjector, setting the flush flow rate and flush amount, etc. The control unit 31 accepts the surgeon's selection of any of the flush information via, for example, the input device 5 (S28), and when the selection is accepted, stores the selected flush information in the medical DB 34a (S29).

The autoinjector performs a flash operation based on the flash conditions set by the surgeon, and then the surgeon uses the intravascular inspection device 1 to capture an OCT image. The control unit 31 acquires the OCT image from the intravascular inspection device 1 (S30), and outputs and displays the acquired OCT image on the display device 4 (S31). The surgeon observes the state of the blood vessel using the displayed OCT image, and performs intravascular treatment as necessary. Note that in intravascular treatment, the surgeon temporarily removes the imaging catheter 10 from the guiding catheter 60, and inserts a treatment catheter such as a balloon catheter into the guiding catheter 60 to perform the procedure.

In addition, when the control unit 31 displays the OCT image taken by the intravascular inspection device 1 on the display device 4, it acquires, for example, information regarding the evaluation of the OCT image from the surgeon via the input device 5 (S32). The evaluation information for the OCT image is, for example, information indicating whether the OCT image is a clear frame that can be used sufficiently for image diagnosis, and may be, for example, information indicating that it is a clear frame (OK) or not a clear frame (NO). The evaluation information for the OCT image may also be the surgeon's satisfaction when using it for image diagnosis (for example, a 5-point scale or each value out of 100 points). The control unit 31 may also determine whether the OCT image has been retaken by the intravascular inspection device 1, and if it has been retaken, give a low evaluation (for example, 1 out of 5) to the flash information at that time, and if it has not been retaken, give a high evaluation (for example, 5 out of 5) to the flash information at that time. Furthermore, the control unit 31 may extract OCT images (clear frames) that have been evaluated by the surgeon as being above a predetermined level, or OCT images (clear frames) that have a lesion area with a predetermined image quality or brightness or higher, calculate the percentage of clear frames, and use the calculated percentage as evaluation information, from among a plurality of OCT images captured by one pullback operation. In addition, the determination of whether an OCT image is a clear frame may be performed using a learning model (neural network) constructed by machine learning. For example, a learning model that is composed of a CNN and has been trained to output information indicating whether an OCT image is a clear image that can be used sufficiently for image diagnosis when an OCT image is input may be used. In this case, the image processing device 3 inputs the OCT image into a trained learning model, and can determine whether the OCT image is a clear frame based on the output information from the learning model. The evaluation information assigned to each flash information in this manner may indicate the success probability of the flash operation, and the success probability may be calculated from such evaluation information. When the control unit 31 acquires evaluation information for the flash information, the control unit 31 stores the acquired evaluation information (probability of success) in association with the flash information in the medical DB 34a. This allows the flash information and the evaluation information for the flash information stored in the medical DB 34a to be used as training data for learning the second learning model M2.

In this embodiment, the above-described process generates conditions (flash information) for the flash operation to be performed when capturing an OCT image based on the IVUS image, the patient's treatment information, and the patient information. This reduces the occurrence of flash failures in which blood cannot be properly removed by the flash operation, improving the likelihood of performing an appropriate flash operation. If an appropriate flash operation can be performed, the need to repeat the flash operation and retake the OCT image is reduced, and an extension of the procedure time can be avoided. In addition, when performing a flash operation using a contrast agent as the flush solution, an increase in the dose of the contrast agent, or more precisely, an increase in the dose of iodine contained in the contrast agent, due to the need to repeat the flash operation can be avoided, minimizing the impact on the patient's renal function and reducing the burden on the renal function.

In the intravascular inspection device 1 using the dual-type imaging catheter 10 as in this embodiment, after intravascular treatment is performed, for example after placing a stent in the lesion, an OCT image is taken to confirm the placement position of the stent. Even in this case, an IVUS image can be taken before taking the OCT image, flash information can be generated from the obtained IVUS image, and the generated flash information can be presented to the surgeon. Even in such a case, the surgeon can select the optimal flash conditions by identifying the content of the flash operation to be performed based on the presented flash information.

FIG. 9 is an explanatory diagram showing a modified example of the learning model. The first learning model M1 and the second learning model M2 described above may be configured as one learning model as shown in FIG. 9. The learning model M3 shown in FIG. 9 has a configuration in which, for example, a plurality of IVUS images taken by one pullback operation, the patient's treatment information, and the patient information are input, and flash information similar to that of the second learning model M2 described above is output. The learning model M3 shown in FIG. 9 has, for example, a CNN, and extracts image features from the input IVUS images by the CNN. The features extracted here correspond to vascular information related to the vascular lumen in the IVUS image. Therefore, the learning model M3 can generate flash information based on the features extracted by the CNN and the input treatment information and patient information. Even when configured as one learning model M3 in this way, flash information can be generated from the IVUS image, the patient's treatment information, and the patient information, and the generated flash information can be presented to the surgeon.

In this embodiment, the process of generating each piece of information for the flash operation from the patient's vascular information, treatment information, and patient information may be configured to be rule-based instead of using the second learning model M2. In this case, for example, a table is prepared in which optimal flash information is associated with each combination of vascular information, treatment information, and patient information, and the image processing device 3 identifies flash information corresponding to the patient's vascular information, treatment information, and patient information from the contents of the table. In this case too, the surgeon can be presented with optimal flash information corresponding to the patient.

In this embodiment, the image processing device 3 locally performs the process of generating flash information based on the IVUS image, the patient's treatment information, and the patient information using the first learning model M1 and the second learning model M2, but this configuration is not limited to this. For example, a server may be provided that performs the process of generating flash information using the first learning model M1 and the second learning model M2. In this case, the image processing device 3 may be configured to transmit the IVUS image, the treatment information, and the patient information to the server and acquire the flash information generated by the server. Note that, if the server here is a server that stores the medical DB 34a, the image processing device 3 can transmit only the IVUS image to the server and acquire the flash information generated by the server. Even with such a configuration, the same process as in this embodiment is possible and the same effect can be obtained.

In this embodiment, the second learning model M2 may be prepared for each treatment situation, such as when emphasizing reduction in the amount of flush liquid (contrast agent) administered or reduction in the flow rate of the flush liquid. The second learning model M2 may also be prepared for each part (e.g., type of blood vessel) to be processed by the flush operation, or for each position of the tip of the guiding catheter 60 relative to the blood vessel. For example, a model for the right coronary artery, a model for the left anterior descending artery, a model for the circumflex artery, a model for when the tip of the guiding catheter 60 is sufficiently engaged with the coronary artery, and a model for when the engagement state is shallow may be prepared. In this case, the image processing device 3 may present the flash information generated using each model to the surgeon. The image processing device 3 may also select a model to be used according to the type of blood vessel to be processed or the setting input by the surgeon, and generate flash information using the selected model to present to the surgeon.

The embodiments disclosed herein are illustrative in all respects and should not be considered limiting. The scope of the present invention is indicated by the claims, not by the meaning described above, and is intended to include all modifications within the scope and meaning equivalent to the claims.

Reference Signs List 1 Intravascular inspection device 2 Fluoroscopic image capture device 3 Image processing device 4 Display device 5 Input device 10 Imaging catheter 11 Diagnostic imaging probe 60 Guiding catheter 61 Sheath 62 Branch hub 70 Syringe 34a Medical treatment DB
M1 First learning model M2 Second learning model

Claims (9)

Calculating information about the lumen of the blood vessel based on an ultrasonic tomographic image obtained by capturing a cross section of the blood vessel using the ultrasonic signal;
determining operation information regarding a flash operation when photographing a cross section of the blood vessel using an optical signal based on the calculated information regarding the lumen, and a success probability of the flash operation based on the operation information;
and outputting the identified operation information and the identified success probability to a display device . The acquired ultrasonic tomographic image is input to a learning model that has been trained to output information indicating a region of a lumen of a blood vessel in the ultrasonic tomographic image when the ultrasonic tomographic image is input, and the region of the lumen of a blood vessel in the ultrasonic tomographic image is output;
The program according to claim 1 , further comprising: calculating information about the lumen based on the output area of the lumen. The information about the lumen includes a cross-sectional diameter of the lumen and a longitudinal length of the lumen;
Obtaining a plurality of ultrasonic tomographic images by photographing a cross section of the blood vessel at a plurality of locations using ultrasonic signals;
The program according to claim 1 or 2, which causes the computer to execute a process of calculating a length of the lumen of the captured blood vessel in a longitudinal direction based on the number of acquired ultrasonic tomographic images. The program according to claim 1 , further comprising: determining the operation information including a flow rate and a flow velocity of a flushing liquid that replaces the blood in the blood vessel during the flushing operation. Inputting the calculated information about the inner cavity into a second learning model that has been trained to output operation information about the flash operation and a success probability for the flash operation based on the operation information when information about the inner cavity is input, and acquiring operation information about the flash operation and a success probability for the flash operation based on the operation information;
The program according to claim 1 , further comprising: outputting the acquired operation information and the success probability to the display device . The second learning model is trained to output operation information regarding the flush operation and a success probability for the flush operation based on the operation information when information regarding the lumen and patient information regarding a patient are input,
Obtaining patient information about the patient;
The program according to claim 5, which causes the computer to execute a process of inputting the calculated information regarding the inner cavity and the acquired patient information into the second learning model , and outputting operation information regarding the flush operation and a success probability for the flush operation based on the operation information . Calculating information about the lumen of the blood vessel based on an ultrasonic tomographic image obtained by capturing a cross section of the blood vessel using the ultrasonic signal;
determining operation information regarding a flash operation when photographing a cross section of the blood vessel using an optical signal based on the calculated information regarding the lumen, and a success probability of the flash operation based on the operation information;
outputting the specified operation information and the specified success probability to a display device . a calculation unit that calculates information about a lumen of a blood vessel based on an ultrasonic tomographic image obtained by photographing a cross section of the blood vessel using an ultrasonic signal;
a determination unit that determines operation information regarding a flash operation when capturing an image of a cross section of the blood vessel using an optical signal based on the calculated information about the lumen, and a success probability of the flash operation based on the operation information;
and an output unit that outputs the specified operation information and the specified success probability to a display device . acquiring training data that associates information about the lumen of a patient's blood vessel, operation information about a flash operation when photographing a cross section of the blood vessel using an optical signal , and a success probability of the flash operation based on the operation information ;
A model generation method in which a computer executes a process of generating a trained learning model that uses the training data to output, when information about the lumen of a blood vessel is input , operation information regarding the flush operation and a success probability for the flush operation based on the operation information .

Images