JP7774259B2 - Information processing device and trained model generation method - Google Patents

Description

The present invention relates to an information processing device and a method for generating a trained model.

A catheter system is used in which an image acquisition catheter is inserted into a tubular organ such as a blood vessel to acquire images (Patent Document 1).

International Publication No. 2017/164071

For example, in areas with complex structures, such as the intracardiac region, it can be difficult to quickly interpret images acquired with an imaging catheter.

In one aspect, the present invention aims to provide an information processing device, etc. that assists in the understanding of images acquired by an image acquisition catheter.

The information processing device includes an image acquisition unit that acquires catheter images obtained by a radial scanning type image acquisition catheter, and a first position information output unit that inputs the acquired catheter image into a medical instrument trained model that outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input, and outputs the first position information , wherein the medical instrument trained model accepts input of the catheter image as an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter is arranged in parallel in order of scanning angle, and includes a plurality of first convolutional layers, and at least one of the plurality of first convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle.

In one aspect, an information processing device or the like can be provided to assist in the understanding of images acquired by an image acquisition catheter.

FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. FIG. 2 is an explanatory diagram illustrating an overview of an image acquisition catheter. FIG. 1 is an explanatory diagram illustrating the configuration of a catheter system. FIG. 1 is an explanatory diagram illustrating an overview of the operation of the catheter system. FIG. 10 is an explanatory diagram schematically showing the operation of the imaging catheter. FIG. 2 is an explanatory diagram schematically showing a catheter image taken by an image acquisition catheter. FIG. 10 is an explanatory diagram illustrating classification data generated based on a catheter image. FIG. 1 is an explanatory diagram illustrating the configuration of a medical instrument trained model. FIG. 2 is an explanatory diagram illustrating the configuration of a classification model. FIG. 10 is an explanatory diagram illustrating an outline of processing related to position information. FIG. 10 is an explanatory diagram illustrating the record layout of a medical instrument position training data DB. 10 is an example of a screen used to create a medical instrument position training data DB. 10 is a flowchart illustrating the processing flow of a program for creating a medical instrument position training data DB. 10 is a flowchart illustrating the processing flow of a medical instrument trained model generation program. 10 is a flowchart illustrating the processing flow of a program for adding data to a medical instrument position training data DB. FIG. 10 is an explanatory diagram illustrating visualization of a medical instrument. FIG. 1 is an explanatory diagram illustrating the configuration of an angle-learned model. FIG. 2 is an explanatory diagram illustrating a location information model. 11 is a flowchart illustrating the flow of processing of a program according to a third embodiment. FIG. 2 is an explanatory diagram illustrating the configuration of a classification model. FIG. 10 is an explanatory diagram illustrating first training data. 10 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. FIG. 13 is an explanatory diagram illustrating the configuration of a classification model according to a fifth embodiment. FIG. 10 is an explanatory diagram illustrating second training data. 13 is a flowchart illustrating the flow of processing of a program according to a fifth embodiment. 10 is a flowchart illustrating the flow of processing in a classification and synthesis subroutine. FIG. 10 is an explanatory diagram illustrating the configuration of a trained model with hints 631. FIG. 10 is an explanatory diagram illustrating the record layout of a hinted model training data DB. 13 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. 10 is a flowchart illustrating a processing flow of a program according to a modified example. FIG. 20 is an explanatory diagram illustrating the configuration of a classification model according to a seventh embodiment. FIG. 10 is an explanatory diagram illustrating the configuration of a classification model according to a modified example. FIG. 20 is an explanatory diagram illustrating an outline of the processing of the eighth embodiment. FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. FIG. 10 is an explanatory diagram illustrating an outline of a process for correcting position information. 13 is a flowchart illustrating the flow of processing of a program according to an eighth embodiment. 10 is a flowchart illustrating a process flow of a three-dimensional display subroutine. FIG. 20 is an explanatory diagram illustrating padding processing in embodiment 9. FIG. 10 is an explanatory diagram illustrating a modified polar padding process. FIG. 20 is an explanatory diagram illustrating the configuration of a catheter system according to a tenth embodiment. FIG. 23 is a functional block diagram of an information processing device according to an eleventh embodiment.

[First Embodiment]
1 is an explanatory diagram illustrating an overview of a catheter system 10. The catheter system 10 of this embodiment is used in IVR (Interventional Radiology), which treats various organs while performing fluoroscopy using an imaging diagnostic device such as an X-ray fluoroscopy device. Medical instruments for treatment can be accurately operated by referring to images acquired by the catheter system 10 placed near the area to be treated.

The catheter system 10 comprises an image acquisition catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The image acquisition catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, mouse, trackball, or microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may be configured as an integrated unit.

Figure 2 is an explanatory diagram illustrating an overview of the imaging catheter 40. The imaging catheter 40 has a probe portion 41 and a connector portion 45 located at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the imaging catheter 40 farthest from the connector portion 45 will be referred to as the tip side.

A shaft 43 is inserted inside the probe section 41. A sensor 42 is connected to the tip of the shaft 43. A guidewire lumen 46 is provided at the tip of the probe section 41. The user inserts a guidewire to a position beyond the target site, and then inserts the guidewire into the guidewire lumen 46 to guide the sensor 42 to the target site. A ring-shaped tip marker 44 is fixed near the tip of the probe section 41.

Sensor 42 may be, for example, an ultrasonic transducer that transmits and receives ultrasonic waves, or an OCT (Optical Coherence Tomography) transceiver that emits near-infrared light and receives reflected light. In the following explanation, the image acquisition catheter 40 will be described as an IVUS (Intravascular Ultrasound) catheter used to capture ultrasound cross-sectional images from inside the circulatory system.

Figure 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 has an information processing device 20, an MDU 33, and an image acquisition catheter 40. The information processing device 20 has a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more CPUs (Central Processing Units), GPUs (Graphics Processing Units), TPUs (Tensor Processing Units), multi-core CPUs, etc. The control unit 21 is connected to each hardware component that constitutes the information processing device 20 via a bus.

The main memory device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs currently being executed by the control unit 21.

The auxiliary storage device 23 is a storage device such as SRAM, flash memory, a hard disk, or a magnetic tape. The auxiliary storage device 23 stores the medical instrument trained model 611, the classification model 62, the program to be executed by the control unit 21, and various data required for executing the program. The communication unit 24 is an interface that communicates between the information processing device 20 and the network.

The display unit 25 is an interface that connects the display device 31 to the bus. The input unit 26 is an interface that connects the input device 32 to the bus. The catheter control unit 271 controls the MDU 33, controls the sensor 42, and generates images based on signals received from the sensor 42.

The MDU 33 rotates the sensor 42 and shaft 43 inside the probe section 41. The catheter control section 271 generates one catheter image 51 (see Figure 4) for each rotation of the sensor 42. The generated catheter image 51 is a cross-sectional image centered on the probe section 41 and approximately perpendicular to the probe section 41.

The MDU 33 can also move the sensor 42 back and forth while rotating the sensor 42 and shaft 43 inside the probe section 41. By rotating the sensor 42 while pulling or pushing it, the catheter control unit 271 continuously generates multiple catheter images 51 that are approximately perpendicular to the probe section 41. The continuously generated catheter images 51 can be used to construct a three-dimensional image. Therefore, the image acquisition catheter 40 functions as a three-dimensional scanning catheter that sequentially acquires multiple catheter images 51 along the longitudinal direction.

The operation of advancing and retracting the sensor 42 includes both the operation of advancing and retracting the entire probe unit 41 and the operation of advancing and retracting the sensor 42 inside the probe unit 41. The operation of advancing and retracting may be performed automatically at a predetermined speed by the MDU 33 or manually by the user.

Note that the imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and moves back and forth. It may also be an electronic radial scanning type imaging catheter 40 that uses a sensor 42 with multiple ultrasonic transducers arranged in a ring shape.

Using the image acquisition catheter 40, catheter images 51 can be taken that include not only biological tissues that make up the circulatory system, such as the heart wall and blood vessel walls, but also reflectors present inside the circulatory system, such as red blood cells, and organs present outside the circulatory system, such as the respiratory and digestive systems.

In this embodiment, an example will be described in which the imaging catheter 40 is used for atrial septal puncture. In atrial septal puncture, the imaging catheter 40 is inserted into the right atrium, and then the Brockenbrough needle is inserted into the fossa ovalis, a thin-walled portion of the atrial septum, under ultrasound guidance. The tip of the Brockenbrough needle reaches the inside of the left atrium.

When performing atrial septal puncture, the catheter image 51 depicts the Brockenbrough needle in addition to the biological tissues that make up the circulatory system, such as the atrial septum, right atrium, left atrium, and aorta, as well as reflectors such as red blood cells contained in the blood flowing within the circulatory system. A doctor or other user can safely perform atrial septal puncture by using the catheter image 51 to confirm the positional relationship between the fossa ovalis and the tip of the Brockenbrough needle. The Brockenbrough needle is an example of a medical instrument in this embodiment.

Note that the use of the catheter system 10 is not limited to atrial septal puncture. For example, the catheter system 10 can be used for procedures such as transcatheter myocardial ablation, transcatheter valve replacement, and stent placement in coronary arteries. The area treated using the catheter system 10 is not limited to the area around the heart. For example, the catheter system 10 can be used to treat various areas, such as the pancreatic duct, bile duct, and blood vessels in the lower limbs.

The functions and configuration of the catheter control unit 271 are similar to those of conventional ultrasound diagnostic devices, so detailed explanations will be omitted. The control unit 21 may also perform the functions of the catheter control unit 271.

The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via an HIS (Hospital Information System) or the like.

In this embodiment, the information processing device 20 is a dedicated ultrasound diagnostic device, or a personal computer, tablet, smartphone, or the like that has the functionality of an ultrasound diagnostic device. The following description will be given taking as an example a case where the information processing device 20 is also used for training trained models such as the medical instrument trained model 611 and for creating training data. A computer or server other than the information processing device 20 may be used for training trained models and creating training data.

The following explanation will mainly focus on the case where the control unit 21 performs software processing. The processing explained using flowcharts and the various trained models may each be implemented using dedicated hardware.

Figure 4 is an explanatory diagram outlining the operation of the catheter system 10. Figure 4 illustrates an example in which multiple catheter images 51 are captured while the sensor 42 is pulled at a predetermined speed, and the images are displayed in real time.

The control unit 21 captures one catheter image 51 (step S501). The control unit 21 acquires position information of the medical instrument depicted in the catheter image 51 (step S502). In Figure 4, the position of the medical instrument in the catheter image 51 is indicated by an "x" mark.

The control unit 21 associates the catheter image 51, the position of the catheter image 51 in the longitudinal direction of the image acquisition catheter 40, and the position information of the medical instrument, and records them in the auxiliary storage device 23 or a large-capacity storage device connected to the HIS (step S503).

The control unit 21 generates classification data 52 by classifying each part of the catheter image 51 by the depicted subject (step S504). In Figure 4, the classification data 52 is shown as a schematic diagram in which the catheter image 51 is colored differently based on the classification results.

The control unit 21 determines whether the user has specified two-dimensional display or three-dimensional display (step S505). If it is determined that the user has specified two-dimensional display (2D in step S505), the control unit 21 displays the catheter image 51 and classification data 52 in two-dimensional display on the display device 31 (step S506).

Note that in step S505 of Figure 4, "2D/3D" is written as if it were a selection between "two-dimensional display" and "three-dimensional display." However, if the user selects "3D," the control unit 21 may display both "two-dimensional display" and "three-dimensional display."

If it is determined that the user has specified three-dimensional display (3D in step S505), the control unit 21 determines whether the position information of the medical instrument recorded sequentially in step S503 is normal (step S511). If it is determined that the position information is not normal (NO in step S511), the control unit 21 corrects the position information (step S512). Details of the processing performed in steps S511 and S512 will be described later.

If it is determined to be normal (YES in step S511), or after step S512 is completed, the control unit 21 performs a three-dimensional display illustrating the structure of the area being observed and the position of the medical instrument (step S513). As mentioned above, the control unit 21 may display both a three-dimensional display and a two-dimensional display on a single screen.

After step S506 or step S513 is completed, the control unit 21 determines whether acquisition of the catheter image 51 has been completed (step S507). For example, if an end instruction is received from the user, the control unit 21 determines that processing is to be terminated.

If it is determined that the processing should not be terminated (NO in step S507), the control unit 21 returns to step S501. If it is determined that the processing should be terminated (YES in step S507), the control unit 21 terminates the processing.

Figure 4 illustrates the processing flow for performing two-dimensional display (step S506) or three-dimensional display (step S513) in real time while capturing a series of catheter images 51. The control unit 21 may also perform two-dimensional display or three-dimensional display in non-real time based on the data recorded in step S503.

Figure 5A is an explanatory diagram that schematically shows the operation of the image acquisition catheter 40. Figure 5B is an explanatory diagram that schematically shows a catheter image 51 captured by the image acquisition catheter 40. Figure 5C is an explanatory diagram that schematically explains classification data 52 generated based on the catheter image 51. Figures 5A to 5C will be used to explain the RT (Radius-Theta) format and the XY format.

As described above, the sensor 42 transmits and receives ultrasound while rotating inside the imaging catheter 40. The catheter control unit 271 acquires radial scanning line data centered on the imaging catheter 40, as schematically shown by the eight arrows in Figure 5A.

The catheter control unit 271 can generate the catheter image 51 shown in Figure 5B in two formats: an RT format catheter image 518 and an XY format catheter image 519, based on the scan line data. The RT format catheter image 518 is an image generated by arranging the respective scan line data in parallel with each other. The horizontal direction of the RT format catheter image 518 indicates the distance from the image acquisition catheter 40.

The vertical direction of the RT catheter image 518 indicates the scan angle. One RT catheter image 518 is formed by arranging the scan line data acquired by the sensor 42 rotating 360 degrees in parallel in order of the scan angle.

In Figure 5B, the left side of the RT catheter image 518 shows a location closer to the image acquisition catheter 40, and the right side of the RT catheter image 518 shows a location farther from the image acquisition catheter 40.

The XY-format catheter image 519 is an image generated by radially arranging and interpolating each scan line data. The XY-format catheter image 519 shows a tomographic image of the subject cut perpendicular to the image acquisition catheter 40 at the position of the sensor 42.

Figure 5C shows a schematic representation of classification data 52, which classifies each part of a catheter image 51 by the depicted subject. Classification data 52 can also be displayed in two formats: RT format classification data 528 and XY format classification data 529. Image conversion methods between RT format and XY format are well known, so explanations are omitted.

In Figure 5C, thick hatching sloping downward to the right indicates biological tissue regions that form cavities into which the imaging catheter 40 is inserted, such as the atrial wall and ventricular wall. Thin hatching sloping downward to the left indicates the inside of the first cavity, which is the blood flow region into which the tip of the imaging catheter 40 is inserted. Thin hatching sloping downward to the right indicates the inside of the second cavity, which is a blood flow region other than the first cavity.

When performing an atrial septal puncture from the right atrium to the left atrium, the first chamber is the right atrium, and the second chamber is the left atrium, right ventricle, left ventricle, aorta, coronary artery, etc. In the following description, the interior of the first chamber will be referred to as the first lumen region, and the interior of the second chamber will be referred to as the second lumen region.

Bold hatching sloping downward to the left indicates a non-lamber region of the non-biological tissue area that is neither the first lumen region nor the second lumen region. The non-lamber region includes extracardiac regions and regions outside the cardiac structure. If the imaging range of the imaging catheter 40 is small and the distal wall of the left atrium cannot be adequately imaged, the interior of the left atrium is also included in the non-lamber region. Similarly, if the distal walls of the lumens of the left ventricle, pulmonary artery, pulmonary vein, aortic arch, etc. cannot be adequately imaged, they are also included in the non-lamber region.

The black areas indicate medical device areas where medical devices such as Brockenbrough needles are depicted. In the following description, the biological tissue area and non-biological tissue area may be collectively referred to as the biological tissue-related area.

Note that the medical instrument is not necessarily inserted into the same first cavity as the imaging catheter 40. Depending on the procedure, the medical instrument may be inserted into the second cavity.

The hatching and black areas shown in Figure 5C are examples of ways in which each region can be distinguished. Each region is displayed on the display device 31 using, for example, a different color. The control unit 21 realizes the function of a first mode output unit that outputs the first lumen region, the second lumen region, and the biological tissue region in a manner in which they can be distinguished from one another. The control unit 21 also realizes the function of a second mode output unit that outputs the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region in a manner in which they can be distinguished from one another.

XY format display is suitable during IVR procedures, such as when confirming the position of the Brockenbrough needle for atrial septal puncture. However, with XY display, information near the imaging catheter 40 is compressed, reducing the amount of data, and data that does not actually exist is added by interpolation at positions away from the imaging catheter 40. Therefore, when analyzing catheter images 51, using RT format images can produce more accurate results than using XY format images.

In the following description, the control unit 21 generates RT format classification data 528 based on the RT format catheter image 518. The control unit 21 converts the XY format catheter image 519 to generate the RT format catheter image 518, and converts the RT format classification data 528 to generate the XY format classification data 529.

Classification data 52 will be explained using specific examples. Pixels classified as "biological tissue region" are recorded with a "biological tissue region label," pixels classified as "first lumen region" with a "first lumen region label," pixels classified as "second lumen region" with a "second lumen region label," pixels classified as "non-lumen region" with a "non-lumen region label," pixels classified as "medical device region" with a "medical device region label," and pixels classified as "non-biological tissue region" with a "non-biological tissue region label." Each label is represented, for example, by an integer.

In addition, the control unit 21 may generate XY format classification data 529 based on the XY format catheter image 519. The control unit 21 may generate RT format classification data 528 based on the XY format classification data 529.

Figure 6 is an explanatory diagram illustrating the configuration of the medical instrument trained model 611. The medical instrument trained model 611 is a model that accepts a catheter image 51 and outputs first position information regarding the position where the medical instrument is depicted. The medical instrument trained model 611 realizes step S502 described using Figure 4. The output layer of the medical instrument trained model 611 performs the function of a first position information output unit that outputs the first position information.

In Figure 6, the input to the medical instrument trained model 611 is the RT format catheter image 518. The first position information is the probability that a medical instrument is depicted for each part on the RT format catheter image 518. In Figure 6, areas where there is a high probability that a medical instrument is depicted are shown with dark hatching, and areas where there is a low probability that a medical instrument is depicted are shown without hatching.

The medical equipment trained model 611 is generated by machine learning using, for example, a neural network structure such as a CNN (Convolutional Neural Network). Examples of CNNs that can be used to generate the medical equipment trained model 611 include R-CNN (Region Based Convolutional Neural Network), YOLO (You only look once), U-Net, and GAN (Generative Adversarial Network). The medical equipment trained model 611 may also be generated using a neural network structure other than a CNN.

The medical instrument trained model 611 may be a model that accepts multiple catheter images 51 acquired in time series and outputs first position information for the most recent catheter image 51. The medical instrument trained model 611 can be generated by combining a model that accepts time series input, such as an RNN (Recurrent Neural Network), with the aforementioned neural network structure.

The RNN is, for example, a long short-term memory (LSTM). When an LSTM is used, the medical instrument trained model 611 includes a memory unit that stores information about previously input catheter images 51. The medical instrument trained model 611 outputs the first position information based on the information stored in the memory unit and the latest catheter image 51.

When using multiple catheter images 51 acquired in chronological order, the medical instrument trained model 611 may include a recursive input unit that inputs output based on previously input catheter images 51 along with the next catheter image 51. The medical instrument trained model 611 outputs first position information based on the latest catheter image 51 and input from the recursive input unit. By using catheter images 51 acquired in chronological order, it is possible to realize a medical instrument trained model 611 that is less susceptible to the influence of image noise, etc., and outputs first position information with high accuracy.

The medical instrument trained model 611 may use the position of a single pixel on the catheter image 51 that has been received as input to output a location where there is a high probability that a medical instrument is depicted. For example, the medical instrument trained model 611 may be a model that calculates the probability that a medical instrument is depicted for each part on the catheter image 51, as shown in FIG. 6, and then outputs the position of the pixel with the highest probability. The medical instrument trained model 611 may output the position of the center of gravity of an area where the probability that a medical instrument is depicted exceeds a predetermined threshold. The medical instrument trained model 611 may output an area where the probability that a medical instrument is depicted exceeds a predetermined threshold.

Note that multiple medical instruments may be used simultaneously. When multiple medical instruments are depicted in the catheter image 51, it is desirable that the medical instrument trained model 611 be a model that outputs the first position information for each of the multiple medical instruments.

The medical instrument trained model 611 may be a model that outputs only the first position information of one medical instrument. The control unit 21 can input the RT format catheter image 518, in which the periphery of the first position information output from the medical instrument trained model 611 is masked, into the medical instrument trained model 611 to obtain the first position information of the second medical instrument. By repeating the same process, the control unit 21 can also obtain the first position information of the third and subsequent medical instruments.

Figure 7 is an explanatory diagram illustrating the configuration of the classification model 62. The classification model 62 is a model that accepts a catheter image 51 and outputs classification data 52 that classifies each part that makes up the catheter image 51 by the depicted subject. The classification model 62 implements step S504 described using Figure 4.

A specific example will be given. The classification model 62 classifies each pixel constituting the input RT-format catheter image 518 into, for example, a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," and a "medical device region," and outputs RT-format classification data 528 that associates the pixel position with a label indicating the classification result.

The classification model 62 may divide the catheter image 51 into regions of any size, such as 3 pixels vertically and 3 pixels horizontally, for a total of 9 pixels, and output classification data 52 that classifies each region. The classification model 62 is, for example, a trained model that performs semantic segmentation on the catheter image 51. Specific examples of the classification model 62 will be described later.

Figure 8 is an explanatory diagram outlining the processing related to position information. Multiple catheter images 51 are captured while the sensor 42 is moved in the longitudinal direction of the image acquisition catheter 40. In Figure 8, the line drawing of an approximately truncated cone schematically shows a biological tissue region constructed in three dimensions based on the multiple catheter images 51. The interior of the approximately truncated cone represents the first lumen region.

The white and black circles indicate the position of the medical device obtained from each catheter image 51. Of these, the black circle is determined to be a false detection because it is located far away from the white circle. The shape of the medical device can be reproduced by the thick line smoothly connecting the white circles. The cross marks indicate complementary information that complements the position information of medical devices that could not be detected.

Details of the processing described using Figure 8 will be described in embodiment 8. The processing described using Figure 8 realizes the processing of steps S511 and S512 described using Figure 4.

For example, when a medical instrument is in contact with a biological tissue region, it is known that even when a skilled doctor, technician, or other user reads a single still catheter image 51, it can be difficult to identify where the medical instrument is located. However, when observing a moving catheter image 51, the user can determine the position of the medical instrument relatively easily. This is because the user reads the image while expecting the medical instrument to be in the same position as in the previous frame.

In the process described using Figure 8, the medical device is reconstructed without any inconsistencies using the position information of the medical device obtained from each of the multiple catheter images 51. By performing this process, a catheter system 10 can be realized that accurately determines the position of the medical device, just as if the user were observing a video, and displays the shape of the medical device in a three-dimensional image.

According to this embodiment, a catheter system 10 can be provided that, by displaying steps S506 and S513, assists in understanding the catheter image 51 acquired using the image acquisition catheter 40. By using the catheter system 10 of this embodiment, the user can accurately grasp the position of the medical instrument, allowing IVR to be performed safely.

[Embodiment 2]
This embodiment relates to a method for generating a medical instrument trained model 611. Explanation of parts common to Embodiment 1 will be omitted. Note that this embodiment will be described taking as an example a case where the medical instrument trained model 611 is generated using the information processing device 20 described using FIG. 3 .

The medical instrument trained model 611 may be created using a computer or the like separate from the information processing device 20. The medical instrument trained model 611 for which machine learning has been completed may be copied to the auxiliary storage device 23 via a network. A medical instrument trained model 611 trained on a single piece of hardware can be used by multiple information processing devices 20.

Figure 9 is an explanatory diagram illustrating the record layout of the medical instrument position training data DB (Database) 71. The medical instrument position training data DB 71 is a database that records catheter images 51 and medical instrument position information in association with each other, and is used to train the medical instrument learned model 611 using machine learning.

The medical instrument position training data DB71 has a catheter image field and a position information field. The catheter image field stores catheter images 51, such as RT format catheter images 518. The catheter image field may store so-called sound ray data indicating ultrasound signals received by the sensor 42. The catheter image field may store scan line data generated based on the sound ray data.

The position information field records the position information of the medical device depicted in the catheter image 51. The position information is, for example, information indicating the position of a single pixel marked by a labeler on the catheter image 51, as described below. The position information may also be information indicating a circular area centered near a point marked by the labeler on the catheter image 51. The circle has dimensions that do not exceed the size of the medical device depicted in the catheter image 51. The circle is, for example, large enough to inscribe a square of 50 pixels or less in length and width.

Figure 10 is an example of a screen used to create the medical instrument position training data DB71. The screen in Figure 10 displays a set of catheter images 51: an RT format catheter image 518 and an XY format catheter image 519. The RT format catheter image 518 and the XY format catheter image 519 are images created based on the same sound ray data.

A control button area 782 is displayed below the catheter image 51. At the top of the control button area 782, there are the frame number of the catheter image 51 currently being displayed, and a jump button that the user can use to input an arbitrary frame number to jump the display.

Below the frame numbers, etc., are various buttons that the user can use to fast forward, rewind, advance frame by frame, etc. These buttons are similar to those commonly used in various image playback devices, so we will not explain them here.

In this embodiment, the user is the person in charge of creating training data by viewing pre-recorded catheter images 51 and labeling the positions of medical instruments. In the following description, the person in charge of creating training data will be referred to as a labeler. A labeler is a doctor or technician skilled in interpreting catheter images 51, or a person who has been trained to perform accurate labeling. Furthermore, in the following description, the work of the labeler adding marks or the like to catheter images 51 in order to label them may be referred to as marking.

The labeler observes the displayed catheter image 51 and determines the position where the medical instrument is depicted. Generally, the area where the medical instrument is depicted is very small compared to the overall area of the catheter image 51. The labeler moves the cursor 781 to approximately the center of the area where the medical instrument is depicted and marks it by a click operation or the like. If the display device 31 is a touch panel, the labeler may mark it by tapping with a finger or a stylus pen or the like. The labeler may also mark it by a so-called flick operation.

The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 may display the mark at the corresponding position on the other catheter image 51.

The control unit 21 creates a new record in the medical instrument position training data DB 71 and records the catheter image 51 in association with the position marked by the labeler. The control unit 21 displays the next catheter image 51 on the display device 31. By repeating the above process multiple times, the medical instrument position training data DB 71 is created.

In other words, the labeler can sequentially mark multiple catheter images 51 by simply performing a click operation or the like on the catheter image 51 without operating each button in the control button area 782. The labeler only needs to perform a single click operation or the like on one catheter image 51 depicting one medical instrument.

As mentioned above, multiple medical instruments may be depicted on the catheter image 51. The labeler can mark each medical instrument with a single click. The following explanation will use as an example a case where one medical instrument is depicted on one catheter image 51.

Figure 11 is a flowchart explaining the processing flow of a program for creating a medical instrument position training data DB71. An example will be explained in which the medical instrument position training data DB71 is created using an information processing device 20. The program in Figure 11 may be executed by hardware separate from the information processing device 20.

Prior to execution of the program in Figure 11, a large number of catheter images 51 are recorded in the auxiliary storage device 23 or an external large-capacity storage device. The following explanation will be given using as an example a case in which the catheter images 51 are recorded in the auxiliary storage device 23 in the form of video data including a plurality of RT format catheter images 518 taken in time series.

The control unit 21 acquires one frame of the RT format catheter image 518 from the auxiliary storage device 23 (step S671). The control unit 21 converts the RT format catheter image 518 to generate an XY format catheter image 519 (step S672). The control unit 21 displays the screen described using Figure 10 on the display device 31 (step S673).

The control unit 21 accepts the input operation of the position information by the labeler via the input device 32 (step S674). Specifically, the input operation is a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

The control unit 21 displays a mark such as a small circle at the position where the input operation was accepted (step S675). Accepting input operations for the image displayed on the display device 31 via the input device 32 and displaying marks on the display device 31 are user interfaces that have been used conventionally, and therefore detailed explanations are omitted.

The control unit 21 determines whether the image for which the input operation was accepted in step S674 is an RT-format catheter image 518 (step S676). If it is determined that the image is an RT-format catheter image 518 (YES in step S676), the control unit 21 also displays a mark at the corresponding position on the XY-format catheter image 519 (step S677). If it is determined that the image is not an RT-format catheter image 518 (NO in step S676), the control unit 21 also displays a mark at the corresponding position on the RT-format catheter image 518 (step S678).

The control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 associates the catheter image 51 with the position information input by the labeler and records it in the medical instrument position training data DB 71 (step S679).

The catheter image 51 recorded in step S679 may be only the RT format catheter image 518 acquired in step S671, or both the RT format catheter image 518 and the XY format catheter image 519 generated in step S672. The catheter image 51 recorded in step S679 may be sound ray data for one rotation received by the sensor 42, or scanning line data generated by signal processing the sound ray data.

The position information recorded in step S679 is, for example, information indicating the position of a single pixel on the RT format catheter image 518 corresponding to the position where the labeler performed a click operation or the like using the input device 32. The position information may also be information indicating the position where the labeler performed a click operation or the like and the surrounding area.

The control unit 21 determines whether to terminate the processing (step S680). For example, when the processing of the catheter image 51 recorded in the auxiliary storage device 23 has been completed, the control unit 21 determines to terminate the processing. When it is determined to terminate (YES in step S680), the control unit 21 terminates the processing.

If it is determined that the processing should not be terminated (NO in step S680), the control unit 21 returns to step S671. In step S671, the control unit 21 acquires the next RT format catheter image 518 and executes the processing from step S672 onwards. In other words, the control unit 21 automatically acquires and displays the next RT format catheter image 518 without waiting for an operation on a button displayed in the control button area 782.

By looping from step S671 to step S680, the control unit 21 records training data based on a large number of RT format catheter images 518 recorded in the auxiliary memory device 23 in the medical instrument position training data DB71.

The control unit 21 may display, for example, a "Save button" on the screen described using Figure 10, and execute step S679 when it accepts selection of the "Save button." Furthermore, the control unit 21 may display, for example, an "AUTO button" on the screen described using Figure 10, and automatically execute step S679 while it is accepting selection of the "AUTO button" without waiting for selection of the "Save button."

In the following explanation, we will use as an example a case where the catheter image 51 recorded in the medical instrument position training data DB71 in step S679 is an RT format catheter image 518, and the position information is the position of a single pixel on the RT format catheter image 518.

Figure 12 is a flowchart explaining the processing flow of the medical equipment trained model 611 generation program. Prior to execution of the program in Figure 12, an untrained model is prepared, for example, by combining a convolutional layer, a pooling layer, and a fully connected layer. As mentioned above, the untrained model is, for example, a CNN model. Examples of CNNs that can be used to generate the medical equipment trained model 611 include R-CNN, YOLO, U-Net, and GAN. The medical equipment trained model 611 may also be generated using a neural network structure other than a CNN.

The control unit 21 obtains a training record to be used for training one epoch from the medical instrument position training data DB 71 (step S571). As described above, the training record recorded in the medical instrument position training data DB 71 is a combination of an RT format catheter image 518 and coordinates indicating the position of the medical instrument depicted in the RT format catheter image 518.

The control unit 21 adjusts the model parameters so that when an RT format catheter image 518 is input to the model's input layer, the output layer outputs the pixel position corresponding to the position information (step S572). When obtaining training records and adjusting the model's parameters, the program may appropriately have functions that cause the control unit 21 to accept user corrections, present the basis for judgment, perform additional learning, etc.

The control unit 21 determines whether to terminate the processing (step S573). For example, the control unit 21 determines to terminate the processing when learning for a predetermined number of epochs has been completed. The control unit 21 may obtain test data from the medical instrument position training data DB71, input it into the model under machine learning, and determine to terminate the processing when an output of a predetermined accuracy is obtained.

If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the learned parameters of the medical instrument position training data DB71 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, a medical instrument learned model 611 is generated that accepts the catheter image 51 and outputs the first position information.

Prior to executing the program of FIG. 12, a model that accepts time-series input, such as an RNN, may be prepared. The RNN is, for example, an LSTM. In step S572, the control unit 21 adjusts the parameters of the model so that when multiple RT-format catheter images 518 taken in time series are input to the input layer of the model, the output layer outputs the position of the pixel corresponding to the position information associated with the last RT-format catheter image 518 in time series.

Figure 13 is a flowchart explaining the processing flow of a program that adds data to the medical instrument position training data DB 71. The program in Figure 13 is a program that adds training data to the medical instrument position training data DB 71 after creating a medical instrument trained model 611. The added training data is used for additional training of the medical instrument trained model 611.

Prior to execution of the program in Figure 13, a large number of catheter images 51 that have not yet been used to create the medical instrument position training data DB71 are recorded in the auxiliary storage device 23 or an external large-capacity storage device. The following explanation will be given using as an example a case in which the catheter images 51 are recorded in the auxiliary storage device 23 in the form of video data including multiple RT format catheter images 518 taken in chronological order.

The control unit 21 acquires one frame of an RT format catheter image 518 from the auxiliary storage device 23 (step S701). The control unit 21 inputs the RT format catheter image 518 into the medical instrument trained model 611 to acquire the first position information (step S702).

The control unit 21 converts the RT format catheter image 518 to generate an XY format catheter image 519 (step S703). The control unit 21 displays the screen described using Figure 10 on the display device 31 with a mark indicating the first position information acquired in step S702 superimposed on each of the RT format catheter image 518 and the XY format catheter image 519 (step S704).

If the labeler determines that the position of the automatically displayed mark is inappropriate, they can input the correct position of the medical device by performing a single click operation. In other words, the labeler inputs correction instructions for the automatically displayed mark.

The control unit 21 determines whether an input operation by the labeler via the input device 32 has been received within a predetermined time (step S705). It is desirable that the predetermined time be set appropriately by the labeler. Specifically, the input operation is a click operation or a tap operation on the RT format catheter image 518 or the XY format catheter image 519.

If it is determined that an input operation has been received (YES in step S705), the control unit 21 displays a mark such as a small circle at the position where the input operation was received (step S706). It is desirable that the mark displayed in step S706 be a different color or shape from the mark indicating the location information obtained in step S702. Note that the control unit 21 may also erase the mark indicating the location information obtained in step S702.

The control unit 21 determines whether the image for which the input operation was accepted in step S705 is an RT-format catheter image 518 (step S707). If it is determined that the image is an RT-format catheter image 518 (YES in step S707), the control unit 21 also displays a mark at the corresponding position on the XY-format catheter image 519 (step S708). If it is determined that the image is not an RT-format catheter image 518 (NO in step S707), the control unit 21 also displays a mark at the corresponding position on the RT-format catheter image 518 (step S709).

The control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 records the correction data that associates the catheter image 51 with the position information input by the labeler in the medical instrument position training data DB 71 (step S710).

If it is determined that no input operation has been received (NO in step S705), the control unit 21 creates a new record in the medical instrument position training data DB 71. The control unit 21 records uncorrected data associating the catheter image 51 with the first position information acquired in step S532 in the medical instrument position training data DB 71 (step S711).

After step S710 or step S711 is completed, the control unit 21 determines whether to terminate the processing (step S712). For example, when the processing of the catheter image 51 recorded in the auxiliary storage device 23 is completed, the control unit 21 determines that the processing is to be terminated. When it is determined that the processing is to be terminated (YES in step S712), the control unit 21 terminates the processing.

If it is determined that the processing should not be terminated (NO in step S712), the control unit 21 returns to step S701. In step S701, the control unit 21 acquires the next RT format catheter image 518 and executes the processing from step S702 onwards. By looping from step S701 to step S712, the control unit 21 adds training data based on the numerous RT format catheter images 518 recorded in the auxiliary storage device 23 to the medical instrument position training data DB 71.

The control unit 21 may display, for example, an "OK button" on the screen described using Figure 10 to approve the output by the medical instrument trained model 611. If the selection of the "OK button" is received, the control unit 21 determines in step S705 that an instruction of "NO" has been received, and executes step S711.

According to this embodiment, the labeler can mark a single medical instrument depicted in the catheter image 51 with just one operation, such as a single click or tap. The control unit 21 may also accept an operation to mark a single medical instrument by a so-called double-click or double-tap operation. This significantly reduces the labor required for marking compared to marking the boundary line of a medical instrument, thereby reducing the burden on the labeler. According to this embodiment, a large amount of training data can be created in a short amount of time.

According to this embodiment, when multiple medical instruments are depicted in the catheter image 51, the labeler can mark each medical instrument with a single click operation, etc.

In addition, the control unit 21 may display, for example, an "OK button" on the screen described using Figure 10, and execute step S679 when it receives selection of the "OK button."

In this embodiment, by superimposing the position information obtained using the medical instrument trained model 611 on the catheter image 51, additional training data can be quickly created while reducing the burden on the labeler.

[Modification 2-1]
The medical instrument position training data DB 71 may have a field for recording the type of medical instrument. In this case, the control unit 21 accepts input of the type of medical instrument, such as "Brockenbrough needle,""guidewire," or "balloon catheter," on the screen described with reference to FIG. 10 .

By performing machine learning using the medical instrument position training data DB71 created in this way, a medical instrument trained model 611 is generated that outputs the type of medical instrument in addition to the position of the medical instrument.

[Third embodiment]
This embodiment relates to a catheter system 10 that uses two trained models to acquire second position information regarding the position of a medical instrument from a catheter image 51. Description of parts common to the second embodiment will be omitted.

Figure 14 is an explanatory diagram illustrating the depiction of medical instruments. In Figure 14, the medical instruments depicted in the RT format catheter image 518 and the XY format catheter image 519 are highlighted.

Generally, medical devices reflect ultrasound more strongly than biological tissue. Ultrasound emitted from the sensor 42 has difficulty reaching farther than the medical device. Therefore, the medical device is depicted by a hyperechoic region indicating the side closer to the imaging catheter 40 and a hypoechoic region following behind it. The hypoechoic region following behind the medical device is referred to as an acoustic shadow. In Figure 14, the acoustic shadow is indicated by vertical hatching.

In the RT catheter image 518, the acoustic shadow is depicted as a horizontal line. In the XY catheter image 519, the acoustic shadow is depicted as a fan shape. In either case, a high-brightness area is depicted in a region closer to the image acquisition catheter 40 than the acoustic shadow. Note that the high-brightness area may also be depicted in the form of a so-called multiple echo, which repeats regularly along the scan line direction.

Based on the scan angle direction of the RT catheter image 518, i.e., the lateral characteristics in Figure 14, the scan angle at which the medical device is depicted can be determined.

Figure 15 is an explanatory diagram explaining the configuration of the angle-trained model 612. The angle-trained model 612 is a model that accepts a catheter image 51 and outputs scanning angle information regarding the scanning angle at which the medical instrument is depicted.

Figure 15 shows a schematic diagram of an angle-trained model 612 that accepts an RT-format catheter image 518 and outputs scanning angle information indicating the probability that a medical instrument is depicted at each scanning angle, i.e., the vertical direction of the RT-format catheter image 518. Because medical instruments are depicted across multiple scanning angles, the total probability of outputting scanning angle information exceeds 100%. The angle-trained model 612 may extract and output angles at which the medical instrument is highly likely to be depicted.

The angle-trained model 612 is generated by machine learning. By extracting the scanning angle from the position information field of the medical instrument position training data DB71 described using Figure 9, it can be used as training data for generating the angle-trained model 612.

The flowchart in Figure 12 will be used to provide an overview of the process for generating the angle-trained model 612. Prior to executing the program in Figure 12, an untrained model, such as a CNN that combines convolutional layers, pooling layers, and fully connected layers, is prepared. The program in Figure 12 adjusts each parameter of the prepared model and performs machine learning.

The control unit 21 obtains a training record to be used for training one epoch from the medical instrument position training data DB 71 (step S571). As described above, the training record recorded in the medical instrument position training data DB 71 is a combination of an RT format catheter image 518 and coordinates indicating the position of the medical instrument depicted in the RT format catheter image 518.

The control unit 21 adjusts the model parameters so that when an RT format catheter image 518 is input to the model's input layer, a scanning angle corresponding to the position information is output from the output layer (step S572). When obtaining training records and adjusting the model's parameters, the program may appropriately have functions that cause the control unit 21 to accept user corrections, present the basis for judgment, perform additional learning, etc.

The control unit 21 determines whether to terminate the processing (step S573). For example, the control unit 21 determines to terminate the processing when learning for a predetermined number of epochs has been completed. The control unit 21 may obtain test data from the medical instrument position training data DB71, input it into the model under machine learning, and determine to terminate the processing when an output of a predetermined accuracy is obtained.

If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the learned parameters of the medical instrument position training data DB71 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, an angle-learned model 612 is generated that accepts the catheter image 51 and outputs information related to the scanning angle.

Prior to executing the program of FIG. 12, a model that accepts time-series input, such as an RNN, may be prepared. The RNN is, for example, an LSTM. In step S572, the control unit 21 adjusts the parameters of the model so that when multiple RT format catheter images 518 taken in time series are input to the input layer of the model, information regarding the scanning angle associated with the last RT format catheter image 518 in time series is output from the output layer.

Instead of using the angle learned model 612, the control unit 21 may determine the scanning angle at which the medical instrument is depicted by pattern matching.

Figure 16 is an explanatory diagram explaining the position information model 619. The position information model 619 is a model that accepts an RT format catheter image 518 and outputs second position information indicating the position of the depicted medical instrument. The position information model 619 includes a medical instrument trained model 611, an angle trained model 612, and a position information synthesis unit 615.

The same RT-format catheter image 518 is input to both the medical instrument trained model 611 and the angle trained model 612. First position information is output from the medical instrument trained model 611. As explained using Figure 6, the first position information is the probability that a medical instrument is depicted at each location on the RT-format catheter image 518. In the following explanation, the probability that a medical instrument is depicted at a position where the distance from the center of the image acquisition catheter 40 is r and the scanning angle is θ is denoted by P1(r, θ).

Scanning angle information is output from the angle-trained model 612. The scanning angle information is the probability that a medical instrument is visualized at each scanning angle. In the following explanation, the probability that a medical instrument is visualized in the direction of scanning angle θ is represented as Pt(θ).

The first position information and the scanning angle information are combined in the position information combination unit 615 to generate second position information. Like the first position information, the second position information is the probability that a medical instrument is depicted at each location on the RT format catheter image 518. The input terminal of the position information combination unit 615 performs the functions of a first position information acquisition unit and a scanning angle information acquisition unit.

Since the medical instrument is depicted with a certain degree of spread in the RT catheter image 518, the sum of P1 and the sum of Pt may both be greater than 1. The second position information P2(r, θ) at a position where the distance from the center of the image acquisition catheter 40 is r and the scanning angle is θ is calculated, for example, by equation (1-1).
P2 (r, θ) = P1 (r, θ) + kPt (θ) (1-1)
k is a coefficient relating to the weighting between the first position information and the scanning angle information.

The second position information P2(r, θ) may be calculated by equation (1-2).
P2 (r, θ) = P1 (r, θ) × Pt (θ) (1-2)

The second position information P2(r, θ) may be calculated by equation (1-3), which is an equation for calculating the average value of the first position information and the scanning angle information.
P2 (r, θ) = (P1 (r, θ) + Pt (θ)) / 2 (1-3)

Note that the second position information P2(r, θ) in equations (1-1) to (1-3) is not a probability, but a numerical value that indicates the relative likelihood that a medical instrument has been depicted. Combining the first position information and the scanning angle information improves the accuracy of the scanning angle direction. Note that the second position information may be information regarding the position where the value of P2(r, θ) is largest. The second position information may be determined by a function other than the formulas exemplified in equations (1-1) to (1-3).

The second position information is an example of the position information of the medical instrument obtained in step S502 described using Figure 4. The medical instrument trained model 611, the angle trained model 612, and the position information synthesis unit 615 work together to realize step S502 described using Figure 4. The output terminal of the position information synthesis unit 615 functions as a second position information output unit that outputs second position information based on the first position information and scanning angle information.

Figure 17 is a flowchart explaining the processing flow of the program in embodiment 3. The flowchart explained using Figure 17 shows details of the processing of step S502 explained using Figure 4.

The control unit 21 acquires one frame of an RT-format catheter image 518 (step S541). The control unit 21 inputs the RT-format catheter image 518 into the medical instrument trained model 611 to acquire first position information (step S542). The control unit 21 inputs the RT-format catheter image 518 into the angle trained model 612 to acquire scanning angle information (step S543).

The control unit 21 calculates the second position information, for example, based on equation (1-1) or equation (1-2) (step S544). The control unit 21 then terminates the processing. Thereafter, the control unit 21 uses the second position information calculated in step S544 as the position information in step S502.

According to this embodiment, a catheter system 10 can be provided that accurately calculates the position information of a medical instrument depicted in a catheter image 51.

[Fourth embodiment]
This embodiment relates to a specific example of the classification model 62 described using Fig. 7. Fig. 18 is an explanatory diagram illustrating the configuration of the classification model 62. The classification model 62 includes a first classification trained model 621 and a classification data conversion unit 629.

The first classification trained model 621 accepts the RT-format catheter image 518 and outputs first classified data 521 in which each part constituting the RT-format catheter image 518 is classified into a "biological tissue region," a "non-biological tissue region," and a "medical instrument region." The first classification trained model 621 further outputs the reliability of the classification result for each part, i.e., the probability that the classification result is correct. The output layer of the first classification trained model 621 functions as a first classified data output unit that outputs the first classified data 521.

The diagram in the upper right of Figure 18 shows a schematic representation of the first classification data 521 in RT format. Thick, downward-sloping hatching indicates biological tissue regions such as the atrial and ventricular walls. Black areas indicate medical device regions in which medical devices such as the Brockenbrough needle are depicted. Checkered hatching indicates non-biological tissue regions that are neither medical device regions nor biological tissue regions.

The first classification data 521 is converted into classification data 52 by the classification data conversion unit 629. The diagram at the bottom right of Figure 18 schematically shows RT format classification data 528. Non-biological tissue regions are classified into three types: first lumen region, second lumen region, and non-lumen region. As in Figure 5C, thin hatching slanting downward to the left indicates the first lumen region. Thin hatching slanting downward to the right indicates the second lumen region. Thick hatching slanting downward to the left indicates the non-lumen region.

An overview of the processing performed by the classification data conversion unit 629 will be explained. Of the non-biological tissue regions, the region in contact with the image acquisition catheter 40, i.e., the rightmost region in the first classification data 521, is classified as the first lumen region. Of the non-biological tissue regions, the region surrounded by biological tissue regions is classified as the second lumen region. Note that the classification of the second lumen region is preferably determined after connecting the upper and lower ends of the RT format catheter image 518 to form a cylindrical shape. Of the non-biological tissue regions, regions that are neither the first lumen region nor the second lumen region are classified as non-lumen regions.

Figure 19 is an explanatory diagram explaining the first training data. The first training data is used when generating the first classification trained model 621 through machine learning. The following explanation will be given using an example in which the first training data is created using the information processing device 20 described using Figure 3. The first training data may also be created using a computer or the like other than the information processing device 20.

The control unit 21 displays two types of catheter images 51, an RT format catheter image 518 and an XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks four types of boundary data: the "boundary line between the first lumen region and the biological tissue region," the "boundary line between the second lumen region and the biological tissue region," the "boundary line between the non-lumen region and the biological tissue region," and the "outline of the medical device region."

The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 displays a boundary line corresponding to the marking at the corresponding position on the other catheter image 51. As a result, the labeler can check both the RT format catheter image 518 and the XY format catheter image 519 and make appropriate markings.

The labeler inputs whether each area separated by the four types of boundary data that have been marked is a "living tissue area," a "non-living tissue area," or a "medical instrument area." The control unit 21 may automatically determine the area, and the labeler may provide correction instructions as necessary. Through the above processing, first classification data 521 is created, which clearly indicates whether each area of the catheter image 51 is classified as a "living tissue area," a "non-living tissue area," or a "medical instrument area."

The first classification data 521 will be explained using specific examples. Pixels classified as "biological tissue region" are recorded with a "biological tissue region label," pixels classified as "first lumen region" with a "first lumen region label," pixels classified as "second lumen region" with a "second lumen region label," pixels classified as "non-lumen region" with a "non-lumen region label," pixels classified as "medical instrument region" with a "medical instrument region label," and pixels classified as "non-biological tissue region" with a "non-biological tissue region label." Each label is represented, for example, by an integer. The first classification data 521 is an example of label data that associates pixel positions with labels.

The control unit 21 associates and records the catheter image 51 and the first classification data 521. By repeating the above process and recording multiple sets of data, a first training data DB is created. The following explanation uses as an example a first training data DB in which an RT format catheter image 518 and an RT format first classification data 521 are associated and recorded.

In addition, the control unit 21 may generate XY format classification data 529 based on the XY format catheter image 519. The control unit 21 may generate RT format classification data 528 based on the XY format classification data 529.

Using the flowchart in Figure 12, we will explain an overview of the process of generating the first classification trained model 621. Prior to executing the program in Figure 12, an untrained model, such as a U-Net structure that realizes semantic segmentation, is prepared.

The U-Net structure includes multiple encoder layers followed by multiple decoder layers. Each encoder layer includes a pooling layer and a convolutional layer. Semantic segmentation assigns a label to each pixel that makes up the input image. The untrained model may be a Mask R-CNN model or any other model that achieves image segmentation.

The control unit 21 acquires training records to be used for one epoch of training from the first training data DB (step S571). The control unit 21 adjusts the model parameters so that when an RT format catheter image 518 is input to the model's input layer, first classification data 521 in RT format is output from the output layer (step S572). When acquiring training records and adjusting the model parameters, the program may appropriately have functions to cause the control unit 21 to accept user corrections, present the basis for judgment, perform additional learning, etc.

The control unit 21 determines whether to terminate the processing (step S573). For example, the control unit 21 determines to terminate the processing when learning for a predetermined number of epochs has been completed. The control unit 21 may also obtain test data from the first training data DB, input it into the model under machine learning, and determine to terminate the processing when an output of a predetermined accuracy is obtained.

If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the parameters of the trained first classification trained model 621 in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, a first classification trained model 621 is generated that accepts the catheter image 51 and outputs the first classification data 521.

Prior to executing the program of FIG. 12, a model that accepts time-series input may be prepared. The model that accepts time-series input may, for example, include a memory unit that stores information about previously input RT format catheter images 518. The model that accepts time-series input may also include a recursive input unit that inputs the output for a previously input RT format catheter image 518 together with the next RT format catheter image 518.

By using catheter images 51 acquired in time series, a first classification trained model 621 can be realized that is less susceptible to the influence of image noise, etc., and outputs first classification data 521 with high accuracy.

The first classification trained model 621 may be created using a computer or the like other than the information processing device 20. The first classification trained model 621 after machine learning may be copied to the auxiliary storage device 23 via a network. The first classification trained model 621 trained on a single piece of hardware can be used by multiple information processing devices 20.

Figure 20 is a flowchart explaining the processing flow of the program of embodiment 4. The flowchart explained using Figure 20 shows details of the processing performed by the classification model 62 explained using Figure 7.

The control unit 21 acquires one frame of the RT-format catheter image 518 (step S551). The control unit 21 inputs the RT-format catheter image 518 into the first classification trained model 621 to acquire the first classification data 521 (step S552). The control unit 21 extracts one continuous non-biological tissue region from the first classification data 521 (step S553). Note that it is desirable that processing subsequent to the extraction of the non-biological tissue region be performed with the upper and lower ends of the RT-format catheter image 518 connected to form a cylindrical shape.

The control unit 21 determines whether the non-biological tissue region extracted in step S552 is on the side in contact with the image acquisition catheter 40, i.e., the portion in contact with the left edge of the RT-format catheter image 518 (step S554). If it is determined that the non-biological tissue region is on the side in contact with the image acquisition catheter 40 (YES in step S554), the control unit 21 determines that the non-biological tissue region extracted in step S553 is the first lumen region (step S555).

If it is determined that the non-living tissue region extracted in step S552 is not in contact with the imaging catheter 40 (NO in step S554), the control unit 21 determines whether the non-living tissue region extracted in step S552 is surrounded by a living tissue region (step S556). If it is determined that the non-living tissue region is surrounded by a living tissue region (YES in step S556), the control unit 21 determines that the non-living tissue region extracted in step S553 is a second lumen region (step S557). Through steps S555 and S557, the control unit 21 realizes the function of the lumen region extraction unit.

If it is determined that the non-biological tissue region is not surrounded by a biological tissue region (NO in step S556), the control unit 21 determines that the non-biological tissue region extracted in step S553 is a non-luminal region (step S558).

After step S555, step S557, or step S558 is completed, the control unit 21 determines whether processing of all non-biological tissue regions has been completed (step S559). If it is determined that processing has not been completed (NO in step S559), the control unit 21 returns to step S553. If it is determined that processing has been completed (YES in step S559), the control unit 21 terminates processing.

The control unit 21 realizes the function of the classification data conversion unit 629 by processing steps S553 to S559.

The first classification trained model 621 may be a model that classifies the XY-format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical instrument region. The first classification trained model 621 may be a model that classifies the RT-format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler does not need to mark the medical instrument region.

According to this embodiment, a first classification trained model 621 can be generated that classifies a catheter image 51 into a biological tissue region, a non-biological tissue region, and a medical device region. According to this embodiment, a catheter system 10 can be provided that generates classification data 52 using the generated first classification trained model 621.

[Modification 4-1]
The labeler may input whether each region separated by the four types of marked boundary data is a “biological tissue region,” a “first lumen region,” a “second lumen region,” a “non-lumen region,” or a “medical instrument region.” By performing machine learning using the first training data DB created in this way, it is possible to generate a first classification trained model 621 that classifies the catheter image 51 into a “biological tissue region,” a “first lumen region,” a “second lumen region,” a “non-lumen region,” and a “medical instrument region.”

By doing this, a classification model 62 can be realized that classifies the catheter image 51 into "biological tissue region," "first lumen region," "second lumen region," "non-lumen region," and "medical device region" without using the classification data conversion unit 629.

Fifth Embodiment
This embodiment relates to a catheter system 10 that uses a composite classification model 626 that combines classification data 52 output from two classification trained models. Explanation of parts common to the fourth embodiment will be omitted.

Figure 21 is an explanatory diagram illustrating the configuration of the classification model 62 in embodiment 5. The classification model 62 includes a composite classification model 626 and a classification data conversion unit 629. The composite classification model 626 includes a first classification trained model 621, a second classification trained model 622, and a classification data synthesis unit 628. The first classification trained model 621 is the same as in embodiment 4, so its description is omitted.

The second classification trained model 622 is a model that accepts the RT-format catheter image 518 and outputs second classification data 522 in which each part constituting the RT-format catheter image 518 is classified into a "biological tissue region," a "non-biological tissue region," and a "medical instrument region." The second classification trained model 622 further outputs the reliability of the classification result for each part, i.e., the probability that the classification result is correct. Details of the second classification trained model 622 will be described later.

The classification data synthesis unit 628 synthesizes the first classification data 521 and the second classification data 522 to generate synthesized classification data 526. That is, the input terminal of the classification data synthesis unit 628 realizes the functions of the first classification data acquisition unit and the second classification data acquisition unit. The output terminal of the classification data synthesis unit 628 realizes the function of the synthesized classification data output unit.

Details of the combined classification data 526 will be described later. The combined classification data 526 is converted into classification data 52 by the classification data conversion unit 629. The processing performed by the classification data conversion unit 629 is the same as in embodiment 4, so explanation will be omitted.

Figure 22 is an explanatory diagram explaining the second training data. The second training data is used when generating the second classification learned model 622 through machine learning. The following explanation will use as an example the case where the second training data is created using the information processing device 20 described using Figure 3. The second training data may also be created using a computer or the like other than the information processing device 20.

The control unit 21 displays two types of catheter images 51, an RT format catheter image 518 and an XY format catheter image 519, on the display device 31. The labeler observes the displayed catheter image 51 and marks two types of boundary data: the "boundary line between the first lumen region and the biological tissue region" and the "outline of the medical device region."

The labeler may mark either the RT format catheter image 518 or the XY format catheter image 519. The control unit 21 displays a boundary line corresponding to the marking at the corresponding position on the other catheter image 51. As a result, the labeler can check both the RT format catheter image 518 and the XY format catheter image 519 and make appropriate markings.

The labeler inputs whether each area separated by the two types of boundary data that have been marked is a "living tissue area," a "non-living tissue area," or a "medical instrument area." The control unit 21 may automatically determine the area, and the labeler may provide correction instructions as necessary. Through the above processing, second classification data 522 is created, which clearly indicates whether each part of the catheter image 51 is classified as a "living tissue area," a "non-living tissue area," or a "medical instrument area."

The second classification data 522 will be explained using a specific example. A "biological tissue region label" is recorded for pixels classified as a "biological tissue region," a "non-biological tissue region label" is recorded for pixels classified as a "non-biological tissue region," and a "medical instrument region label" is recorded for pixels classified as a "medical instrument region." Each label is represented, for example, by an integer. The second classification data 522 is an example of label data that associates the position of a pixel with a label.

The control unit 21 associates and records the catheter image 51 and the second classification data 522. By repeating the above process and recording many sets of data, a second training data DB is created. By performing a process similar to the machine learning described in embodiment 4 using the second training data DB, a second classification trained model 622 can be generated.

The second classification trained model 622 may be a model that classifies the XY-format catheter image 519 into a biological tissue region, a non-biological tissue region, and a medical instrument region. The second classification trained model 622 may be a model that classifies the RT-format catheter image 518 into a biological tissue region and a non-biological tissue region. In such a case, the labeler does not need to mark the medical instrument region.

The second classification data 522 can be created in a shorter time than the first classification data 521. The labeler who creates the second classification data 522 can be trained in a shorter time than the labeler who creates the first classification data 521. As a result, a larger amount of training data can be registered in the second training data DB than in the first training DB.

Because a large amount of training data can be used, it is possible to generate a second classification trained model 622 that can distinguish the boundary between the first lumen region and the biological tissue region and the outline of the medical instrument region with higher accuracy than the first classification trained model 621. However, since the second classification trained model 622 has not learned about non-biological tissue regions other than the first lumen region, it cannot distinguish them from biological tissue regions.

The processing performed by the classification data synthesis unit 628 will be described. The same RT format catheter image 518 is input to both the first classification trained model 621 and the second classification trained model 622. First classification data 521 is output from the medical instrument trained model 611. Second classification data 522 is output from the second classification trained model 622.

In the following explanation, the first classification trained model 621 and the second classification trained model 622 will be described using an example in which they output a classified label and the reliability of that label for each pixel of the RT format catheter image 518. Note that the first classification trained model 621 and the second classification trained model 622 may output a classified label and probability for each range of, for example, 3 pixels vertically and 3 pixels horizontally of the RT format catheter image 518, for a total of 9 pixels.

For a pixel that is at a distance r from the center of the image acquisition catheter 40 and a scanning angle θ, the confidence that the first classification trained model 621 is in a biological tissue region is indicated by Q1t(r,θ). Note that for pixels that the first classification trained model 621 classifies as being in a region other than biological tissue, Q1t(r,θ) = 0.

Similarly, for a pixel whose distance from the center of the image acquisition catheter 40 is r and whose scanning angle is θ, the confidence that the second classification trained model 622 believes it is a biological tissue region is denoted by Q2t(r,θ). Note that for pixels that the second classification trained model 622 classifies as being in a region other than a biological tissue region, Q2t(r,θ) = 0.

The classification data synthesis unit 628 calculates the synthesis value Qt(r, θ) based on, for example, equation (5-1). Note that Qt(r, θ) is not the probability that the classification as a biological tissue region is correct, but is a numerical value that relatively indicates the degree of reliability that the region is a biological tissue region.
Qt (r, θ) = Q1t (r, θ) × Q2t (r, θ) (5-1)
The classification data synthesis unit 628 classifies pixels with Qt(r, θ) of 0.5 or more into the biological tissue region.

Similarly, the confidence that the first classification trained model 621 is in the medical equipment domain is indicated by Q1c(r,θ), and the confidence that the second classification trained model 622 is in the medical equipment domain is indicated by Q2c(r,θ).

The classification data synthesis unit 628 calculates the synthesis value Qc(r, θ) based on, for example, equation (5-2). Note that Qc(r, θ) is not the probability that the classification as a medical instrument area is correct, but is a numerical value that relatively indicates the reliability of the medical instrument area.
Qc (r, θ) = Q1c (r, θ) × Q2c (r, θ) (5-2)

The classification data synthesis unit 628 classifies pixels with Qc(r, θ) of 0.5 or greater into the medical instrument region. The classification data synthesis unit 628 classifies pixels that are not classified into either the medical instrument region or the biological tissue region into the non-biological tissue region. As a result, the classification data synthesis unit 628 generates synthesized classification data 526 by synthesizing the first classification data 521 and the second classification data 522. The synthesized classification data 526 is converted into RT-format classification data 528 by the classification data conversion unit 629.

Note that equations (5-1) and (5-2) are examples. The thresholds used by the classification data synthesis unit 628 when performing classification are also examples. The classification data synthesis unit 628 may be a trained model that accepts the first classification data 521 and the second classification data 522 and outputs the synthesized classification data 526.

The first classification data 521 may be classified into "biological tissue region," "first lumen region," "second lumen region," "non-lumen region," and "medical instrument region" by the classification data conversion unit 629 described in embodiment 4, and then input to the classification data synthesis unit 628.

The first classification trained model 621 may be a model that classifies the catheter image 51 described in variant example 4-1 into a "biological tissue region," a "first lumen region," a "second lumen region," a "non-lumen region," and a "medical device region."

When data already classified into "first lumen region," "second lumen region," and "non-lumen region" for non-biological tissue regions is input to the data synthesis unit 628, the classification data synthesis unit 628 can output composite classification data 526 already classified into "biological tissue region," "first lumen region," "second lumen region," "non-lumen region," and "medical device region." In this case, there is no need to input the composite classification data 526 to the classification data conversion unit 629 to convert it into RT-format classification data 528.

Figure 23 is a flowchart explaining the processing flow of the program of embodiment 5. The flowchart explained using Figure 23 shows details of the processing performed by the classification model 62 explained using Figure 7.

The control unit 21 acquires one frame of the RT format catheter image 518 (step S581). By step S581, the control unit 21 realizes the function of the image acquisition unit. The control unit 21 inputs the RT format catheter image 518 into the first classification trained model 621 to acquire the first classified data 521 (step S582). The control unit 21 inputs the RT format catheter image 518 into the second classification trained model 622 to acquire the second classified data 522 (step S583).

The control unit 21 starts a classification synthesis subroutine (step S584). The classification synthesis subroutine is a subroutine that combines the first classification data 521 and the second classification data 522 to generate combined classification data 526. The processing flow of the classification synthesis subroutine will be described later.

The control unit 21 extracts one continuous non-biological tissue region from the composite classification data 526 (step S585). Note that it is desirable that the processing subsequent to the extraction of the non-biological tissue region be performed with the upper and lower ends of the RT format catheter image 518 connected to form a cylindrical shape.

The control unit 21 determines whether the non-biological tissue region extracted in step S585 is on the side in contact with the imaging catheter 40 (step S554). The subsequent processing up to step S559 is the same as the processing flow of the program of embodiment 4 described using Figure 20, and therefore will not be described further.

The control unit 21 determines whether processing of all non-biological tissue regions has been completed (step S559). If it is determined that processing has not been completed (NO in step S559), the control unit 21 returns to step S585. If it is determined that processing has been completed (YES in step S559), the control unit 21 terminates processing.

Figure 24 is a flowchart explaining the processing flow of the classification synthesis subroutine. The classification synthesis subroutine is a subroutine that combines the first classification data 521 and the second classification data 522 to generate combined classification data 526.

The control unit 21 selects a pixel to be processed (step S601). The control unit 21 obtains the reliability Q1t(r,θ) that the pixel being processed is a biological tissue region from the first classification data 521 (step S602). The control unit 21 obtains the reliability Q2t(r,θ) that the pixel being processed is a biological tissue region from the second classification data 522 (step S603).

The control unit 21 calculates the composite value Qt(r, θ), for example, based on equation (5-1) (step S604). The control unit 21 determines whether the composite value Qt(r, θ) is greater than or equal to a predetermined threshold (step S605). The predetermined threshold is, for example, 0.5.

If it is determined that the difference is greater than or equal to the predetermined threshold (YES in step S605), the control unit 21 classifies the pixel being processed as a "biological tissue region" (step S606). If it is determined that the difference is less than the predetermined threshold (NO in step S605), the control unit 21 obtains from the first classification data 521 the reliability Q1c(r,θ) that the pixel being processed is a medical device region (step S611). The control unit 21 obtains from the second classification data 522 the reliability Q2c(r,θ) that the pixel being processed is a medical device region (step S612).

The control unit 21 calculates the composite value Qc(r, θ), for example, based on equation (5-2) (step S613). The control unit 21 determines whether the composite value Qc(r, θ) is greater than or equal to a predetermined threshold (step S614). The predetermined threshold is, for example, 0.5.

If it is determined that the difference is equal to or greater than the predetermined threshold (YES in step S614), the control unit 21 classifies the pixel being processed as a "medical device region" (step S615). If it is determined that the difference is less than the predetermined threshold (NO in step S614), the control unit 21 classifies the pixel being processed as a "non-biological tissue region" (step S616).

After step S606, step S615, or step S616 is completed, the control unit 21 determines whether processing of all pixels has been completed (step S607). If it is determined that processing has not been completed (NO in step S607), the control unit 21 returns to step S601. If it is determined that processing has been completed (YES in step S607), the control unit 21 terminates processing. The control unit 21 realizes the function of the classification data synthesis unit 628 through the classification synthesis subroutine.

According to this embodiment, a catheter system 10 can be provided that generates RT-format classification data 528 using composite classification data 526 obtained by combining classification data 52 output from two classification trained models. By combining a second classification trained model 622, which allows for relatively easy collection of a large amount of training data and high classification accuracy, with a first classification trained model 621, which requires more effort to collect training data, a catheter system 10 can be provided that offers a good balance between the cost of generating the trained model and classification accuracy.

Sixth Embodiment
This embodiment relates to a catheter system 10 that uses position information of a medical instrument as a hint to classify each part that constitutes a catheter image 51. Explanation of parts common to the first embodiment will be omitted.

Figure 25 is an explanatory diagram illustrating the configuration of the hinted trained model 631. The hinted trained model 631 is used in step S604 described using Figure 4 instead of the classification model 62 described using Figure 7.

The hinted trained model 631 is a model that accepts the RT-format catheter image 518 and the positional information of the medical device depicted in the RT-format catheter image 518, and outputs hinted classification data 561 that classifies each part of the RT-format catheter image 518 into a "biological tissue region," a "non-biological tissue region," and a "medical device region." The first classification trained model 621 further outputs the reliability of the classification result for each part, i.e., the probability that the classification result is correct.

Figure 26 is an explanatory diagram illustrating the record layout of the hinted training data DB 72. The hinted training data DB 72 is a database that records, in association with each other, catheter images 51, positional information of medical instruments depicted in the catheter images 51, and classification data 52 that classifies each part of the catheter images 51 by the subjects depicted.

The classification data 52 is data created by a labeler based on the procedure described, for example, using Figure 19. A hinted trained model 631 can be generated by performing processing similar to the machine learning described in embodiment 4 using the hinted training data DB 72.

Figure 27 is a flowchart explaining the processing flow of the program in embodiment 6. The flowchart explained using Figure 27 shows details of the processing performed in step S504 explained using Figure 4.

The control unit 21 acquires one frame of the RT format catheter image 518 (step S621). The control unit 21 inputs the RT format catheter image 518 into the medical instrument trained model 611 described using Figure 6, for example, to acquire position information of the medical instrument (step S622). The control unit 21 inputs the RT format catheter image 518 and the position information into the hint-trained model 631 to acquire the hint-based classification data 561 (step S623).

The control unit 21 extracts one continuous non-biological tissue region from the hint-included classification data 561 (step S624). Note that it is desirable that the processing subsequent to the extraction of the non-biological tissue region be performed with the upper and lower ends of the RT format catheter image 518 connected to form a cylindrical shape.

The control unit 21 determines whether the non-biological tissue region extracted in step S624 is on the side in contact with the imaging catheter 40 (step S554). The processing from this point onward up to step S559 is the same as the processing flow of the program of embodiment 4 described using Figure 20, and therefore will not be described again.

The control unit 21 determines whether processing of all non-biological tissue regions has been completed (step S559). If it is determined that processing has not been completed (NO in step S559), the control unit 21 returns to step S624. If it is determined that processing has been completed (YES in step S559), the control unit 21 terminates processing.

According to this embodiment, a catheter system 10 can be provided that accurately generates classification data 52 by inputting location information of the medical instrument as a hint.

[Modification 6-1]
28 is a flowchart illustrating the flow of processing of the program of the modified example. The processing explained using FIG. 28 is executed instead of the processing explained using FIG.

The control unit 21 acquires one frame of the RT format catheter image 518 (step S621). The control unit 21 acquires the position information of the medical instrument (step S622). The control unit 21 determines whether or not the acquisition of the position information of the medical instrument was successful (step S631). For example, if the reliability output from the medical instrument trained model 611 is higher than a threshold, the control unit 21 determines that the acquisition of the position information was successful.

Note that "success" in step S631 means that the medical instrument is depicted in the RT-format catheter image 518 and the control unit 21 was able to obtain position information for the medical instrument with a reliability higher than the threshold. Cases where "success" is not achieved include, for example, when the medical instrument is not present within the imaging range of the RT-format catheter image 518, and when the medical instrument is in close contact with the surface of the biological tissue region and is not clearly depicted.

If it is determined that the acquisition of the position information was successful (YES in step S631), the control unit 21 inputs the RT-format catheter image 518 and the position information into the hinted trained model 631 to acquire hinted classified data 561 (step S623). If it is determined that the acquisition of the position information was not successful (NO in step S631), the control unit 21 inputs the RT-format catheter image 518 into the non-hint trained model 632 to acquire non-hint classified data (step S632).

The unhinted trained model 632 is, for example, the classification model 62 described using Figure 7, Figure 18, or Figure 21. Similarly, the unhinted classified data is the classified data 52 output from the classification model 62.

After step S623 or step S632 is completed, the control unit 21 extracts one continuous non-biological tissue region from the classification data with hints 561 or the classification model 62 (step S624). The subsequent processing is the same as the processing flow described using Figure 27, so further description will be omitted.

The hinted classification data 561 is an example of the first data. The hinted trained model 631 is an example of a first trained model that outputs the first data when a catheter image 51 and position information of a medical instrument are input. The output layer of the hinted trained model 631 is an example of a first data output unit that outputs the first data.

The unclassified hint data is an example of the second data. The unclassified hint trained model 632 is an example of the second trained model and the second model that outputs the second data when the catheter image 51 is input. The output layer of the unclassified hint trained model 632 is an example of the second data output unit.

According to this modified example, if location information is not successfully acquired, a classification model 62 that does not require the input of location information is used. Therefore, a catheter system 10 can be provided that prevents malfunctions caused by inputting incorrect hints into the hint-trained model 631.

Seventh Embodiment
This embodiment relates to a catheter system 10 that generates synthetic data 536 by synthesizing the output of a trained model with hints 631 and the output of a trained model without hints 632. Explanation of parts common to the sixth embodiment will be omitted. The synthetic data 536 is data that is used in place of the classification data 52 that is the output of step S504 described using FIG. 4 .

Figure 29 is an explanatory diagram illustrating the configuration of the classification model 62 in embodiment 7. The classification model 62 includes a position classification analysis unit 66 and a third synthesis unit 543. The position classification analysis unit 66 includes a position information acquisition unit 65, a trained model with hints 631, a trained model without hints 632, a first synthesis unit 541, and a second synthesis unit 542.

The position information acquisition unit 65 acquires position information indicating the position at which the medical instrument is depicted, for example, from the medical instrument trained model 611 described using Figure 6 or the position information model 619 described using Figure 16. The trained model with hints 631 is the same as in embodiment 6, so its description will be omitted. The trained model without hints 632 is, for example, the classification model 62 described using Figure 7, Figure 18, or Figure 21.

The operation of the first synthesis unit 541 will be described. The first synthesis unit 541 synthesizes the hint-included classified data 561 output from the hint-included trained model 631 and the hint-unclassified data output from the hint-unclassified trained model 632 to create classification information. The input terminal of the first synthesis unit 541 functions as a first data acquisition unit that acquires the hint-included classified data 561 and a second data acquisition unit that acquires the hint-unclassified data. The output terminal of the first synthesis unit 541 functions as a first synthesized data output unit that outputs first synthesized data obtained by synthesizing the hint-included classified data 561 and the hint-unclassified data.

When data is input in which the non-biological tissue region is not classified into the first lumen region, the second lumen region, and the non-lumen region, the first synthesis unit 541 performs the function of the classification data conversion unit 629 and classifies the non-biological tissue region.

For example, if the location information acquisition unit 65 successfully acquires location information, the first synthesis unit 541 increases the weight of the hinted trained model 631 compared to the weight of the hintless trained model 632 and synthesizes the two. Methods for weighted image synthesis are well known, so explanations will be omitted.

The first synthesis unit 541 may determine the weighting of the hint-containing classified data 561 and the hint-uncontaining classified data based on the reliability of the location information acquired by the location information acquisition unit 65 and synthesize them.

The first synthesis unit 541 may synthesize the hint-containing classified data 561 and the hint-unclassified data based on the reliability of each region of the hint-containing classified data 561 and the hint-unclassified data. Synthesis based on the reliability of the classified data 52 can be performed, for example, by processing similar to that of the classified data synthesis unit 628 described in embodiment 5.

The first synthesis unit 541 treats the medical device regions output from the hint-trained model 631 and the hint-untrained model 632 in the same way as adjacent non-biological tissue regions. For example, if a medical device region exists in the first lumen region, the first synthesis unit 541 treats the medical device region in the same way as the first lumen region. Similarly, if a medical device region exists in the second lumen region, the first synthesis unit 541 treats the medical device region in the same way as the second lumen region.

A trained model that does not output medical equipment regions may be used for either the trained model with hints 631 or the trained model without hints 632. Therefore, as shown in the center of Figure 29, the classification information output from the first synthesis unit 541 does not include information regarding medical equipment regions.

The first synthesis unit 541 may function as a switch that switches between the hint-containing classified data 561 and the hint-unclassified data based on whether the location information acquisition unit 65 has successfully acquired location information. The first synthesis unit 541 may also function as the classified data conversion unit 629.

Specifically, if the location information acquisition unit 65 succeeds in acquiring location information, the first synthesis unit 541 outputs classification information based on the hinted classification data 561 output from the hinted trained model 631. If the location information acquisition unit 65 does not succeed in acquiring location information, the first synthesis unit 541 outputs classification information based on the hint-unclassified data output from the hint-untrained model 632.

The operation of the second synthesis unit 542 will be described. If the location information acquisition unit 65 is successful in acquiring location information, the second synthesis unit 542 outputs the medical instrument area output from the hinted trained model 631. If the location information acquisition unit 65 is not successful in acquiring location information, the second synthesis unit 542 outputs the medical instrument area included in the hint-unclassified data.

It is desirable to use the second classification trained model 622 described using Figure 21 for the hint-free trained model 632. As mentioned above, a large amount of training data can be used to train the second classification trained model 622, allowing for accurate extraction of the medical device area.

If the location information acquisition unit 65 is unsuccessful in acquiring location information, the second synthesis unit 542 may synthesize and output the medical device area included in the classification data with hints 561 and the medical device area included in the classification data without hints. The synthesis of the classification data with hints 561 and the classification data without hints can be performed, for example, by processing similar to that of the classification data synthesis unit 628 described in embodiment 5.

The output terminal of the second synthesis unit 542 functions as a second synthesis data output unit that outputs second synthesis data obtained by synthesizing the medical device area of the classification data with hints 561 and the medical device area of the classification data without hints.

The operation of the third synthesis unit 543 will be described. The third synthesis unit 543 outputs synthesized data 536 in which the medical instrument area output from the second synthesis unit 542 is superimposed on the classification information output from the first synthesis unit 541. In Figure 29, the superimposed medical instrument area is shown in black.

Instead of the first synthesis unit 541, the third synthesis unit 543 may perform the function of the classification data conversion unit 629, which classifies non-biological tissue regions into first lumen regions, second lumen regions, and non-lumen regions.

Some or all of the multiple trained models that make up the position classification analysis unit 66 may be models that accept multiple catheter images 51 acquired in chronological order and output information for the most recent catheter image 51.

This embodiment provides a catheter system 10 that acquires positional information of medical instruments with high accuracy and outputs it in combination with classification information. The control unit 21 may generate composite data 536 based on each of multiple catheter images 51 captured continuously along the longitudinal direction of the image acquisition catheter 40, and then stack the composite data 536 to construct and display three-dimensional data of the biological tissue and medical instruments.

[Modification 7-1]
30 is an explanatory diagram illustrating the configuration of a modified classification model 62. An X% hint trained model 639 is added to the position classification analysis unit 66. The X% hint trained model 639 is a model trained using the hinted training data DB 72 under the condition that position information is input for X percent of the training data and not input for (100-X) percent. In the following description, the data output from the X% hint trained model 639 will be referred to as X% hint classification data.

The X% hint trained model 639 is identical to the hint trained model 631 when X is "100", and is identical to the no-hint trained model 632 when X is "0". X is, for example, "50".

The first synthesis unit 541 outputs data obtained by synthesizing the classification data 52 obtained from the trained model with hints 631, the trained model without hints 632, and the trained model with X% hints 639 based on a predetermined weighting. The weighting changes depending on whether the location information acquisition unit 65 successfully acquires location information.

For example, if the location information acquisition unit 65 succeeds in acquiring location information, the output of the hinted trained model 631 and the output of the X% hint trained model 639 are combined. If the location information acquisition unit 65 fails to acquire location information, the output of the no-hint trained model 632 and the output of the X% hint trained model 639 are combined. The weighting during combination may vary based on the reliability of the location information acquired by the location information acquisition unit 65.

The location classification analysis unit 66 may include multiple X% hint trained models 639. For example, an X% hint trained model 639 where X is "20" and an X% hint trained model 639 where X is "50" can be used in combination.

In clinical settings, there are cases where it is not possible to extract a medical instrument region from the catheter image 51. For example, this occurs when the medical instrument is not inserted into the first cavity, or when the medical instrument is in close contact with the surface of biological tissue. This modified example makes it possible to realize a classification model 62 that matches the actual situation in such clinical settings. As a result, it is possible to provide a catheter system 10 that can accurately detect and classify position information.

Eighth Embodiment
This embodiment relates to three-dimensional display of a catheter image 51. Explanation of parts common to the seventh embodiment will be omitted. Fig. 31 is an explanatory diagram for explaining an outline of the processing of the eighth embodiment.

In this embodiment, multiple RT-format catheter images 518 captured continuously along the longitudinal direction of the image acquisition catheter 40 are used. The control unit 21 inputs each of the multiple RT-format catheter images 518 to the position classification analysis unit 66 described in embodiment 7. The position classification analysis unit 66 outputs classification information and medical device regions corresponding to each RT-format catheter image 518. The control unit 21 inputs the classification information and medical device information to the third synthesis unit 543 to synthesize synthesized data 536.

The control unit 21 creates biological three-dimensional data 551 representing the three-dimensional structure of biological tissue based on multiple composite data 536. The biological three-dimensional data 551 is, for example, voxel data that records values representing biological tissue labels, first lumen region labels, second lumen region labels, non-lumen region labels, etc. for each volumetric grid in three-dimensional space. The biological three-dimensional data 551 may also be polygon data composed of multiple polygons representing the boundaries of each region. Methods for creating three-dimensional data 55 based on multiple RT format data are well known, so explanations are omitted.

The control unit 21 acquires position information indicating the position of the medical instrument depicted in each RT format catheter image 518 from the position information acquisition unit 65 included in the position classification analysis unit 66. The control unit 21 creates medical instrument three-dimensional data 552 indicating the three-dimensional shape of the medical instrument based on the multiple pieces of position information. Details of the medical instrument three-dimensional data 552 will be described later.

The control unit 21 generates three-dimensional data 55 by combining the biological three-dimensional data 551 and the medical instrument three-dimensional data 552. The three-dimensional data 55 is used for the "3D display" in step S513 described using FIG. 4. When combining the three-dimensional data 55, the control unit 21 replaces the medical instrument area included in the combined data 536 with a blank area or a non-biological area, and then combines the medical instrument three-dimensional data 552. The control unit 21 may generate the biological three-dimensional data 551 using classification information output from the first combination unit 541 included in the position classification analysis unit 66.

Figures 32A to 32D are explanatory diagrams outlining the process of correcting position information. Figures 32A to 32D are schematic diagrams showing, in chronological order, the state in which catheter images 51 are captured while the image acquisition catheter 40 is pulled to the right in the figure. The thick cylinder schematically shows the inner surface of the first cavity.

In Figure 32A, three catheter images 51 have been taken. The position information of the medical device extracted from each catheter image 51 is indicated by a white circle. Figure 32B shows the state after the fourth catheter image 51 has been taken. The position information of the medical device extracted from the fourth catheter image 51 is indicated by a black circle.

The medical device has been detected in a location that is clearly different from the three catheter images 51 taken previously. Medical devices used in IVR generally have a certain degree of rigidity and are unlikely to bend suddenly. Therefore, the location information indicated by the black circle is likely to be a false detection.

In Figure 32C, two more catheter images 51 have been taken. The position information of the medical device extracted from each catheter image 51 is shown by the white circles. The five white circles are aligned almost in a line along the longitudinal direction of the image acquisition catheter 40, but the black circles are far apart, clearly indicating a false detection.

In Figure 32D, the positional information interpolated based on the five white circles is indicated by crosses. By using the positional information indicated by the crosses instead of the positional information indicated by the black circles, the shape of the medical device in the first cavity can be correctly displayed in the three-dimensional image.

If the location information acquisition unit 65 is not successful in acquiring location information, the control unit 21 may use, as location information, a representative point of the medical instrument region acquired from the second synthesis unit 542 included in the location classification analysis unit 66. For example, the center of gravity of the medical instrument region can be used as the representative point.

Figure 33 is a flowchart illustrating the processing flow of the program of embodiment 8. The program described using Figure 33 is a program executed when it is determined in step S505 described using Figure 4 that the user has specified three-dimensional display (3D in step S505).

The program in Figure 33 can be executed while multiple catheter images 51 are being captured along the longitudinal direction of the image acquisition catheter 40. Prior to execution of the program in Figure 33, classification information and location information have already been generated for each of the captured catheter images 51, and this will be explained using an example in which this information has been stored in the auxiliary storage device 23 or an external mass storage device.

The control unit 21 acquires position information corresponding to one catheter image 51 and records it in the main memory device 22 or the auxiliary memory device 23 (step S641). The control unit 21 processes the series of catheter images 51 in order, starting with the catheter image 51 that was stored first. In step S641, the control unit 21 may acquire and record position information from the first few catheter images 51 in the series of catheter images 51.

The control unit 21 acquires position information corresponding to the next catheter image 51 (step S642). In the following description, the position information being processed will be referred to as first position information. The control unit 21 extracts the position information closest to the first position information from the position information acquired in step S641 and previous steps S641 (step S643). In the following description, the position information extracted in step S643 will be referred to as second position information.

In step S642, the distances between the positional information are compared while projecting the multiple catheter images 51 onto a plane perpendicular to the image acquisition catheter 40. In other words, when extracting the second positional information, the longitudinal distance of the image acquisition catheter 40 is not taken into consideration.

The control unit 21 determines whether the distance between the first position information and the second position information is equal to or less than a predetermined threshold (step S644). The threshold is, for example, 3 millimeters. If it is determined that the distance is equal to or less than the threshold (YES in step S644), the control unit 21 records the second position information in the main memory device 22 or the auxiliary memory device 23 (step S645).

If it is determined that the threshold value is exceeded (NO in step S644), or after step S645 is completed, the control unit 21 determines whether processing of the recorded location information has been completed (step S646). If it is determined that processing has not been completed (NO in step S646), the control unit 21 returns to step S642.

The location information indicated by black circles in Figure 32 is an example of location information that is determined to exceed the threshold in step S644. The control unit 21 ignores such location information without recording it in step S645. By processing when the control unit 21 determines NO in step S644, the control unit 21 realizes the function of an exclusion unit that excludes location information that does not satisfy predetermined conditions. Note that the control unit 21 may record location information that is determined to exceed the threshold in step S644 by attaching a flag indicating an "error."

If it is determined that the process has ended (YES in step S646), the control unit 21 determines whether or not the location information can be complemented based on the location information recorded in steps S641 and S645 (step S647). If it is determined that the location information can be complemented (YES in step S647), the control unit 21 complements the location information (step S648).

In step S648, the control unit 21 complements position information that replaces the position information determined to exceed the threshold in step S644, for example. The control unit 21 may complement position information between catheter images 51. The complementation can be performed using any method, such as linear interpolation, spline interpolation, Lagrange interpolation, or Newtonian interpolation. In step S648, the control unit 21 realizes the function of a complementing unit that adds complementary information to the position information.

If it is determined that the position information cannot be complemented (NO in step S647), or after step S648 is completed, the control unit 21 starts a three-dimensional display subroutine (step S649). The three-dimensional display subroutine is a subroutine that performs three-dimensional display based on a series of catheter images 51. The processing flow of the three-dimensional display subroutine will be described later.

The control unit 21 determines whether to terminate the processing (step S650). For example, if the MDU 33 starts a new pullback operation, i.e., starts capturing a catheter image 51 to be used to generate a three-dimensional image, the control unit 21 determines to terminate the processing.

If it is determined that the processing should not be terminated (NO in step S650), the control unit 21 returns to step S642. If it is determined that the processing should be terminated (YES in step S650), the control unit 21 terminates the processing.

In parallel with the execution of the program in Figure 33, the control unit 21 generates and records classification information and position information based on the newly captured catheter image 51. In other words, if it is determined that the process has ended in step S646, steps S647 and onward are executed, but new position information and classification information may have been generated while steps S647 to S650 are being executed.

Figure 34 is a flowchart explaining the processing flow of the three-dimensional display subroutine. The three-dimensional display subroutine is a subroutine that performs three-dimensional display based on a series of catheter images 51. The three-dimensional display subroutine enables the control unit 21 to realize the functions of the three-dimensional output unit.

The control unit 21 acquires composite data 536 corresponding to the series of catheter images 51 (step S661). The control unit 21 creates biological three-dimensional data 551 showing the three-dimensional structure of biological tissue based on the series of composite data 536 (step S662).

As described above, when synthesizing the three-dimensional data 55, the control unit 21 replaces the medical instrument area included in the synthesized data 536 with a blank area or a non-biological area, and then synthesizes the medical instrument three-dimensional data 552. The control unit 21 may generate the biological three-dimensional data 551 using classification information output from the first synthesis unit 541 included in the position classification analysis unit 66. The control unit 21 may also generate the biological three-dimensional data 551 based on the first classification data 521 described using FIG. 18. In other words, the control unit 21 can generate the biological three-dimensional data 551 directly based on multiple first classification data 521.

The control unit 21 may generate the biometric three-dimensional data 551 indirectly based on the plurality of first classification data 521. "Indirectly based" means generating the biometric three-dimensional data 551 based on a plurality of synthetic data 536 generated using the plurality of first classification data 521, as explained using Figure 31, for example. The control unit 21 may generate the biometric three-dimensional data 551 based on a plurality of data different from the synthetic data 536 generated using the first plurality of classification data 521.

The control unit 21 assigns thickness information to the curve defined by the series of positional information recorded in steps S641 and S645 of the program described using FIG. 33 and the complementary information complemented in step S648 (step S663). The thickness information is preferably the thickness of a medical instrument commonly used in IVR procedures. The control unit 21 may also accept information about the medical instrument currently in use and assign thickness information corresponding to that medical instrument. By assigning thickness information, the three-dimensional shape of the medical instrument is reproduced.

The control unit 21 synthesizes the three-dimensional shape of the medical instrument generated in step S662 with the three-dimensional biological data 551 generated in step S662 (step S664). The control unit 21 displays the synthesized three-dimensional data 55 on the display device 31 (step S665).

The control unit 21 accepts instructions from the user for rotating, changing the cross section, enlarging, reducing, etc., the three-dimensionally displayed image, and changes the display. Since accepting instructions for three-dimensionally displayed images and changing the display have been done conventionally, explanations will be omitted. The control unit 21 then ends processing.

This embodiment provides a catheter system 10 that eliminates the effects of erroneous position information detection and displays the correct shape of the medical instrument. The user can easily grasp the positional relationship between, for example, the Brockenbrough needle and the fossa ovalis, and perform IVR procedures.

Instead of performing the processes from step S643 to step S645, abnormal location information that is significantly different from other location information may be removed by performing clustering processing on multiple location information.

[Modification 8-1]
This modification relates to a catheter system 10 that, when no medical instrument is erroneously detected, performs three-dimensional display based on a medical instrument region detected from a catheter image 51. Explanation of parts common to the eighth embodiment will be omitted.

In step S663 of the subroutine described using Figure 34, the control unit 21 determines the thickness of the medical instrument based on the medical instrument region output, for example, from the trained model with hints 631 or the trained model without hints 632. However, for a catheter image 51 whose position information is determined to be incorrect, the thickness information is complemented based on the medical instrument regions of the previous and next catheter images 51.

This modified example makes it possible to provide a catheter system 10 that can appropriately display medical instruments whose thickness changes along the way, such as a medical instrument with a needle protruding from a sheath, in a three-dimensional image.

Ninth Embodiment
This embodiment relates to padding processing suitable for a trained model that processes RT-format catheter images 518 acquired using a radial scanning imaging catheter 40. Explanation of parts common to the first embodiment will be omitted.

Padding is the process of adding data around input data before performing convolution processing. In convolution processing immediately after the input layer that accepts image input, the input data is the input image. In convolution processing other than immediately after the input layer, the input data is the feature map extracted in the previous stage. In trained models that process image data, so-called zero padding is commonly performed, in which "0" data is added around the input data input to the convolution layer.

Figure 35 is an explanatory diagram illustrating padding processing in embodiment 9. The right end of Figure 35 is a schematic diagram of input data input to the convolutional layer. The convolutional layer is an example of, for example, the first convolutional layer included in the medical instrument trained model 611 and the second convolutional layer included in the angle trained model 612. The convolutional layer may be a convolutional layer included in any trained model used to process catheter images 51 captured using a radial scanning type image acquisition catheter 40.

The input data is in RT format, with the horizontal direction corresponding to the distance from the sensor 42 and the vertical direction corresponding to the scanning angle. A schematic diagram enlarging the upper right and lower left corners of the input data is shown in the center of Figure 35. Each frame corresponds to a pixel, and the numerical value within the frame corresponds to the pixel value.

The right side of Figure 35 is a schematic diagram of the data after padding processing in this embodiment has been performed. The numbers in italics indicate the data added by the padding processing. Data "0" is added to the left and right ends of the input data. The data indicated by "A" at the bottom of the data before padding processing is copied to the top of the input data. The data indicated by "B" at the top of the data before padding processing is copied to the bottom of the input data.

That is, at the right end of Figure 35, the same data as on the side with a larger scan angle is added to the outside of the side with a smaller scan angle, and the same data as on the side with a smaller scan angle is added to the outside of the side with a larger scan angle. In the following explanation, the padding process described using Figure 35 will be referred to as polar padding process.

In a radial scanning imaging catheter 40, the top and bottom of the RT catheter image 518 are approximately the same. For example, a single medical device or lesion may be separated into the top and bottom of the RT catheter image 518. Polar padding processing takes advantage of this characteristic.

According to this embodiment, a trained model can be generated that fully reflects the information at the top and bottom of RT format images.

Polar padding may be performed on all convolutional layers contained in a trained model, or on only some of the convolutional layers.

Figure 35 shows an example of padding in which one piece of data is added to each side of the input data, but the padding process may also add multiple pieces of data. The number of pieces of data added in the polar padding process is selected depending on the size and stride amount of the filter used in the convolution process.

[Modification 9-1]
36 is an explanatory diagram illustrating a modified polar padding process. The modified polar padding process is effective for the convolution layer at the stage where the RT format catheter image 518 is first processed.

The upper part of Figure 36 shows a schematic diagram of radial scanning performed while the sensor 42 is pulled to the right. A single RT catheter image 518, shown schematically in the lower left of Figure 36, is generated based on the scan line data acquired during one rotation of the sensor 42. The RT catheter image 518 is formed from top to bottom as the sensor 42 rotates.

The bottom right of Figure 36 shows a schematic diagram of the state after padding has been performed on the RT format catheter image 518. Data from the end of the RT format catheter image 518 one rotation before, indicated by hatching slanting downward to the left, is added to the top of the RT format catheter image 518. Data from the beginning of the RT format catheter image 518 one rotation after, indicated by hatching slanting downward to the right, is added to the bottom of the RT format catheter image 518. Data "0" is added to the left and right of the RT format catheter image 518.

In this variant, padding processing is performed based on actual scan line data, making it possible to generate a trained model that more accurately reflects the information above and below the RT format image.

[Embodiment 10]
37 is an explanatory diagram illustrating the configuration of a catheter system 10 according to a tenth embodiment. This embodiment relates to a configuration in which the catheter system 10 according to the tenth embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an image acquisition catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted.

The catheter control device 27 is an ultrasound diagnostic device for IVUS that controls the MDU 33, the sensor 42, and generates transverse and longitudinal images based on signals received from the sensor 42. The function and configuration of the catheter control device 27 are similar to those of conventional ultrasound diagnostic devices, so a detailed description will be omitted.

The catheter system 10 of this embodiment includes a computer 90. The computer 90 is equipped with a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, tablet, smartphone, or server computer.

The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read the program 97 stored in a semiconductor memory 98, such as a flash memory, implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

Program 97 is installed as a control program for computer 90, loaded into main memory device 22, and executed, thereby causing computer 90 to function as the information processing device 20 described above.

Computer 90 is a general-purpose personal computer, tablet, smartphone, mainframe computer, virtual machine running on a mainframe computer, cloud computing system, or quantum computer. Computer 90 may also be multiple personal computers performing distributed processing.

[Embodiment 11]
38 is a functional block diagram of an information processing device 20 according to embodiment 11. The information processing device 20 includes an image acquisition unit 81 and a first position information output unit 83. The image acquisition unit 81 acquires a catheter image 51 obtained by a radial scanning type imaging catheter 40.

When a catheter image 51 is input, the first position information output unit 83 inputs the acquired catheter image 51 into the medical instrument learned model 611, which outputs first position information regarding the position of the medical instrument contained in the catheter image 51, and outputs the first position information.

(Appendix A1)
an image acquisition unit that acquires a catheter image obtained by an image acquisition catheter inserted into the first cavity;
a first classified data output unit that, when the catheter image is input, outputs first classified data in which a non-living tissue region including a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted and a living tissue region are classified as different regions; and
The first classification trained model is generated using first training data that explicitly specifies at least the non-biological tissue region including the first lumen region and the second lumen region, and the biological tissue region.

(Appendix A2)
a lumen region extraction unit that extracts the first lumen region and the second lumen region from the non-biological tissue region in the first classified data;
The information processing device according to Appendix A1, further comprising: a first format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, and the biological tissue region to be distinguished from one another, and outputs the converted data.

(Appendix A3)
extracting a non-luminal region that is neither the first lumen region nor the second lumen region from the non-biological tissue region in the first classified data;
the information processing device according to Appendix A1 or A2, further comprising: a second format output unit that converts the first classification data into a format that allows the first lumen region, the second lumen region, the non-lumen region, and the biological tissue region to be distinguished from one another, and outputs the converted data.

(Appendix A4)
The information processing device described in Appendix A3, wherein when the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the first lumen region, the second lumen region, and the non-lumen region are classified as different regions.

(Appendix A5)
the imaging catheter is a radial scanning type tomographic imaging catheter,
the catheter image is an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle;
The information processing device according to any one of Appendix A1 to Appendix A4, wherein the first classification data is a classification result of each pixel in the RT format image.

(Appendix A6)
The first classification trained model is
It includes multiple convolutional layers,
The information processing device described in Appendix A5, wherein at least one of the plurality of convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle.

(Appendix A7)
The information processing device according to any one of Appendix A1 to Appendix A6, wherein the first classification trained model, when inputting a plurality of catheter images acquired in time series, outputs the first classification data that classifies the latest catheter image among the plurality of catheter images into the non-biological tissue region and the biological tissue region.

(Appendix A8)
The first classification trained model is
a memory unit for storing information about the catheter image previously input;
The information processing device according to Appendix A7, wherein the first classified data is output based on the information stored in the memory unit and the latest catheter image among the plurality of catheter images.

(Appendix A9)
The information processing device described in any one of Appendix A1 to Appendix A8, wherein, when the catheter image is input, the first classification trained model outputs the first classification data in which the biological tissue region, the non-biological tissue region, and a medical instrument region indicating a medical instrument inserted into the first cavity or the second cavity are classified as different regions.

(Appendix A10)
a second classified data acquisition unit that inputs the acquired catheter image into a second classification trained model that outputs second classified data in which the non-biological tissue region including the first lumen region and the biological tissue region are classified as different regions when the catheter image is input, and acquires the second classified data to be output;
a combined classification data output unit that outputs combined classification data obtained by combining the first classification data with the second classification data,
The information processing device described in any one of Appendix A1 to Appendix A9, wherein the second classification trained model is generated using second training data in which only the first lumen region is explicitly indicated among the non-biological tissue regions.

(Appendix A11)
The information processing device described in Appendix A10, wherein, when the catheter image is input, the second classification trained model outputs the second classification data in which the biological tissue region, the non-biological tissue region, and a medical instrument region indicating a medical instrument inserted into the first cavity or the second cavity are classified as different regions.

(Appendix A12)
The first classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region;
The second classification trained model further outputs a probability that each portion of the catheter image is the biological tissue region or the non-biological tissue region;
The information processing device described in Appendix A10 or A11, wherein the composite classification data output unit outputs composite classification data obtained by combining the first classification data with the second classification data based on a result of calculating a probability that each part of the catheter image is the biological tissue region or the non-biological tissue region.

(Appendix A13)
The information processing device according to any one of Appendix A1 to Appendix A12, wherein the image acquisition catheter is a three-dimensional scanning catheter that sequentially acquires a plurality of the catheter images along a longitudinal direction of the image acquisition catheter.

(Appendix A14)
The information processing device according to Appendix A13, further comprising a three-dimensional output unit that outputs a three-dimensional image generated based on the plurality of first classification data generated from the plurality of acquired catheter images.

(Appendix A15)
Acquiring a catheter image obtained by an image acquisition catheter inserted into the first cavity;
an information processing method for causing a computer to execute a process of inputting an acquired catheter image into a first classification trained model that outputs first classification data in which the acquired catheter image is input, the first classification trained model being generated using first training data that explicitly specifies a non-living tissue region including at least a first lumen region that is the inside of the first cavity and a second lumen region that is the inside of a second cavity into which the image acquisition catheter is not inserted, and a living tissue region, and that outputs first classification data in which the non-living tissue region and the living tissue region are classified as different regions when the catheter image is input.

(Appendix A16)
Acquiring a catheter image obtained by an image acquisition catheter inserted into the first cavity;
a program that causes a computer to execute a process of inputting an acquired catheter image into a first classification trained model that outputs first classification data that classifies the non-living tissue region and the living tissue region as different regions when the catheter image is input, the first classification trained model being generated using first training data that explicitly specifies a non-living tissue region that includes at least a first lumen region that is inside the first cavity and a second lumen region that is inside the second cavity into which the image acquisition catheter is not inserted, and a living tissue region, and outputting the first classification data.

(Appendix A17)
acquires a plurality of sets of training data in which catheter images obtained by an image acquisition catheter inserted into a first cavity and label data to which a plurality of labels are assigned, the labels including a biological tissue region label indicating that each portion of the catheter image is a biological tissue region, a first lumen region indicating that the region is the inside of the first cavity, a second lumen region indicating that the region is the inside of a second cavity into which the image acquisition catheter is not inserted, and a non-lunmen region including neither the first lumen region nor the second lumen region, and
A trained model generation method using the plurality of sets of training data to generate a trained model that, when the catheter image is input, outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image, with the catheter image as input and the label data as output.

(Appendix A18)
the non-biological tissue region labels of the plurality of sets of training data include a first lumen region label indicating the first lumen region, a second lumen region label indicating the second lumen region, and a non-lumen region label indicating the non-lumen region;
The trained model generating method described in Appendix A17, wherein the plurality of sets of training data are used to generate a trained model in which, when the catheter image is input, the catheter image is input and the label data is output, and the trained model outputs the biological tissue region label, the first lumen region label, the second lumen region label, and the non-lumen region label for each part of the catheter image.

(Appendix A19)
acquiring a plurality of sets of training data in which a catheter image obtained by an image acquisition catheter inserted into a first cavity and label data to which a plurality of labels are assigned, the labels including a biological tissue region label indicating a biological tissue region and a non-biological tissue region label including a first lumen region indicating the inside of the first cavity, the biological tissue region label being generated based on boundary data indicating the inner boundary of the first cavity in the catheter image, are associated and recorded;
A trained model generation method using the plurality of sets of training data to generate a trained model that, when the catheter image is input, outputs the biological tissue region label and the non-biological tissue region label for each part of the catheter image, with the catheter image as input and the label data as output.

(Appendix A20)
the catheter image is an RT format image in which scanning line data for one rotation obtained by the radial scanning type imaging catheter is arranged in parallel in order of scanning angle,
the trained model includes a plurality of convolutional layers;
At least one of the convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a smaller scanning angle is added to the outside of the side with a larger scanning angle. The method for generating a trained model described in any one of Appendix A17 to Appendix A19.

(Appendix B1)
an image acquisition unit that acquires a catheter image obtained by a radial scanning type image acquisition catheter;
and a first position information output unit that inputs the acquired catheter image into a medical instrument trained model that outputs first position information regarding a position of a medical instrument included in the catheter image when the catheter image is input, and outputs the first position information.

(Appendix B2)
The information processing device according to Appendix B1, wherein the first position information output unit outputs the first position information using a position of one pixel included in the catheter image.

(Appendix B3)
The first position information output unit includes: a first position information acquisition unit that acquires the time-series first position information corresponding to each of the plurality of catheter images obtained in time series; and an exclusion unit that excludes the first position information that does not satisfy a predetermined condition from the time-series first position information.
and a complementing unit that adds complementary information that satisfies a predetermined condition to the first position information in time series.

(Appendix B4)
The information processing device according to any one of Appendix B1 to Appendix B3, wherein, when a plurality of catheter images acquired in time series are input, the medical instrument trained model outputs the first position information regarding the latest catheter image among the plurality of catheter images.

(Appendix B5)
The medical instrument trained model is
a memory unit for storing information about the catheter image previously input;
The information processing device according to Appendix B4, wherein the first position information is output based on information stored in the memory unit and the latest catheter image among the plurality of catheter images.

(Appendix B6)
The medical instrument trained model is
receiving input of the catheter image in an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle;
a plurality of first convolution layers;
The information processing device according to any one of Appendix B1 to Appendix B5, wherein at least one of the plurality of first convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a smaller scanning angle is added to the outside of the side with a larger scanning angle.

(Appendix B7)
a scanning angle information acquisition unit that inputs the acquired catheter image into an angle-trained model that outputs scanning angle information related to the position of a medical instrument included in the catheter image when the catheter image is input, and acquires the output scanning angle information;
and a second position information output unit that outputs second position information regarding a position of a medical instrument included in the catheter image based on the first position information output from the medical instrument trained model and the scanning angle information output from the angle trained model.

(Appendix B8)
The angle-trained model is
receiving input of the catheter image in an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle;
a plurality of second convolutional layers;
At least one of the plurality of second convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle. The information processing device described in Appendix B7.

(Appendix B9)
The information processing device according to any one of Appendix B1 to Appendix B8, wherein the medical instrument trained model is generated using a plurality of sets of training data in which the catheter image and the position of the medical instrument included in the catheter image are recorded in association with each other.

(Appendix B10)
The training data is
displaying the catheter image obtained by the image acquisition catheter;
Accepting the position of the medical instrument included in the catheter image by a single click or tap on the catheter image;
The information processing device according to Appendix B9, wherein the catheter image is generated by a process of associating the catheter image with a position of a medical instrument and storing the associated catheter image.

(Appendix B11)
The training data is
inputting the catheter image into the medical instrument trained model;
The first position information output from the medical instrument trained model is superimposed on the input catheter image and displayed;
if a correction instruction regarding the position of the medical instrument included in the catheter image is not received, storing uncorrected data in which the catheter image is associated with the first position information as the training data;
The information processing device described in Appendix B9, wherein, when a correction instruction regarding the position of a medical instrument included in the catheter image is received, correction data that associates the catheter image with information regarding the position of the medical instrument based on the correction instruction is generated by a process of storing the correction data as the training data.

(Appendix B12)
acquiring a plurality of sets of training data in which catheter images acquired by an image acquisition catheter and first position information relating to the position of a medical instrument included in the catheter images are recorded in association with each other;
A trained model generation method for generating a trained model based on multiple sets of training data, which outputs first position information regarding the position of a medical instrument included in the catheter image when the catheter image is input.

(Appendix B13)
The method for generating a trained model described in Appendix B12, wherein the first position information is information regarding the position of one pixel included in the catheter image.

(Appendix B14)
Displaying a catheter image including a lumen obtained by the image acquisition catheter;
receiving, by a single click or tap on the catheter image, first position information relating to a position of the medical instrument inserted into the lumen, the first position information being included in the catheter image;
and storing training data that associates the catheter image with the first position information.

(Appendix B15)
The training data generation method according to Appendix B14, wherein the first position information is information relating to the position of one pixel included in the catheter image.

(Appendix B16)
When the first position information is received for the catheter image,
The training data generation method according to Appendix B14 or B15, wherein different catheter images obtained successively in time series are displayed.

(Appendix B17)
the imaging catheter is a radial scanning type tomographic imaging catheter,
The catheter image is displayed by displaying two images side by side: an RT format image in which a plurality of scanning line data acquired from the imaging catheter are arranged in parallel in order of scanning angle, and an XY format image in which data based on the scanning line data are arranged radially around the imaging catheter;
The training data generation method according to any one of Appendix B14 to Appendix B16, wherein the first position information is accepted from either the RT format image or the XY format image.

(Appendix B18)
inputting a catheter image obtained by an image acquisition catheter into a medical instrument trained model that outputs first position information regarding a position of a medical instrument included in the catheter image when the catheter image is input;
The first position information output from the medical instrument trained model is superimposed on the input catheter image and displayed;
if a correction instruction regarding the position of the medical instrument included in the catheter image is not received, storing uncorrected data associating the catheter image with the first position information as training data;
When a correction instruction regarding the position of a medical instrument included in the catheter image is received, correction data that associates the catheter image with information regarding the received position of the medical instrument is stored as the training data.

(Appendix B19)
The training data generation method according to Appendix B18, wherein the uncorrected data and the corrected data are data relating to the position of one pixel included in the catheter image.

(Appendix B20)
A plurality of the catheter images obtained in time series are input into the medical instrument trained model in order;
The training data generation method according to Appendix B18 or B19, wherein each of the output positions is displayed in order by being superimposed on the input catheter image.

(Appendix B21)
The training data generation method of any one of Appendix B18 to Appendix B20, wherein the position of the medical instrument is accepted by a single click or tap.

(Appendix B22)
the imaging catheter is a radial scanning type tomographic imaging catheter,
The catheter image is displayed by displaying two images side by side: an RT format image in which a plurality of scanning line data acquired from the imaging catheter are arranged in parallel in order of scanning angle, and an XY format image in which data based on the scanning line data are arranged radially around the imaging catheter;
The training data generation method according to any one of Appendix B18 to Appendix B21, wherein the position of the medical instrument is accepted from both the RT format image and the XY format image.

(Appendix C1)
an image acquisition unit for acquiring a catheter image including a lumen obtained by an image acquisition catheter;
a position information acquiring unit that acquires position information regarding a position of a medical instrument inserted into the lumen, the position information being included in the catheter image;
and a first data output unit that, when the catheter image and the position information are input, outputs first data in which each region of the catheter image is classified into at least three regions: a biological tissue region, a medical device region where the medical device is present, and a non-biological tissue region, and inputs the acquired catheter image and the acquired position information into a first trained model, and outputs the first data.

(Appendix C2)
The location information acquisition unit
The information processing device described in Appendix C1, wherein the acquired catheter image is input into a medical instrument trained model that outputs the position information included in the catheter image when the catheter image is input, and the position information is acquired from the medical instrument trained model.

(Appendix C3)
The information processing device according to Appendix C2, further comprising: a second data acquisition unit that inputs the acquired catheter image into a second model that outputs second data in which, when the catheter image is input without inputting the position information, each region of the catheter image is classified into at least three regions: a biological tissue region, a medical device region where the medical device is present, and a non-biological tissue region, and acquires the second data; and a composite data output unit that outputs composite data obtained by combining the first data and the second data.

(Appendix C4)
The composite data output unit
a first combined data output unit that outputs first combined data obtained by combining data related to the biological tissue-related regions classified as the biological tissue region and the non-biological tissue region out of the first data and the second data;
The information processing device according to Appendix C3, further comprising: a second combined data output unit configured to output second combined data obtained by combining data relating to the medical instrument region out of the first data and the second data.

(Appendix C5)
The second combined data output unit
When the position information can be acquired from the medical instrument trained model, the second combined data is output using data related to the medical instrument region included in the first data;
When the position information cannot be acquired from the medical instrument trained model, the information processing device described in Appendix C4 outputs the second combined data using data related to the medical instrument region included in the second data.

(Appendix C6)
The information processing device according to Appendix C4, wherein the composite data output unit outputs the second composite data obtained by combining data related to the medical device region based on weighting according to the reliability of the first data and the reliability of the second data.

(Appendix C7)
The information processing device according to Appendix C6, wherein the reliability is determined based on whether the position information can be acquired from the medical instrument trained model.

(Appendix C8)
The composite data output unit
If the location information can be acquired from the medical instrument trained model, the reliability of the first data is set to be higher than the reliability of the second data;
The information processing device according to Appendix C6, wherein, when the position information cannot be acquired from the medical instrument trained model, the reliability of the first data is set to be lower than the reliability of the second data.

(Appendix C9)
The information processing device according to any one of Appendix C1 to Appendix C8, wherein the image acquisition catheter is a three-dimensional scanning catheter that sequentially acquires a plurality of the catheter images along a longitudinal direction of the image acquisition catheter.

(Appendix C10)
Acquiring a catheter image including a lumen obtained by an image acquisition catheter;
acquiring position information relating to a position of a medical instrument inserted into the lumen, the position information being included in the catheter image;
an information processing method for causing a computer to execute a process of inputting the acquired catheter image and the acquired position information into a first trained model that outputs first data in which, when the catheter image and position information regarding the position of the medical instrument included in the catheter image are input, each region of the catheter image is classified into at least three regions: a biological tissue region, a medical instrument region where the medical instrument is present, and a non-biological tissue region, and outputting the first data.

(Appendix C11)
Acquiring a catheter image including a lumen obtained by an image acquisition catheter;
acquiring position information relating to a position of a medical instrument inserted into the lumen, the position information being included in the catheter image;
a first trained model that outputs first data in which, when the catheter image and positional information regarding the position of the medical instrument included in the catheter image are input, the first trained model classifies each region of the catheter image into at least three regions: a biological tissue region, a medical instrument region where the medical instrument is present, and a non-biological tissue region; and a program that causes a computer to execute a process of inputting the acquired catheter image and the acquired positional information into the first trained model and outputting the first data.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

REFERENCE SIGNS LIST 10 Catheter system 20 Information processing device 21 Control unit 22 Main memory device 23 Auxiliary memory device 24 Communication unit 25 Display unit 26 Input unit 27 Catheter control device 271 Catheter control unit 29 Reading unit 31 Display device 32 Input device 33 MDU
37 Diagnostic imaging device 40 Catheter for image acquisition 41 Probe section 42 Sensor 43 Shaft 44 Tip marker 45 Connector section 46 Guide wire lumen 51 Catheter image 518 RT format catheter image (catheter image)
519 XY format catheter image 52 Classification data (hint unclassified data, second data)
521 First classification data (label data)
522 Second Classification Data (Label Data)
526 Composite classification data 528 RT format classification data 529 XY format classification data 536 Composite data
541 First synthesis unit 542 Second synthesis unit 543 Third synthesis unit 55 Three-dimensional data 551 Living body three-dimensional data 552 Medical instrument three-dimensional data 561 Classification data with hints (first data)
611 Medical instrument trained model 612 Angle trained model 615 Position information synthesis unit 619 Position information model 62 Classification model (second model)
621 First classification trained model 622 Second classification trained model 626 Synthesized classification model 628 Classification data synthesis unit 629 Classification data conversion unit 631 Hint-included trained model (first trained model)
632 Hint-free trained model (second trained model)
639 X% hint trained model 65 Location information acquisition unit 66 Location classification analysis unit 71 Medical instrument location training data DB
72 Hint training data DB
781 Cursor 782 Control button area 81 Image acquisition unit 82 First classified data output unit 90 Computer 96 Portable recording medium 97 Program 98 Semiconductor memory

Claims (17)

an image acquisition unit that acquires a catheter image obtained by a radial scanning type image acquisition catheter;
a first position information output unit that inputs the acquired catheter image into a medical instrument trained model that outputs first position information regarding a position of a medical instrument included in the catheter image when the catheter image is input, and outputs the first position information;
The medical instrument trained model is
receiving input of the catheter image in an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle;
a plurality of first convolution layers;
At least one of the plurality of first convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle.
Information processing device. The information processing device according to claim 1 , wherein the first position information output unit outputs the first position information using a position of one pixel included in the catheter image. The first position information output unit
a first position information acquisition unit that acquires the first position information in time series corresponding to each of the plurality of catheter images acquired in time series; and an exclusion unit that excludes the first position information that does not satisfy a predetermined condition from the first position information in time series.
The information processing device according to claim 1 or 2, further comprising: a complementing unit that adds complementary information that satisfies a predetermined condition to the first time-series position information. The medical instrument trained model is
a memory unit for storing information about the catheter image previously input;
The first position information is output based on the information stored in the memory unit and the latest catheter image among the plurality of catheter images.
4. The information processing device according to claim 1 . The medical instrument trained model is
a recursive input unit that inputs an output for the catheter image that has been input in the past together with the latest catheter image;
outputting the first position information based on the latest catheter image and the input from the recursive input unit;
The information processing device according to claim 4 . The padding process is performed in a convolutional layer that processes the catheter image first among the plurality of first convolutional layers.
6. The information processing device according to claim 1. a scanning angle information acquisition unit that inputs the acquired catheter image into an angle-trained model that outputs scanning angle information related to the position of a medical instrument included in the catheter image when the catheter image is input, and acquires the output scanning angle information;
7. The information processing device according to claim 1, further comprising: a second position information output unit that outputs second position information regarding a position of a medical instrument included in the catheter image based on the first position information output from the medical instrument trained model and the scanning angle information output from the angle trained model. The angle-trained model is
receiving input of the catheter image in an RT format image in which a plurality of scanning line data acquired from the image acquisition catheter are arranged in parallel in order of scanning angle;
a plurality of second convolutional layers;
8. The information processing device according to claim 7, wherein at least one of the second convolutional layers is trained by performing padding processing in which the same data as on the side with a larger scanning angle is added to the outside of the side with a smaller scanning angle, and the same data as on the side with a larger scanning angle is added to the outside of the side with a larger scanning angle. The information processing device according to claim 1 , wherein the medical instrument trained model is generated using a plurality of sets of training data in which the catheter images and positions of medical instruments included in the catheter images are recorded in association with each other. The training data is
displaying the catheter image obtained by the image acquisition catheter;
Accepting the position of the medical instrument included in the catheter image by a single click or tap on the catheter image;
The information processing device according to claim 9 , wherein the catheter image is generated by a process of storing the catheter image in association with the position of the medical instrument. The training data is
inputting the catheter image into the medical instrument trained model;
The first position information output from the medical instrument trained model is superimposed on the input catheter image and displayed;
if a correction instruction regarding the position of the medical instrument included in the catheter image is not received, storing uncorrected data in which the catheter image is associated with the first position information as the training data;
The information processing device according to claim 9, wherein, when a correction instruction regarding the position of a medical instrument included in the catheter image is received, correction data that associates the catheter image with information regarding the position of the medical instrument based on the correction instruction is generated by a process of storing the correction data as the training data. acquiring a plurality of sets of training data in which catheter images in an RT format image in which a plurality of scanning line data obtained by an image acquisition catheter are arranged in parallel in order of scanning angle and first position information relating to the position of a medical instrument included in the catheter images are associated and recorded;
providing a model including a plurality of first convolutional layers;
a trained model generating method for generating a trained model that, when the catheter image is input , outputs first position information regarding the position of a medical instrument included in the catheter image while performing padding processing in at least one of the plurality of first convolutional layers, based on a plurality of sets of the training data, to add the same data as on the side with a larger scanning angle to the outside of the side with a smaller scanning angle and to add the same data as on the side with a smaller scanning angle to the outside of the side with a larger scanning angle. The method for generating a trained model according to claim 12 , wherein the first position information is information relating to the position of one pixel included in the catheter image. The padding process is performed in a convolutional layer that processes the catheter image first among the plurality of first convolutional layers.
The method for generating a trained model according to claim 12. an image acquisition unit that acquires a catheter image obtained by a radial scanning type image acquisition catheter;
a first position information output unit that inputs the acquired catheter image into a medical instrument trained model that outputs first position information regarding a position of a medical instrument included in the catheter image when the catheter image is input, and outputs the first position information ;
The medical instrument trained model is
a memory unit for storing information about the catheter image previously input;
The first position information is output based on the information stored in the memory unit and the latest catheter image.
Information processing device. The medical instrument trained model is
a recursive input unit that inputs an output for the catheter image that has been input in the past together with the latest catheter image;
outputting the first position information based on the latest catheter image and the input from the recursive input unit;
The information processing device according to claim 15. acquiring a plurality of sets of training data in which a plurality of catheter images photographed in time series by an image acquisition catheter are recorded in association with first position information relating to the positions of the medical instrument included in the plurality of catheter images;
Prepare a model that accepts time-series input,
A method for generating a trained model, which generates a trained model by adjusting parameters of the model based on multiple sets of the training data so that when multiple catheter images are input to the input layer, first position information regarding the position of a medical instrument associated with the last catheter image in chronological order is output from the output layer.