JP7747865B2 - Program, information processing method and information processing device - Google Patents

Description

The present invention relates to a program, an information processing method, and an information processing device .

A catheter system is used in which a diagnostic imaging catheter is inserted into a hollow organ such as a blood vessel to capture cross-sectional images (Patent Document 1).

International Publication No. 2017/164071

Information gaps may appear in cross-sectional images due to the influence of guidewires, stents, or heavily calcified areas. Experienced doctors and other specialists perform catheter treatments while filling in the missing information through inference. For this reason, extensive training is required to use catheter systems.

One aspect of the present invention is to provide a program or the like that makes it easy to use a catheter system.

The program acquires a catheter image generated using an imaging diagnostic catheter inserted into a tubular organ, acquires complementary information that complements missing information in the acquired catheter image, determines a display mode for the complementary information based on low-reliability areas of the acquired complementary information that have a lower reliability than other areas, and causes a computer to execute a process of displaying the catheter image and the complementary information in the display mode .

In one aspect, a program or the like can be provided that makes it easy to use the catheter system.

FIG. 1 is an explanatory diagram illustrating an overview of a catheter system. FIG. 1 is an explanatory diagram illustrating an overview of a catheter for diagnostic imaging. FIG. 1 is an explanatory diagram illustrating the configuration of a catheter system. 10 is an example of a screen displayed by the catheter system. 10 is an example of a screen displayed by the catheter system. 10 is an example of a screen displayed by the catheter system. 10 is an example of a screen displayed by the catheter system. 10 is a flowchart illustrating the flow of processing of a program. 10 is an example of a screen displayed by the catheter system of the second embodiment. 10 is a flowchart illustrating the flow of processing of a program according to a second embodiment. FIG. 2 is an explanatory diagram illustrating the configuration of a learning model. 10 is an example of a screen displayed by the catheter system of the third embodiment. 10 is an example of a screen displayed by the catheter system of the third embodiment. 11 is a flowchart illustrating the flow of processing of a program according to a third embodiment. FIG. 10 is an explanatory diagram illustrating the configuration of a learning model according to a fourth embodiment. An explanatory diagram explaining how to use the learning model in embodiment 4. 13 is an example of a screen displayed by the catheter system of the fourth embodiment. 13 is an example of a screen displayed by the catheter system of the fourth embodiment. 13 is an example of a screen displayed by the catheter system of the fourth embodiment. 13 is an example of a screen displayed by the catheter system of the fourth embodiment. 13 is an example of a screen displayed by the catheter system of the fourth embodiment. 10 is a flowchart illustrating the flow of processing of a program according to a fourth embodiment. An explanatory diagram explaining a method for generating a screen to be displayed by the catheter system of embodiment 5. 13 is a flowchart illustrating the flow of processing of a program according to a fifth embodiment. An explanatory diagram explaining a method for generating a screen displayed by the catheter system of embodiment 6. 13 is a flowchart illustrating the flow of processing of a program according to a sixth embodiment. FIG. 10 is an explanatory diagram illustrating the record layout of a training data DB. 13 is a flowchart illustrating the flow of processing of a program according to a seventh embodiment. FIG. 1 is an explanatory diagram illustrating an overview of a method for generating a learning model. 13 is a flowchart illustrating the flow of processing of a program according to an eighth embodiment. FIG. 13 is an explanatory diagram illustrating a method for generating an image displayed by the catheter system of embodiment 9. FIG. 20 is an explanatory diagram illustrating a method for generating a complementary line according to a ninth embodiment. FIG. 10 is an explanatory diagram illustrating a method for generating a complementary line in mode 2. 13 is a flowchart illustrating the processing flow of a program according to a ninth embodiment. 10 is a flowchart illustrating a process flow of a subroutine for generating complementary lines. 13 is an example of a screen displayed by the catheter system of embodiment 9. 13 is an example of a screen displayed by the catheter system of embodiment 9. 13 is an example of a screen displayed by the catheter system of embodiment 9. 13 is an example of a screen displayed by the catheter system of embodiment 9. 13 is an example of a screen displayed by the catheter system of embodiment 9. 13 is an example of a screen displayed by the catheter system of embodiment 9. FIG. 23 is a functional block diagram of an information processing system according to a tenth embodiment. FIG. 20 is an explanatory diagram illustrating the configuration of a catheter system according to an eleventh embodiment.

[First Embodiment]
FIG. 1 is an explanatory diagram illustrating an overview of a catheter system 10. The catheter system 10 includes a diagnostic imaging catheter 40, an MDU (Motor Driving Unit) 33, and an information processing device 20. The diagnostic imaging catheter 40 is connected to the information processing device 20 via the MDU 33. A display device 31 and an input device 32 are connected to the information processing device 20. The input device 32 is an input device such as a keyboard, a mouse, a trackball, or a microphone. The display device 31 and the input device 32 may be stacked together to form a touch panel. The input device 32 and the information processing device 20 may also be configured as an integrated unit.

Figure 2 is an explanatory diagram outlining the diagnostic imaging catheter 40. Figure 2 shows an example of a diagnostic imaging catheter 40 for IVUS (Intravascular Ultrasound), which is used to generate ultrasound tomographic images from inside blood vessels. The ultrasound tomographic image is an example of a catheter image generated using the diagnostic imaging catheter 40. The IVUS catheter is an example of a catheter for generating tomographic images.

The diagnostic imaging catheter 40 has a probe portion 41 and a connector portion 45 located at the end of the probe portion 41. The probe portion 41 is connected to the MDU 33 via the connector portion 45. In the following description, the side of the diagnostic imaging catheter 40 farther from the connector portion 45 will be referred to as the tip side.

A shaft 43 is inserted inside the probe unit 41. A sensor 42 is connected to the tip of the shaft 43. The shaft 43 and sensor 42 can rotate and move forward and backward inside the probe unit 41.

The sensor 42 is an ultrasonic transducer that transmits and receives ultrasonic waves. An annular tip marker 44 is fixed near the tip of the probe section 41. The tip marker 44 is made of a material that is opaque to X-rays, such as metal.

The diagnostic imaging catheter 40 may be a catheter for generating optical tomographic images, such as for OCT (Optical Coherence Tomography) or OFDI (Optical Frequency Domain Imaging), which generate optical tomographic images using near-infrared light. Optical tomographic images are an example of catheter images generated using the diagnostic imaging catheter 40. The sensor 42 of these diagnostic imaging catheters 40 is a transceiver that emits near-infrared light and receives reflected light. A catheter for generating optical tomographic images is an example of a catheter for generating tomographic images.

The diagnostic imaging catheter 40 may have both an ultrasound transducer and a transceiver unit for OCT or OFDI as sensors 42. The diagnostic imaging catheter 40 may have a total of three sensors 42: an ultrasound transducer, a transceiver unit for OCT, and a transceiver unit for OFDI.

The diagnostic imaging catheter 40 is not limited to a mechanical scanning type that mechanically rotates and moves back and forth. It may also be an electronic radial scanning type diagnostic imaging catheter 40 that uses a sensor 42 with multiple ultrasonic transducers arranged in a ring shape.

The diagnostic imaging catheter 40 may have a so-called linear scanning sensor 42 in which multiple ultrasound transducers are arranged in a row along the longitudinal direction. The diagnostic imaging catheter 40 may also have a so-called two-dimensional array sensor 42 in which multiple ultrasound transducers are arranged in a matrix.

The tubular organs into which the diagnostic imaging catheter 40 is inserted and used include, for example, blood vessels, the pancreatic duct, the bile duct, or the bronchi. In the following explanation, the diagnostic imaging catheter 40 will be described as an example of a mechanically scanned IVUS catheter, as shown in Figure 2.

Using the diagnostic imaging catheter 40, it is possible to generate tomographic images that include not only the walls of hollow organs such as blood vessel walls, but also reflectors present inside hollow organs, such as red blood cells, and organs present outside hollow organs, such as the epicardium and heart.

Figure 3 is an explanatory diagram illustrating the configuration of the catheter system 10. As described above, the catheter system 10 includes an information processing device 20, an MDU 33, and an imaging diagnostic catheter 40. The information processing device 20 includes a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a catheter control unit 271, and a bus.

The control unit 21 is an arithmetic and control device that executes the program of this embodiment. The control unit 21 uses one or more central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), multi-core CPUs, etc. The control unit 21 is connected to each hardware component that constitutes the information processing device 20 via a bus.

The main memory device 22 is a storage device such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), or flash memory. The main memory device 22 temporarily stores information required during processing performed by the control unit 21 and programs currently being executed by the control unit 21.

The auxiliary storage device 23 is a storage device such as SRAM, flash memory, a hard disk, or magnetic tape. The auxiliary storage device 23 stores programs to be executed by the control unit 21 and various data required for executing the programs. The communication unit 24 is an interface that performs communication between the information processing device 20 and the network.

The display unit 25 is an interface that connects the display device 31 to the bus. The input unit 26 is an interface that connects the input device 32 to the bus. The catheter control unit 271 controls the MDU 33, controls the sensor 42, and generates images based on signals received from the sensor 42.

The MDU 33 rotates the sensor 42 and shaft 43 inside the probe section 41. The catheter control section 271 generates one image for each rotation of the sensor 42. The generated image is a cross-sectional image centered on the probe section 41 and approximately perpendicular to the probe section 41.

The MDU 33 can also move the sensor 42 and shaft 43 back and forth while rotating them inside the probe section 41. By performing a pullback operation, which rotates the sensor 42 while pulling it toward the MDU 33 at a constant speed, the catheter control section 271 continuously generates multiple cross-sectional images approximately perpendicular to the probe section 41 at a predetermined interval.

The function and configuration of the catheter control unit 271 are similar to those of conventional ultrasound diagnostic devices, so detailed explanation will be omitted. Note that the control unit 21 may also implement the functions of the catheter control unit 271.

The information processing device 20 is connected to various imaging diagnostic devices 37, such as an X-ray angiography device, an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, a PET (Positron Emission Tomography) device, or an ultrasound diagnostic device, via an HIS (Hospital Information System) or the like.

The information processing device 20 in this embodiment is a dedicated ultrasound diagnostic device, or a PC, tablet, smartphone, or other device that has the functionality of an ultrasound diagnostic device.

Figures 4 to 7 are examples of screens displayed by the catheter system 10. The screen shown in Figure 4 includes a first image field 51, a stop button 581, a select button 582, and an complement button 583. Figure 4 shows a state in which the stop button 581 is not selected. The select button 582 and complement button 583 are set to an unselectable state. A tomographic image obtained using the diagnostic imaging catheter 40 is displayed in real time in the first image field 51. In the following description, an image displayed in real time will be referred to as a real-time image.

In the real-time image shown in Figure 4, a small crescent-shaped guidewire image 472 is displayed at the 5 o'clock position relative to a circular catheter image 471 displayed in the center of the first image field 51. Because the ultrasound emitted from the sensor 42 does not reach the back surface of the guidewire, which is a strong reflector, a roughly fan-shaped information gap 473 is formed outside the guidewire image 472. This type of information gap 473 is called an acoustic shadow. Note that the information gap 473 is not limited to acoustic shadows. Information gaps can also be formed by artifacts such as ringdown caused by vibration of the ultrasonic transducer, multiple echoes formed by strong reflectors such as the guidewire, and external noise.

For example, in areas where a stent has been placed, information gaps 473 are formed outside the image showing the wires that make up the mesh of the stent, which is a strong reflector (see Figure 23). Similarly, information gaps 473 are also formed outside highly calcified areas and high-attenuation plaque.

A doctor or other expert skilled in handling the diagnostic imaging catheter 40 can distinguish between information-missing areas 473 formed by the influence of strong reflectors or high attenuation areas and areas with truly low ultrasound reflectivity, and can infer and supplement the information missing due to the information-missing areas 473.

However, when it comes to the parts that are completed through guesswork, there may be variations in judgment depending on the person. For example, if there are variations in judgment between the doctor in charge of operating the diagnostic imaging catheter 40 and the medical staff, such as nurses and technicians, who assist the doctor with the procedure, it may be difficult for the doctor's instructions to be communicated promptly, and in some cases misunderstandings may occur.

Figure 5 shows an example of a screen displayed when the control unit 21 accepts selection of the stop button 581. The selection button 582 is set to a selectable state. The second image field 52 is displayed in the upper right corner of the screen. A real-time image is displayed in the second image field 52. The first image field 51 displays a still image that was displayed at the time selection of the stop button 581 was accepted.

In the following description, the image displayed at the time selection of the stop button 581 is accepted is referred to as the paused image. Note that the control unit 21 may also accept selection of the stop button 581 by voice recognition or operation of a foot switch (not shown).

Figure 6 shows an example of a screen displayed when the control unit 21 accepts selection of the selection button 582. A cursor 68 is displayed in the first image field 51. The user operates the cursor 68 via the input device 32 to place a designated point mark 571, for example, on the edge of a hollow organ.

Figure 7 shows an example of a screen displayed when the control unit 21 accepts selection of the completion button 583. The control unit 21 displays a circular, i.e., closed curved, completion line 572 that completes the edge of the tubular organ based on the designated point marks 571 placed by the user. The completion line 572 is generated by connecting the designated point marks 571 placed by the user with a curve, such as a spline curve. The completion line 572 is an example of completion information that completes the missing information portion 473.

Figure 8 is a flowchart explaining the flow of program processing. The program shown in Figure 8 is launched when the user selects the selection button 582.

The control unit 21 accepts the user's specification of the position of the designated point mark 571 (step S501). The control unit 21 determines whether or not the input of the designated point mark 571 has ended (step S502). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has ended when the user instructs the end of input. The control unit 21 may also determine that the input has ended when a predetermined number of designated point marks 571 have been accepted.

If it is determined that the process has not ended (NO in step S502), the control unit 21 returns to step S501. If it is determined that the process has ended (YES in step S502), the control unit 21 generates a circular complementary line 572 that connects the specified point marks 571 received in step S501 (step S503).

The control unit 21 superimposes the generated complementary line 572 on the paused image and displays the screen described using Figure 7 (step S504). The control unit 21 then ends the processing.

The control unit 21 may calculate parameters such as area, maximum diameter, and minimum diameter based on the completed interpolated line 572 and display them on the screen described using FIG. 7. The control unit 21 may also accept specification of the type of parameter to be calculated or the calculation formula.

This embodiment provides a catheter system 10 that easily embodies and displays the completion line 572 that the user imagines in their mind based on the tomographic image. Therefore, it is possible to provide a catheter system 10 that is easy for the user to use.

By displaying the completion state imagined by the user in a manner that can be recognized by other staff using the completion line 572, the catheter system 10 can be provided to facilitate communication between related medical staff.

In this embodiment, the user places the designated point mark 571 where the image actually exists, making it less likely that the user will have any confusion. This makes it possible to provide a catheter system 10 that allows the user to quickly create the completion line 572.

This section explains the procedure using an example in which a stent is placed in a blood vessel. A doctor inserts a diagnostic imaging catheter 40 into the blood vessel where the stent is to be placed and determines the state of stenosis in the blood vessel based on the tomographic image. To determine which stent to use, the doctor or medical staff under the doctor's instructions operate the input device 32 to measure the area and inner diameter of the blood vessel lumen in the tomographic image. The doctor then determines which stent to place based on the measurement results.

The control unit 21 can calculate the area, inner diameter, etc. of the blood vessel lumen based on the aforementioned interpolated line 572. In other words, a catheter system 10 can be provided that calculates the area, inner diameter, etc. of the blood vessel lumen. The control unit 21 may also display information about a recommended stent based on the calculated area, inner diameter, etc. Note that since the calculation of the area and inner diameter of the portion surrounded by a closed curve has been performed conventionally, detailed explanation will be omitted.

After the stent is placed, the doctor inserts an imaging diagnostic catheter 40 into the same blood vessel and uses cross-sectional images to confirm that no gaps exist between the blood vessel wall and the placed stent. If a gap exists between the blood vessel wall and the stent, the doctor will perform treatment such as re-expanding the stent.

The tomographic image after stent placement contains numerous information gaps 473 caused by the stent wires. Therefore, the external elastic lamina is displayed as a broken line interrupted by the information gaps 473. By displaying the aforementioned complementary lines 572, not only the doctor who set the position of the designated point mark 571, but also other medical staff can easily understand the status of the stent placement.

Part or all of the program may be executed on a large computer connected via a network, a virtual machine running on a large computer, or a cloud computing system. Part or all of the program may be executed on multiple personal computers performing distributed processing.

Instead of real-time images, video or still images stored in the auxiliary storage device 23 or the like may be used. This provides a catheter system 10 that can be used for recording medical records after a case is completed. In this case, the information processing device 20 may be a general-purpose PC, tablet, smartphone, or the like that does not have the functionality to connect the MDU 33 and the diagnostic imaging catheter 40.

[Modification]
In step S501, if the control unit 21 receives, for example, five or more designated point marks 571, the control unit 21 generates (step S503) and displays (step S504) complementary lines 572. Thereafter, the control unit 21 returns to step S501 and receives further designation of designated point marks 571.

The user places an additional designated point mark 571 in a location where they actually want to draw the completion line 572, but where the location is far from the displayed completion line 572. In this way, the user can create the desired completion line 572 using a small number of designated point marks 571.

[Embodiment 2]
This embodiment relates to a catheter system 10 that automatically displays a candidate point mark 573 at a location similar to a designated point mark 571 designated by the user. Explanation of parts common to the first embodiment will be omitted.

Figure 9 shows an example of a screen displayed by the catheter system 10 of embodiment 2. As explained using Figure 6, the user operates the cursor 68 via the input device 32 to specify the position of the specified point mark 571.

The control unit 21 detects an area similar to the periphery of the designated point mark 571 for which designation has been accepted. Specifically, the control unit 21 extracts a template area 574 of a predetermined number of pixels centered on the designated point mark 571. In Figure 9, the template area 574 is indicated by a virtual line.

The control unit 21 uses a template matching technique to detect an area in the paused image that is similar to the template area 574. Template matching techniques have been used for a long time, so a detailed explanation will be omitted. It is desirable for the control unit 21 to exclude areas near locations where the designated point mark 571 has already been placed from the template matching target.

When a similar area is detected, the control unit 21 displays a candidate point mark 573 in the center of the area. The control unit 21 may display multiple candidate point marks 573 simultaneously. The control unit 21 may also display only the candidate point mark 573 that has the highest degree of similarity during template matching. The candidate point mark 573 is a mark with a different form from the designated point mark 571, allowing the user to easily distinguish between the two.

The user decides whether to approve the displayed candidate point mark 573 and operates the input device 32. For example, the user approves by clicking on the candidate point mark 573, or rejects by dragging and dropping it outside the first image field 51. The user may also input approval or rejection instructions by voice input.

The control unit 21 changes the candidate point mark 573 approved by the user to a specified point mark 571. The control unit 21 deletes the candidate point mark 573 rejected by the user. Note that the user may reserve judgment on the displayed candidate point mark 573 and specify the position of the next candidate point mark 573.

For example, if the user places a designated point mark 571 on the external elastic lamina, the control unit 21 extracts a point indicating the external elastic lamina using template matching and displays a candidate point mark 573. Similarly, if the user places a designated point mark 571 on the inner surface of the lumen wall, the control unit 21 extracts a point indicating the inner surface of the lumen wall and displays a candidate point mark 573.

Figure 10 is a flowchart illustrating the processing flow of the program in embodiment 2. The control unit 21 accepts the user's position specification of the designated point mark 571 (step S511). The control unit 21 determines whether the input of the designated point mark 571 has ended (step S512). For example, the control unit 21 determines that the acceptance of the designated point mark 571 has ended when the user instructs the end of input. The control unit 21 may also determine that the input has ended when a predetermined number of designated point marks 571 have been accepted.

If it is determined that processing has not ended (NO in step S512), the control unit 21 extracts a template area 574 from the paused image, centered on the position accepted in step S511 (step S513). The control unit 21 then executes template matching processing to detect an area in the paused image that is similar to the template area 574 (step S514). Template matching is a conventional technique, so detailed explanation will be omitted.

The control unit 21 determines whether or not the detection of an area similar to the template area 574 was successful (step S515). For example, the control unit 21 determines that the detection was successful if the similarity between the template area 574 and the area detected in step S514 exceeds a predetermined threshold.

The similarity can be evaluated, for example, by the sum of squared differences (SSD), sum of absolute differences (SAD), or normalized cross-correlation (NCC) of pixel values of pixels at the same position in two regions. Note that the above methods are merely examples. Methods for evaluating similarity are not limited to these.

If it is determined that the matching was not successful (NO in step S515), the control unit 21 returns to step S511. If it is determined that the matching was successful (YES in step S515), the control unit 21 displays a candidate point mark 573 in the center of the area that matches the template area 574 (step S516).

The control unit 21 displays a candidate point mark 573, for example, at the center of the area with the highest similarity. The control unit 21 may also display a candidate point mark 573 at the center of each of multiple areas determined to satisfy the conditions in step S515. The control unit 21 returns to step S511.

Although not shown in the flowchart, the user can accept or reject the displayed candidate point mark 573 at any time by operating the cursor 68 or by voice input, etc. The control unit 21 changes the accepted candidate point mark 573 to the designated point mark 571 and deletes the rejected candidate point mark 573.

If it is determined that the process has ended (YES in step S512), the control unit 21 generates a circular complementary line 572 connecting the designated point marks 571 received in step S501 (step S517). The control unit 21 superimposes the generated complementary line 572 on the paused image and displays the screen described using FIG. 7 (step S518). The control unit 21 ends the process.

This embodiment provides a catheter system 10 that automatically displays a candidate point mark 573 at a position similar to the position where the user specified the designated point mark 571. This provides a catheter system 10 that can reduce the user's burden in inputting the designated point mark 571.

[Third embodiment]
This embodiment relates to a catheter system 10 that automatically generates a complementary line 572 in real time. Explanation of parts common to the first embodiment will be omitted.

Figure 11 is an explanatory diagram illustrating the configuration of the learning model 61. The learning model 61 is a model that accepts an input image and outputs a prediction regarding an output image showing the interpolated line 572. The learning model 61 is generated by machine learning. The method for generating the learning model 61 will be described later.

The learning model 61 is created for each complementary line to be generated, such as a model that outputs a complementary line 572 corresponding to the external elastic lamina, a model that outputs a complementary line 572 corresponding to the inner surface of the lumen wall, a model that outputs a complementary line 572 corresponding to the periphery of a calcified area, and a model that outputs a complementary line 572 corresponding to the periphery of a plaque. A learning model 61 may also be created for each combination of the site into which the diagnostic imaging catheter 40 is inserted, such as the coronary artery, lower limb aorta, bile duct, pancreatic duct, and bronchi, and the complementary line to be generated.

Each learning model 61 is stored in the auxiliary storage device 23. The learning model 61 may also be stored in an external large-capacity storage device connected to the information processing device 20. The control unit 21 may acquire the learning model 61 stored in a server or the like via a network whenever necessary.

The learning model 61 may be a model that accepts an input image and outputs an output image in which the interpolated line 572 is superimposed on the input image. The input image input to the learning model 61 may be a longitudinal cross-sectional image. The learning model 61 may be a model that accepts an input image and outputs a group of coordinates through which the interpolated line 572 passes.

Figures 12 and 13 are examples of screens displayed by the catheter system 10 of embodiment 3. Figure 12 shows an example of a screen displayed by the control unit 21 on the display device 31 before creating the completion line 572. The screen shown in Figure 12 includes a first image field 51, a target selection button 591, and a start button 584.

The first image field 51 displays a real-time image obtained using the diagnostic imaging catheter 40. The target selection button 591 is a pull-down menu button, and the user selects the area for which they want to display the completion line 572. In Figure 12, "EEM," or the external elastic lamina, has been selected.

The user observes the real-time image displayed in the first image field 51 and determines the area where the complementary line 572 will be displayed. The user operates the target selection button 591 to select the area where the complementary line 572 will be displayed, and then selects the start button 584.

Figure 13 shows an example of a screen that the control unit 21 displays in the first image field 51 after accepting the user's selection of the start button 584. The screen shown in Figure 13 includes the first image field 51, a target selection button 591, and an end button 585. A real-time image with a complementary line 572 corresponding to "EEM" superimposed thereon is displayed in the first image field 51. To end the display of the complementary line 572, the user selects the end button 585.

The screen shown in FIG. 13 may display buttons or the like that allow changes to be made to the display mode of the completion line 572, such as the color and thickness. The screen shown in FIG. 13 may also display buttons or the like that allow the completion line 572 to be temporarily hidden.

Figure 14 is a flowchart explaining the processing flow of the program in embodiment 3. The control unit 21 acquires the part to be completed that the user has set using the target selection button 591 (step S521). The control unit 21 selects a learning model 61 that corresponds to the part to be completed acquired in step S521 (step S522). In subsequent processing, the control unit 21 uses the learning model 61 selected in step S522.

The control unit 21 acquires a real-time image from the catheter control unit 271 (step S523). The control unit 21 inputs the acquired real-time image into the learning model 61 and acquires an output image showing the interpolated line 572 (step S524). The control unit 21 displays the real-time image with the interpolated line 572 superimposed in the first image field 51 (step S525).

The control unit 21 determines whether to terminate the processing (step S526). For example, if the end button 585 is selected, or if the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines that the processing is to terminate.

If it is determined not to end the processing (NO in step S526), the control unit 21 returns to step S523. If it is determined to end the processing (YES in step S526), the control unit 21 ends the processing.

If the frame rate of the real-time image is high and the processing of step S524 cannot be completed in time, the control unit 21 may execute step S524, for example, once every two or three frames. The control unit 21 may also determine the frame for which the processing of step S524 is performed to be synchronized with the electrocardiogram.

This embodiment provides a catheter system 10 that automatically displays a completion line 572 on a real-time image. This embodiment also provides a catheter system 10 that allows the user to specify the area indicated by the completion line 572.

[Fourth embodiment]
This embodiment relates to a catheter system 10 that displays an image in which an information-missing portion 473 has been complemented. Explanation of parts common to the first embodiment will be omitted.

Figure 15 is an explanatory diagram illustrating the configuration of a learning model 61 in embodiment 4. The learning model shown in Figure 15 is a model that, when an input image having an information-missing portion 473 is input, outputs a prediction regarding an output image in which the information-missing portion 473 has been complemented. The output image in Figure 15 is an example of a complemented image generated to complement the information-missing portion 473.

The learning model 61 includes a missing area extraction model 611, a complement model 612, a cutout unit 613, and a synthesis unit 614. The missing area extraction model 611 and the complement model 612 are neural network models with learnable parameters. The cutout unit 613 and the synthesis unit 614 are computing units that calculate the pixel values of each pixel that makes up the image.

The missing area extraction model 611 is a model that accepts an input image containing missing information portions 473 and generates a missing area image 483 by extracting the portion showing the missing information portions 473. The missing area extraction model 611 has the configuration of, for example, a Mask-RCNN (Region Convolutional Neural Network), which is a type of object detection model.

In Figure 15, the missing information portion 473 in the missing area image 483 is indicated by hatching. For example, the missing area image 483 is an image of the same size as the input image, in which the pixel value of the pixel corresponding to the missing information portion 473 is "1" and the pixel value of the pixels in the other areas is "0."

The interpolation model 612 is a model that accepts an input image including an information-missing portion 473 and generates an estimated image 486 without the information-missing portion 473. The cropping unit 613 generates a cropped image 487 by replacing the pixel values of pixels in the estimated image 486 that do not correspond to the information-missing portion 473 with pixel values of the background color. The interpolation model 612 has, for example, a configuration in which multiple convolutional layers are connected.

The synthesis unit 614 synthesizes the input image and the cropped image 487 to generate an output image. The output image is an image in which only the portion of the input image corresponding to the missing information portion 473 has been replaced with an image generated by the interpolation model 612. The cropped image 487 is an example of complementary information that complements the missing information portion 473.

The learning model 61 has been trained so that when an input image having information-missing portions 473 is input, it will output an output image that does not appear unnatural to an expert. The method for generating the learning model 61 will be described later.

Figure 16 is an explanatory diagram illustrating how to use the learning model 61 in embodiment 4. The control unit 21 inputs an input image to the learning model 61 and obtains the cropped image 487 output from the cropping unit 613.

Figures 17 to 21 are examples of screens displayed by the catheter system 10 of embodiment 4. The screen shown in Figure 17 includes a first image field 51, a no complement button 580, a complement button 583, a color button 586, a border button 587, and a shaded button 588. Either the no complement button 580 or the complement button 583 is always set to be selected.

Figure 17 shows the state when the No Completion button 580 is selected. The Completion button 583, Color button 586, and Border button 587 are set to an unselectable state. A real-time image including missing information portion 473 is displayed in the first image column 51. In Figure 17, missing information portion 473 has not been completed.

Figure 18 shows an example of a screen displayed when the control unit 21 accepts selection of the complement button 583. The no complement button 580 is deselected, and the color button 586, border button 587, and shaded button 588 are set to a selectable state. The second image field 52 is displayed in the upper right corner of the screen.

The first image field 51 displays an image that combines the real-time image with the cropped image 487 obtained from the learning model 61. The second image field 52 displays the real-time image. The user can determine the cross-sectional shape, cross-sectional area, inner diameter, etc. of the hollow organ based on the image in which the missing information portion 473 displayed in the first image field 51 has been filled in.

The control unit 21 can calculate the area and inner diameter of the blood vessel lumen based on an image without the missing information portion 473 displayed in the first image field 51. The control unit 21 may also display information about the recommended stent based on the calculated area and inner diameter.

If necessary, the user can observe the real-time image before completion displayed in the second image field 52. By comparing the first image field 51 and the second image field 52, the user can confirm which parts have been completed by the learning model 61.

Figure 19 shows an example of a screen displayed when the control unit 21 accepts selection of the border button 587. The control unit 21 combines the real-time image with a cropped image 487 with a border added to the edge, and displays it in the first image field 51. The user can easily distinguish between the portion complemented by the learning model 61 and the portion actually acquired by the imaging diagnostic catheter 40.

Figure 20 shows an example of a screen displayed when the control unit 21 accepts selection of the border button 587 and the shaded button 588. The control unit 21 combines the real-time image with a cropped image 487 that has borders added to its edges and is shaded, and displays the combined image in the first image field 51. This provides a catheter system 10 that prevents even an inexperienced user from mistaking the portion completed by the learning model 61 for the portion actually acquired by the diagnostic imaging catheter 40.

The user can select any combination of the coloring button 586, border button 587, and shading button 588. The coloring button 586, border button 587, and shading button 588 are examples of means for selecting the marking style for the cropped image 487. Marking methods are not limited to these. Any style of marking can be set.

Figure 21 shows an example of a screen displayed when the control unit 21 accepts selection of the coloring button 586. The control unit 21 combines the real-time image with a cropped image 487 in which hyperechoic areas are colored, and displays the combined image in the first image field 51. The user can easily distinguish between the areas complemented by the learning model 61 and the areas actually acquired by the diagnostic imaging catheter 40.

Figure 22 is a flowchart explaining the processing flow of the program in embodiment 4. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S531). The control unit 21 determines whether selection of the complement button 583 has been accepted (step S532). If it is determined that selection has not been accepted (NO in step S532), the control unit 21 displays the real-time image in the first image field 51 (step S534).

If it is determined that the selection has been accepted (YES in step S532), the control unit 21 inputs the acquired real-time image into the learning model 61 to acquire the cropped image 487 (step S533). The control unit 21 then determines whether the selection of marking for the cropped image 487 has been accepted via the coloring button 586, border button 587, and hatching button 588 (step S535).

If it is determined that a marking selection has not been accepted (NO in step S535), the control unit 21 combines the real-time image with the cropped image 487 (step S536). If it is determined that a marking selection has been accepted (YES in step S535), the control unit 21 combines the real-time image with the cropped image 487 with the specified marking (step S537).

After step S536 or step S537 is completed, the control unit 21 displays the synthesized image (step S538). After step S534 or step S538 is completed, the control unit 21 determines whether to end the processing (step S539). For example, if the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines to end the processing.

If it is determined that the process should not end (NO in step S539), the control unit 21 returns to step S531. If it is determined that the process should end (YES in step S539), the control unit 21 ends the process.

This embodiment provides a catheter system 10 that maintains the portion actually acquired using the diagnostic imaging catheter 40, while complementing and displaying the missing information portion 473 in a natural manner.

This embodiment provides a catheter system 10 that allows the user to identify the completed area using various marking methods.

Fifth Embodiment
This embodiment relates to a catheter system 10 that complements an information-missing portion 473 in a transverse cross-sectional image by using a transverse cross-sectional image of another frame. Description of parts common to the first embodiment will be omitted.

Figure 23 is an explanatory diagram illustrating a method for generating a screen displayed by the catheter system 10 in embodiment 5. This embodiment will be described using an example in which an imaging diagnostic catheter 40 is inserted into a site where a stent has already been placed and images are acquired.

The first and second tomographic images are transverse images acquired at slightly different positions in the longitudinal direction of the probe portion 41. There is no significant difference in the structure of the luminal organ itself between the two transverse images. The stent wires have created multiple radial information gaps 473. Because the stent is mesh-like, the positions of the information gaps 473 will differ if the slice plane of the transverse image is shifted in the longitudinal direction of the probe portion 41.

For example, the control unit 21 extracts the information-missing portions 473 from the second tomographic image. The control unit 21 cuts out the portions corresponding to the information-missing portions 473 from the first tomographic image and combines them with the second tomographic image. As a result, a combined image that does not include the information-missing portions 473 is generated.

Figure 24 is a flowchart explaining the processing flow of the program in embodiment 5. The control unit 21 acquires a first real-time image from the catheter control unit 271 and records it in the main storage device 22 or the auxiliary storage device 23 (step S551). The first real-time image is an example of a catheter image generated using the diagnostic imaging catheter 40.

The control unit 21 extracts information-missing portions 473 from the first real-time image (step S552). The information-missing portions 473 are extracted by template matching, using, for example, a fan-shaped hypoechoic region as a template. A fan-shaped region whose central end is hyperechoic and the remaining portions are hypoechoic may be used as the template. The information-missing portions 473 may also be extracted by a learning model using an object detection algorithm such as Mask-RCNN.

The control unit 21 acquires a second real-time image from the catheter control unit 271 and records it in the main memory device 22 or the auxiliary memory device 23 (step S553). Note that while this program is running, the user slightly advances and retreats the diagnostic imaging catheter 40 near the area to be observed. This program may also be executed during a pullback operation using the MDU 33. When executed during a pullback operation, the control unit 21 acquires the first real-time image, and then acquires the second real-time image one to several frames later. The second real-time image is an example of a second catheter image generated at a different time than the first catheter image.

The control unit 21 extracts the information missing portion 473 in the second real-time image (step S554). The control unit 21 determines whether the information missing portion 473 extracted in step S552 and the information missing portion 473 extracted in step S554 overlap (step S555). For example, the control unit 21 determines that the two information missing portions 473 overlap if the area of the overlapping portion exceeds a predetermined area.

If it is determined that there is overlap (YES in step S555), the control unit 21 returns to step S553. If it is determined that there is no overlap (NO in step S555), the control unit 21 combines the first real-time image and the second real-time image (step S556).

Specifically, for example, the control unit 21 cuts out the portion corresponding to the information missing portion 473 extracted in step S554 from the first real-time image and combines it with the second real-time image. The control unit 21 may also cut out the portion corresponding to the information missing portion 473 extracted in step S552 from the second real-time image and combine it with the first real-time image.

The control unit 21 displays the synthesized image, for example, in the first image field 51 of the screen described using Figures 17 to 21 (step S557). The control unit 21 then ends the processing.

According to this embodiment, the missing information area 473 is filled in using transverse images from other frames, making it possible to provide a catheter system 10 that is less susceptible to artifacts caused by image synthesis.

Sixth Embodiment
This embodiment relates to a catheter system 10 that complements an information missing portion 473 in an image by using an image of the same frame. Description of parts common to the first embodiment will be omitted.

Figure 25 is an explanatory diagram illustrating a method for generating a screen displayed by the catheter system 10 in embodiment 6. This embodiment will be described using an example of an image including an information-missing area 473 due to a guidewire.

For example, the control unit 21 extracts the information-missing portion 473 from the original image. The control unit 21 copies a paste area 485 of a shape corresponding to the information-missing portion 473 from a portion of the original image where no information is missing, and combines it with the information-missing portion 473. As a result, a composite image that does not include the information-missing portion 473 is generated.

Figure 26 is a flowchart explaining the processing flow of the program in embodiment 6. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S561). The control unit 21 extracts information-missing portions 473 from the real-time image (step S562).

The control unit 21 extracts candidates for the paste area 485 from the real-time image acquired in step S561 (step S563). The candidates for the paste area 485 are extracted by, for example, cutting out from the real-time image a portion corresponding to the shape obtained by rotating the information missing portion 473 around the center of the image as an axis.

The control unit 21 combines the real-time image acquired in step S561 with the candidate for paste area 485 extracted in step S563 (step S564). The control unit 21 extracts information missing portion 473 from the combined image combined in step S564 (step S565). The method for extracting information missing portion 473 in step S565 is the same as that in step S562.
The control unit 21 determines whether the composite image includes the missing information portion 473 (step S566). Specifically, if the missing information portion 473 having an area larger than a predetermined value is extracted, the control unit 21 determines that the missing information portion 473 is included in the composite image.

If it is determined that the missing information portion 473 is included (YES in step S566), the control unit 21 returns to step S563. If it is determined that the missing information portion 473 is not included (NO in step S566), the control unit 21 displays the image synthesized in step S564, for example, in the first image field 51 of the screen described using Figures 17 to 21 (step S567). The control unit 21 ends the processing.

In this embodiment, the missing information portion 473 is complemented using the attachment area 485 obtained from the transverse image of the same frame, thereby providing a catheter system 10 with little time lag associated with the complementation process.

Seventh Embodiment
This embodiment relates to a method for generating the learning model 61 of the third embodiment described with reference to Fig. 11. Explanation of parts common to the third embodiment will be omitted.

Figure 27 is an explanatory diagram illustrating the record layout of the training DB. The training data DB is a database that records the detection target, input image, and output image in association with each other, and is used to train the learning model 61 using machine learning. The training DB has a target field, an input image field, and an output image field.

The target field records the name of the target for which the interpolated line 572 is to be created. The input image field records the image acquired using the diagnostic imaging catheter 40. The output image field records an image of the interpolated line representing the target recorded in the target field.

The training DB contains a large number of combinations of subject names, input images generated using the diagnostic imaging catheter 40, and output images that have been verified as correct by an expert or other professional. The training DB is generated, for example, by a specialist or other professional familiar with the specifications of the diagnostic imaging catheter 40, based on case records using embodiment 1 or embodiment 2.

Figure 28 is a flowchart illustrating the processing flow of the program in embodiment 7. The explanation uses as an example a case where machine learning of the learning model 61 is performed using the information processing device 20.

The program in Figure 28 may be executed on hardware separate from the information processing device 20, and the learning model 61 after machine learning has been completed may be copied to the auxiliary storage device 23 via a network. A learning model 61 trained on a single piece of hardware may be used by multiple information processing devices 20.

Prior to running the program in Figure 28, an untrained model is prepared, for example, by combining a convolutional layer, a pooling layer, and a fully connected layer. The program in Figure 28 adjusts each parameter of the prepared model and performs machine learning.

The control unit 21 obtains the training record to be used for one epoch of training from the training DB (step S571). The control unit 21 adjusts the model parameters so that when an input image is input to the input layer of the model, an output image is output from the output layer (step S572).

The control unit 21 determines whether to terminate the processing (step S573). For example, the control unit 21 determines to terminate the processing when learning for a predetermined number of epochs has been completed. The control unit 21 may also obtain test data from the training DB, input it into the model under machine learning, and determine to terminate the processing when an output with a predetermined accuracy is obtained.

If it is determined not to end the processing (NO in step S573), the control unit 21 returns to step S571. If it is determined to end the processing (YES in step S573), the control unit 21 records the parameters of the trained model in the auxiliary storage device 23 (step S574). Thereafter, the control unit 21 ends the processing. Through the above processing, a trained model is generated.

According to this embodiment, the learning model 61 described in embodiment 3 can be generated by machine learning.

Eighth Embodiment
This embodiment relates to a method for generating the learning model 61 of the fourth embodiment described with reference to Fig. 15. Explanation of parts common to the fourth embodiment will be omitted.

Figure 29 is an explanatory diagram outlining a method for generating the learning model 61. As explained using Figure 15, the learning model 61 is a model that accepts an input image and generates an output image. The classifier 65 is a model that accepts an output image output from the learning model 61 and determines whether it is a true image or a false image. The classifier 65 has a configuration that connects, for example, a repetition of a convolutional layer and a pooling layer, a fully connected layer, and a softmax layer.

By using Generative Adversarial Networks (GAN), which trains the classifier 65 and the learning model 61 and adjusts their parameters, the learning model 61 is able to generate natural output images.

Figure 30 is a flowchart illustrating the processing flow of the program in embodiment 8. An example will be described in which machine learning of the learning model 61 is performed using the information processing device 20.

The program in Figure 30 may be executed on hardware separate from the information processing device 20, and the learning model 61 after machine learning has been completed may be copied to the auxiliary storage device 23 via a network. A learning model 61 trained on a single piece of hardware may be used by multiple information processing devices 20.

Prior to executing the program in Figure 30, a model is prepared that combines an untrained learning model 61 having the configuration described using Figure 15 with an untrained classifier 65.

The control unit 21 acquires multiple input images (step S581). The input images are recorded, for example, in the input image field of the training DB described using Figure 27. The input images include images that do not include the information missing portion 473.

The control unit 21 adjusts the parameters of the classifier 65 so that it outputs "false" for images input to the classifier 65 via the learning model 61, and outputs "true" for images that do not contain the information missing portion 473 and that are input to the classifier 65 without via the learning model 61 (step S582).

The control unit 21 adjusts the parameters of the learning model 61, specifically the parameters of the missing area extraction model 611 and the complementation model 612, so that the classifier 65 outputs "true" and "false" with equal probability (step S583). The control unit 21 may repeat steps S582 and S583 multiple times.

The control unit 21 records the parameters of the trained learning model 61 in the auxiliary storage device 23 (step S584). The control unit 21 then terminates the processing. Through the above processing, the trained learning model 61 is generated.

According to this embodiment, the learning model 61 described in embodiment 4 can be generated by machine learning.

Ninth Embodiment
This embodiment relates to a catheter system 10 that distinguishes between highly reliable and less reliable portions of a complementary line 572. Description of portions common to the third embodiment will be omitted.

Figure 31 is an explanatory diagram illustrating a method for generating an image displayed by the catheter system 10 of embodiment 9. In this embodiment, an example is given of a case where the display range of the tomographic image obtained using the diagnostic imaging catheter 40 is small, i.e., an input image displaying the vicinity of the diagnostic imaging catheter 40 is used.

The input image displays a guidewire image 472 and a shadowed area image 474. The shadowed area image 474 is an image that shows a strong reflector, such as a highly calcified area, a stent, or another medical device being used simultaneously with the diagnostic imaging catheter 40. Outside the guidewire image 472 and the shadowed area image 474, information-missing areas 473 due to acoustic shading are formed. In other words, the guidewire image 472 and the shadowed area image 474 are examples of images that show a shadowed area.

The control unit 21 generates two complementary lines 572, a first complementary line 561 and a second complementary line 562, based on the input image (step S101). In this embodiment, the first complementary line 561 represents the inner surface of the blood vessel, and the second complementary line 562 represents the external elastic lamina. Note that the number of complementary lines 572 generated in step S101 may be one or three or more.

The control unit 21 can generate an image with complementary lines by combining the input image with the complementary lines 572 (step S102). Note that the image with complementary lines is shown for convenience of explanation and does not need to be generated in reality.

The control unit 21 extracts the guidewire image 472 and the shadow-forming portion image 474 from the input image or the image with the complementary lines (step S103). The guidewire image 472 is an area with a brightness higher than a predetermined brightness and located inside the first complementary line 561. The control unit 21 may extract the guidewire image 472 by pattern matching based on a pre-specified shape and dimensions.

The shadowed area image 474 is an area with a higher brightness than a predetermined level and located outside the first complementary line 561. The shadowed area image 474 corresponds to, for example, a stent wire or a heavily calcified area. The guidewire image 472 and the shadowed area image 474 are examples of areas where the cause of the acoustic shadow formation is depicted.

The control unit 21 determines a low-reliability region 55 based on the extracted guidewire image 472 and shadow-forming portion image 474. The low-reliability region 55 is a roughly fan-shaped region that combines the region near the guidewire image 472 with the region outside the guidewire image 472, and a roughly fan-shaped region that combines the region near the shadow-forming portion image 474 with the region outside the shadow-forming portion image 474.

Because the guidewire image 472 or the shadow-forming portion image 474 is present between the diagnostic imaging catheter 40 and the low-reliability region 55, there is a high possibility that an acoustic shadow is occurring. Therefore, the reliability of the interpolated line 572 is low within the low-reliability region 55. Note that the control unit 21 may include, in the low-reliability region 55, an area of a specified width that is closer to the diagnostic imaging catheter than the guidewire image 472 and the shadow-forming portion image 474.

Figure 32 is an explanatory diagram illustrating a method for generating a complementary line 572 in embodiment 9. The learning model 61 in this embodiment is a model that accepts an input image and outputs classification data. The classification data is data that associates each part that makes up the input image with a label that classifies each subject depicted in that part. Each part is, for example, each pixel. The classification data can be used to generate a classification image in which the input image is colored differently for each subject depicted in the input image.

An example will be given below. The learning model 61 outputs classification data in which each pixel constituting the input image is classified into, for example, a first label, a second label, or a third label. An example of a classified image generated based on the classification data is shown below. A first label region 541, a second label region 542, and a third label region 543 are arranged in approximately concentric circles with the catheter image 471 at the center.

The first label area 541 indicates the lumen of the luminal organ into which the diagnostic imaging catheter 40 has been inserted, i.e., the region of the blood vessel lumen through which blood flows. The second label area 542 indicates the lumen wall, i.e., the blood vessel wall. The third label area 543 indicates the outside of the lumen wall, i.e., the region outside the external elastic lamina which indicates the outer surface of the luminal organ. The third label area 543 includes, for example, muscle, nerves, fat, and other blood vessels adjacent to the blood vessel into which the diagnostic imaging catheter 40 has been inserted.

The boundary line between the first label area 541 and the second label area 542 corresponds to the first complementary line 561 described above. The boundary line between the second label area 542 and the third label area 543 corresponds to the second complementary line 562 described above. As described above, the complementary line 572 created based on the input image may be referred to as the mode 1 complementary line 572 in the following description.

Figure 32 shows a schematic representation of an input image displayed in so-called XY format and a classification image in which classification data is displayed in XY format. The learning model 61 may accept an input image in so-called RT format, which is formed by arranging scan line data formed by the sensor 42 transmitting and receiving ultrasonic waves in parallel in order of scan angle, and output classification data. The method of converting from RT format to XY format is well known, so a description will be omitted. More appropriate classification data is generated because it is not affected by interpolation processes, etc., that occur when converting the input image from RT format to XY format.

The learning model 61 is, for example, a trained model that performs semantic segmentation on an input image. The trained model that performs semantic segmentation is generated by machine learning using training data that combines input images generated using the diagnostic imaging catheter 40 with classified images in which an expert colors the input images according to the subjects depicted.

The learning model 61 may be created for each site into which the diagnostic imaging catheter 40 is inserted, such as the coronary arteries, lower limb aorta, bile duct, pancreatic duct, and bronchi. The learning model 61 may also be created for each display range in which a tomographic image is generated using the diagnostic imaging catheter 40. The learning model 61 may also be created for each patient attribute, such as the patient's age or gender.

The complement line 572 of mode 1 may be generated using the learning model 61 of embodiment 3 described using Figure 11. When using the learning model 61 of embodiment 3, the first complement line 561 representing the inner surface of the blood vessel and the second complement line 562 representing the external elastic plate are each generated using an appropriate learning model 61. The complement line 572 of mode 1 may also be generated using any other method.

Figure 33 is an explanatory diagram illustrating a method for generating a mode 2 interpolation line 572. As explained using Figure 31, the explanation begins with the process after the generation of a mode 1 interpolation line 572 and the determination of a low-reliability region 55 are completed. In Figure 33, the guidewire image 472 and the shadow-forming portion image 474 are not shown.

The control unit 21 deletes the portion of the mode 1 complement line 572 that is included in the low-reliability region 55 (step S111). The complement line 572 is broken in part. The control unit 21 generates a change line 565 that smoothly connects the broken portion of the complement line 572. In this way, the control unit 21 generates the second mode complement line 572 (step S112). In Figure 33, the change lines 565 corresponding to each complement line 572 are shown in bold.

Any interpolation method can be used to generate the modified line, such as spline interpolation, Lagrange interpolation, or linear interpolation. The control unit 21 may accept a user specification of the generation method. The second mode complementary line 572 is less susceptible to artifacts such as multiple echoes formed by strong reflectors such as guidewires. The second mode complementary line described above is a complementary line obtained by correcting areas of the complementary line 572 that are less reliable than other areas based on areas that are not less reliable.

If a large portion of the complementary line 572 is included in the low-reliability region 55, the control unit 21 determines that the reliability of the entire complementary line 572 is low. For example, if the shadow-forming portion image 474 is a stent wire or if there is extensive, intense calcification, the range of the low-reliability region 55 is wide, and a large portion of the complementary line 572 is included in the low-reliability region 55.

Figure 34 is a flowchart explaining the processing flow of the program of embodiment 9. The control unit 21 acquires a real-time image from the catheter control unit 271 (step S601). The control unit 21 determines whether an instruction to perform completion has been received from the user (step S602). An example of a screen for receiving instructions from the user will be described later.

If it is determined that the request has not been received (NO in step S602), the control unit 21 displays a real-time image in the first image field 51 (step S603). If it is determined that the request has been received (YES in step S602), the control unit 21 starts a complementary line generation subroutine (step S611). The complementary line generation subroutine is a subroutine that generates a complementary line 572. The processing flow of the complementary line generation subroutine will be described later.

The control unit 21 extracts high-brightness areas from the real-time image (step S612). A high-brightness area is, for example, an area where a greater number of pixels than a predetermined threshold are clustered together. The brightness threshold and pixel count threshold may be set by the user as appropriate. Step S612 extracts the guidewire image 472 and the shadow-forming area image 474.

The control unit 21 sets the range of the low-reliability region 55 (step S613). In the XY format image, the low-reliability region 55 is a roughly fan-shaped region outside the guidewire image 472 and the shadow-forming portion image 474. The control unit 21 calculates the proportion of the length of the range included in the low-reliability region 55 to the total length for one of the complementary lines 572 generated in step S611 (step S614).

The control unit 21 determines whether the proportion of the range included in the low-reliability region 55 is greater than a predetermined threshold (step S615). If it is determined that it is greater (YES in step S615), the control unit 21 temporarily records in the main memory 22 or auxiliary memory 23 that the proportion of the range included in the low-reliability region 55 for the interpolated line 572 being processed is large (step S616).

If it is determined that the number is not large (NO in step S615), the control unit 21 determines whether an instruction to display the completion lines 572 in mode 2 has been received from the user (step S617). An example of a screen for receiving instructions from the user will be described later.

If it is determined that the request has been accepted (YES in step S617), the control unit 21 generates the complementary line 572 of mode 2 described using FIG. 33 (step S618). If it is determined that the request has not been accepted (NO in step S617), after step S618 or step S616, the control unit 21 determines whether processing of all complementary lines 572 generated in step S611 has been completed (step S619).

If it is determined that the process has not ended (NO in step S619), the control unit 21 returns to step S614. If it is determined that the process has ended (YES in step S619), the control unit 21 displays the real-time image and the complementary line 572 in the first image field 51 (step S620).

After step S603 or step S620 is completed, the control unit 21 determines whether to terminate the processing (step S621). For example, if the diagnostic imaging catheter 40 is removed from the MDU 33, the control unit 21 determines that the processing is to terminate.

If it is determined that the process should not end (NO in step S621), the control unit 21 returns to step S601. If it is determined that the process should end (YES in step S621), the control unit 21 ends the process.

Figure 35 is a flowchart illustrating the processing flow of the complementary line generation subroutine. The control unit 21 inputs the real-time image into the learning model 61 and acquires classification data (step S631). That is, in step S631, the control unit 21 acquires labels corresponding to the subjects depicted in each part that makes up the real-time image. The control unit 21 extracts complementary lines corresponding to the boundaries between areas with different acquired labels (step S632). The control unit 21 then terminates the processing.

Figure 36 is an example of a screen displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screen shown in Figure 36 on the display device 31 in step S603 of the flowchart described using Figure 34.

The screen shown in Figure 36 includes a first image field 51, two target selection buttons 591, an intraluminal high-brightness area button 594, and an intramural high-brightness area button 595. Below each target selection button 591, two mode selection buttons 592 and a non-display selection button 593 are displayed. The first image field 51 displays a real-time image obtained using the diagnostic imaging catheter 40.

In Figure 36, none of the buttons have been selected by the user. In step S602 of the flowchart described using Figure 34, the control unit 21 determines that an instruction to perform completion has not been received from the user (NO in step S602), and executes step S603.

The target selection button 591 accepts the selection of the complementary line 572 to be extracted. In FIG. 36, the user can select "intraluminal surface" and "EEM." Note that "intraluminal surface" and "EEM" are examples. The control unit 21 appropriately displays the type of complementary line 572 corresponding to the luminal organ into which the diagnostic imaging catheter 40 is inserted. The control unit 21 may display the target selection button 591 for the type of complementary line 572 previously set by the user.

The mode selection button 592 accepts a selection regarding the mode of the completion line 572. When a selection of "Mode 1" is accepted, the control unit 21 generates the completion line 572 of Mode 1, which was described using FIG. 31. When a selection of "Mode 2" is accepted, the control unit 21 generates the completion line 572 of Mode 2, which was described using FIG. 33.

The hide selection button 593 accepts a selection as to whether or not to display the completion line 572 for the portion included in the low-reliability region 55. The intraluminal high-brightness area button 594 accepts a selection as to whether or not to display information about high-brightness areas present inside the lumen of the luminal organ. The intramural high-brightness area button 595 accepts a selection as to whether or not to display information about high-brightness areas present inside the luminal wall, i.e., between the inner wall of the luminal organ and the external elastic lamina.

Figures 37 to 41 are examples of screens displayed by the catheter system 10 of embodiment 9. The control unit 21 displays the screens shown in Figures 37 to 41 on the display device 31 in step S620 of the flowchart described using Figure 34.

In Figure 37, the selection of the object selection button 591 corresponding to "Luminous Inner Surface" and "EEM" and the mode selection button 592 for "Mode 1" are accepted. A legend column 596 is displayed to the right of the words "Luminous Inner Surface" and "EEM."

The first image field 51 displays an image in which a first complementary line 561 and a second complementary line 562 are superimposed on a real-time image obtained using the diagnostic imaging catheter 40. Of the first complementary line 561 corresponding to the "inner surface of the lumen," the portion that does not overlap with the low-reliability region 55 described using Figure 31 is displayed as a thick solid line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin solid line. Of the second complementary line 562 corresponding to the "EEM," the portion that does not overlap with the low-reliability region 55 is displayed as a thick dashed line, and the portion that overlaps with the low-reliability region 55 is displayed as a thin dashed line.

In other words, in Figure 37, areas of the complementary lines 572 that are less reliable than other areas are displayed in a different manner from areas with higher reliability. Note that the complementary lines 572 may be displayed differently from each other, and the areas with lower reliability may be displayed differently from the areas with higher reliability, using different colors or brightness, etc.

The screen shown in FIG. 37 allows the user to easily distinguish between highly reliable and less reliable portions of the complementary lines 572 generated by the control unit 21. By operating the target selection button 591, the user can appropriately select the complementary lines 572 to be displayed in the first image field 51.

In Figure 38, the hide selection button 593 corresponding to "EEM" has been selected. In the first image field 51, the portion of the second completion line 562 that overlaps with the low-reliability region 55 has been erased.

The screen shown in Figure 38 allows the user to visually check the unreliable parts of the second complementary line 562 and make appropriate decisions based on their specialized knowledge.

In Figure 39, the intraluminal high-brightness area button 594 and the intramural high-brightness area button 595 are selected. The first image field 51 displays the low-reliability area 55 generated by the guidewire image 472, which is an intraluminal high-brightness area, and the low-reliability area 55 generated by the intramural high-brightness area 55.

The control unit 21 may display the guidewire image 472 and the shadow-forming area image 474 in color. This allows the user to confirm which areas the control unit 21 has determined will cause the low-reliability region 55.

Figure 40 shows a real-time image obtained at a site where a stent has been placed. Numerous shadow-forming area images 474 formed by the stent wires are depicted in a circular arrangement. Therefore, low-reliability regions 55 corresponding to each shadow-forming area image 474 are generated.

Figure 40 shows an example of a screen displayed by the control unit 21 when it is determined in step S615 described using Figure 34 that the range included in the low-reliability region 55 for the second complementary line 562 is greater than the threshold value.

The control unit 21 does not display the second complementary line 562 in the first image field 51. Instead of the legend field 596, the control unit 21 displays a notification field 597 to the right of the characters "EEM" indicating that it is "not displayable." The user can understand that the second complementary line 562 generated by the control unit 21 is not displayed in the first image field 51 because its reliability is low.

Figure 41 shows an example of a screen displayed by the control unit 21 when the user, for example, clicks on the notification field 597 to instruct the display of the second complementary line 562 with low reliability. The entire second complementary line 562 is displayed as a thin dashed line indicating low reliability. A notification in the lower left corner of the screen saying "EEM has low reliability" indicates that the reliability of the second complementary line 562 is low.

This embodiment provides a catheter system 10 that distinguishes between highly reliable and less reliable portions of the completion line 572.

The learning model 61 may output labels corresponding to the guidewire image 472 and the shadow-forming portion image 474. In this case, step S612 described using Figure 34 is not necessary.

The learning model 61 may output a label corresponding to, for example, high-attenuation plaque. In this case, the control unit 21 may also set the area outside the high-attenuation plaque as a low-reliability area 55 in step S613 described using FIG. 34.

Instead of real-time images, video or still images stored in the auxiliary storage device 23 or the like may be used. This provides a catheter system 10 that can be used for recording medical records after a case is completed. In this case, the information processing device 20 may be a PC, tablet, smartphone, or the like that does not have the functionality to connect the MDU 33 and the diagnostic imaging catheter 40.

[Embodiment 10]
42 is a functional block diagram of an information processing system 10 according to a tenth embodiment. The information processing system 10 includes an image acquisition unit 81, a complementary information acquisition unit 82, and a display unit 83. The image acquisition unit 81 acquires a catheter image generated using a diagnostic imaging catheter 40 inserted into a hollow organ. The complementary information acquisition unit 82 acquires complementary information that complements an information-missing portion 473 of the catheter image acquired by the image acquisition unit 81. The display unit 83 displays the catheter image acquired by the image acquisition unit 81 and the complementary information acquired by the complementary information acquisition unit 82.

[Embodiment 11]
43 is an explanatory diagram illustrating the configuration of a catheter system 10 according to an eleventh embodiment. This embodiment relates to a configuration in which the catheter system 10 according to this embodiment is realized by combining and operating a catheter control device 27, an MDU 33, an imaging diagnostic catheter 40, a general-purpose computer 90, and a program 97. Explanations of parts common to the first embodiment will be omitted. The catheter system 10 is also an example of the information processing system according to this embodiment.

The catheter control device 27 is an IVUS ultrasound diagnostic device that controls the MDU 33, the sensor 42, and generates transverse and longitudinal images based on signals received from the sensor 42. The function and configuration of the catheter control device 27 are similar to those of conventional ultrasound diagnostic devices, so a detailed description will be omitted.

The catheter system 10 of this embodiment includes a computer 90. The computer 90 is equipped with a control unit 21, a main memory device 22, an auxiliary memory device 23, a communication unit 24, a display unit 25, an input unit 26, a reading unit 29, and a bus. The computer 90 is an information device such as a general-purpose personal computer, tablet, smartphone, or server computer.

The program 97 is recorded on a portable recording medium 96. The control unit 21 reads the program 97 via the reading unit 29 and stores it in the auxiliary storage device 23. The control unit 21 may also read the program 97 stored in a semiconductor memory 98, such as a flash memory, implemented in the computer 90. Furthermore, the control unit 21 may download the program 97 from another server computer (not shown) connected via the communication unit 24 and a network (not shown) and store it in the auxiliary storage device 23.

Program 97 is installed as a control program for computer 90, loaded into main memory device 22, and executed. This causes computer 90 to function as the information processing device 20 described above.

Computer 90 is a general-purpose personal computer, tablet, smartphone, mainframe computer, virtual machine running on a mainframe computer, cloud computing system, or quantum computer. Computer 90 may also be multiple personal computers performing distributed processing.

The technical features (constituent elements) described in each embodiment can be combined with each other, and by combining them, new technical features can be formed.
The embodiments disclosed herein are illustrative in all respects and should not be considered as limiting. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

(Appendix 1)
Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ;
acquiring complementary information for complementing missing information in the acquired catheter image;
A program that causes a computer to execute a process of displaying the catheter image and the complementary information.

(Appendix 2)
The program described in Appendix 1, which inputs the acquired catheter image into a learning model that outputs complementary information when a catheter image is input, and obtains complementary information output from the learning model.

(Appendix 3)
The program according to claim 2, wherein the complementary information is a complementary image generated to complement the information-missing portion.

(Appendix 4)
The program according to claim 1, wherein the missing information portion is complemented based on a second catheter image acquired at a different time from the catheter image.

(Appendix 5)
The program according to claim 1, further comprising: complementing the missing information portion based on a portion of the catheter image that is different from the missing information portion.

(Appendix 6)
the diagnostic imaging catheter is a catheter for generating a tomographic image,
the catheter image is a tomographic image generated using the tomographic image generating catheter,
6. The program according to claim 1, wherein the information missing portion is a shaded portion.

(Appendix 7)
The program according to claim 6, wherein the catheter for generating a tomographic image is a catheter for generating an ultrasonic tomographic image.

(Appendix 8)
The program according to claim 6, wherein the catheter for generating a tomographic image is a catheter for generating an optical tomographic image.

(Appendix 9)
The program according to claim 6, wherein the catheter for generating a tomographic image is a catheter equipped with a sensor for generating an ultrasonic tomographic image and a sensor for generating an optical tomographic image.

(Appendix 10)
extracting a plurality of points representing the inner surface of a lumen wall constituting the hollow organ from the tomographic image;
The program according to any one of Supplementary Note 6 to Supplementary Note 9, wherein the shaded portion is complemented by a curve connecting the extracted points.

(Appendix 11)
11. The program according to claim 1, wherein the acquired catheter image and a completed image in which the missing information portion has been completed are displayed side by side.

(Appendix 12)
Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ;
acquiring complementary information for complementing missing information in the acquired catheter image;
An information processing method that causes a computer to execute a process of displaying the catheter image and the complementary information.

(Appendix 13)
an image acquisition unit that acquires a catheter image generated using a diagnostic imaging catheter inserted into a hollow organ;
a complementary information acquisition unit that acquires complementary information that complements information missing portions of the acquired catheter image;
an information processing system comprising a display unit that displays the catheter image and the complementary information.

(Appendix 14)
acquiring training data in which catheter images generated using a diagnostic imaging catheter inserted into a hollow organ and complementary information for complementing missing information in the catheter images are associated with each other and recorded;
A learning model generation method for generating a learning model that inputs the catheter image, outputs the complementary information, and outputs a prediction regarding the complementary information when an image generated using a diagnostic imaging catheter is input.

10. Catheter system (information processing system)
20 Information processing device 21 Control unit 22 Main memory device 23 Auxiliary memory device 24 Communication unit 25 Display unit 26 Input unit 27 Catheter control device 271 Catheter control unit 29 Reading unit 31 Display device 32 Input device 33 MDU
37 Diagnostic imaging device 40 Diagnostic imaging catheter 41 Probe section 42 Sensor 43 Shaft 44 Tip marker 45 Connector section 471 Catheter image 472 Guidewire image 473 Missing information section 474 Shadow-forming section image 483 Missing area image 485 Attached area 486 Estimated image 487 Cut-out image (complementary information)
51 First image field 52 Second image field 541 First label area 542 Second label area 543 Third label area 55 Low-reliability area 561 First complementary line 562 Second complementary line 565 Change line 571 Designated point mark 572 Complementary line (complementary information)
573 Candidate point mark 574 Template area 580 No completion button 581 Stop button 582 Select button 583 Completion button 584 Start button 585 End button 586 Coloring button 587 Frame button 588 Shading button 591 Target selection button 592 Mode selection button 593 Hide selection button 594 Intraluminal high-intensity area button 595 Intramural high-intensity area button 596 Legend column 597 Notification column 61 Learning model 611 Missing area extraction model 612 Complementary model 613 Cutting section 614 Synthesis section 65 Classifier 68 Cursor 90 Computer 96 Portable recording medium 97 Program 98 Semiconductor memory

Claims (10)

Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ;
acquiring complementary information that complements the information missing portion of the acquired catheter image;
determining a display mode of the complementary information based on a low-reliability area of the acquired complementary information that has a lower reliability than other areas ;
a program causing a computer to execute a process of displaying the catheter image and the complementary information in the display mode ; The low-confidence region includes the region outside the guidewire image and the shadow-forming region image.
The program according to claim 1. The shadowed area image includes an area corresponding to at least one of a calcified area, a stent wire, or a high-attenuation plaque.
The program according to claim 2. The complementary information is a line representing the external elastic lamina of a blood vessel.
The program according to claim 1. The display mode is determined based on whether the proportion of the low-reliability region in the complementary information is greater than a threshold value.
The program according to claim 1. The display mode includes a mode in which the complementary information is not displayed.
The program according to claim 1. The complementary information is complementary information output from a learning model that outputs complementary information when a catheter image is input to the learning model.
The program according to any one of claims 1 to 6. inputting the acquired catheter image into a learning model that outputs labels corresponding to the respective parts constituting the catheter image when the catheter image is input, and acquiring labels output from the learning model;
The complementary information includes a complementary line indicating a boundary between regions having different labels.
The program according to any one of claims 1 to 6. Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ;
acquiring complementary information that complements the information missing portion of the acquired catheter image;
determining a display mode of the complementary information based on a low-reliability area of the acquired complementary information that has a lower reliability than other areas ;
an information processing method in which a computer executes a process of displaying the catheter image and the complementary information in the display mode ; An information processing device including a control unit,
The control unit
Acquiring a catheter image generated using an imaging diagnostic catheter inserted into a hollow organ;
acquiring complementary information that complements the information missing portion of the acquired catheter image;
determining a display mode of the complementary information based on a low-reliability area of the acquired complementary information that has a lower reliability than other areas ;
Displaying the catheter image and the complementary information in the display mode
Information processing device .