JP7715528B2 - Information processing device, ultrasound diagnostic device, and program - Google Patents

Description

The embodiments disclosed in this specification relate to an information processing device, an ultrasound diagnostic device, and a program.

Conventionally, ultrasound diagnostic devices have been widely used to easily observe the state inside a subject's body by transmitting ultrasound waves from the body surface into the body and displaying ultrasound images based on the reflected waves. For example, because ultrasound diagnostic devices can display ultrasound images on a monitor in near real time, they are used when punctures are performed, such as in biopsies, radiofrequency ablation (RFA), and irreversible electroporation (IRE) treatments.

When performing such a puncture using a puncture needle, the puncture plan is made based on images taken in advance, such as CT (Computed Tomography) or MRI (Magnetic Resonance Imaging), and the puncture needle is inserted during treatment using an ultrasound image as a guide. When performing the puncture using an ultrasound image as a guide, a needle adapter is sometimes used to fix the angle of the puncture needle so that it can be visualized in the ultrasound image. Using a needle adapter ensures that the puncture needle is on the same plane as the ultrasound image, making it useful when it is difficult to visualize the needle during puncture.

In addition, when using ultrasound images to guide the puncture, it can be difficult to visualize the affected area using ultrasound, so a fusion function may be used to align the CT, MRI, or other images used to create the puncture plan with real-time ultrasound images, matching the angles and positions of the planes of both images and displaying them.

In recent years, the simultaneous insertion of multiple needles has become commonplace for therapeutic punctures. For example, multiple needles are used when the affected area is too large to be treated in one session, or when multiple needles are theoretically necessary depending on the treatment method. One example of puncture using multiple needles is irreversible electroporation, also known as Nanoknife (registered trademark). Because Nanoknife uses a different needle for each electrode, at least two needles must be inserted simultaneously, and in some cases, up to six needles can be inserted simultaneously. With Nanoknife, pulsed waves are applied for 30 minutes to an hour after the puncture to treat the target between the electrodes. Due to the complexity of the procedure, such multiple punctures are difficult to perform without prior planning. Therefore, puncture planning is performed in advance using CT, MRI, and ultrasound images. The punctures are performed according to this plan, guided by real-time ultrasound images.

In addition, research is underway into robotic ultrasound diagnosis, which uses a robotic arm to perform ultrasound scans, and research is also underway into using robotic arms for ultrasound-guided puncture treatment. When using such robotic arms for puncture treatment, it is expected that treatment will be planned in advance using 3D images such as CT or MRI, and the robotic arm will guide the puncture along the planned puncture location and route.

JP 2019-13788 A

One of the problems that the embodiments disclosed herein attempt to solve is to easily achieve accurate puncture when assisting puncture using a robotic arm. However, the problems solved by the embodiments disclosed herein are not limited to the above problem. Problems corresponding to the effects of each configuration shown in the embodiments described below can also be considered as other problems solved by the embodiments disclosed herein.

The information processing device of this embodiment includes a setting unit and a puncture control unit. The setting unit sets control information for a puncture needle held by a second robot arm based on puncture information input for an ultrasound image collected by an ultrasound probe held by a first robot arm. The puncture control unit controls the movement of the second robot arm holding the puncture needle based on the control information.

FIG. 1 is a diagram schematically illustrating an example of a puncture support system according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of the ultrasound diagnostic apparatus according to the first embodiment. FIG. 3 is a flowchart illustrating a processing procedure according to the first embodiment. FIG. 4 is a diagram illustrating an example of an arm that holds the ultrasound probe according to the first embodiment. FIG. 5 is a diagram illustrating an example of processing by the setting function according to the first embodiment. FIG. 6 is a diagram illustrating an example of processing by the control function according to the first embodiment. FIG. 7 is a block diagram showing an example of the configuration of an ultrasound diagnostic apparatus according to the second embodiment. FIG. 8 is a schematic diagram showing an example of a pseudo-puncture device according to the second embodiment.

Embodiments of an information processing device, an ultrasound diagnostic device, and a program according to the present application will be described in detail below with reference to the accompanying drawings. Note that the information processing device, the ultrasound diagnostic device, and the program according to the present application are not limited to the embodiments described below. Furthermore, in the following description, similar components will be assigned common reference numerals, and duplicate descriptions will be omitted.

(First embodiment)
First, a puncture assistance system including an information processing device according to the present application will be described. The puncture assistance system includes an ultrasound diagnostic device and a robot arm device, and adjusts a puncture needle held by the robot arm device under the control of the information processing device. Fig. 1 is a diagram schematically illustrating an example of a puncture assistance system 1000 according to the first embodiment. For example, as shown in Fig. 1, the puncture assistance system 1000 includes an ultrasound diagnostic device (US device) 100 and a robot arm device 200, and assists in puncturing a subject.

The ultrasound diagnostic device 100 has an ultrasound probe 12 and generates ultrasound images based on ultrasound transmitted and received by the ultrasound probe 12. For example, the ultrasound diagnostic device 100 generates an ultrasound image of the area of the subject to be punctured that has been scanned by the ultrasound probe 12.

The robot arm device 200 includes, for example, a robot arm control device 21, and arms 22a, 22b, and 22c. Arm 22a holds the puncture needle 31 and, under the control of the robot arm control device 21, changes its position and moves the puncture needle 31 in the insertion direction (longitudinal direction of the puncture needle 31). Arm 22b holds the puncture needle 32 and, under the control of the robot arm control device 21, changes its position and moves the puncture needle 31 in the insertion direction (longitudinal direction of the puncture needle 31). Arm 22c holds the ultrasound probe 12 and, under the control of the robot arm control device 21, changes its position to perform an ultrasound scan on the subject.

The robot arm control device 21 adjusts the positions of the puncture needle 31, the puncture needle 32, and the ultrasound probe 12 relative to the subject by moving the positions of the arms 22a, 22b, and 22c. Here, the robot arm control device 21 can control the positions of the arms 22a, 22b, and 22c based on control signals received from the information processing device. That is, the robot arm control device 21 moves the arms 22a, 22b, and 22c based on the arm movement amounts included in the control signals received from the information processing device. The positional relationship between the arms 22a, 22b, and 22c and the subject can be determined based on ultrasound images collected by the ultrasound probe 12 held by the arm 22c.

Note that while Figure 1 shows a robot arm device 200 with three arms, arm 22a, arm 22b, and arm 22c, the embodiment is not limited to this, and the number of arms can be any number as long as it has an arm that holds a puncture needle and an arm that holds an ultrasound probe 12. For example, it may have one or three or more arms that hold a puncture needle, or it may have two or more arms that hold an ultrasound probe 12.

The information processing device sends control signals to the robot arm device 200 to adjust the positions of arms 22a, 22b, and 22c of the robot arm device 200, thereby assisting in puncturing the subject. Here, the information processing device of the present application can easily achieve accurate puncture when assisting in puncture using a robot arm.

As mentioned above, when performing a puncture, a puncture plan is made based on three-dimensional images taken in advance, such as those from CT (Computed Tomography) or MRI (Magnetic Resonance Imaging), and the puncture is performed according to this puncture plan. However, because the patient's position and respiratory phase differ between when the CT or MRI is taken and when the puncture is performed, the actual puncture location and puncture route are almost always determined by checking an image of the patient's position immediately before treatment. Therefore, even in puncture assistance using a robotic arm, it is necessary to easily determine the puncture site based on the patient's position immediately before treatment and perform the puncture accurately.

Furthermore, if treatment using multiple needles, which currently requires a great deal of skill when performed manually, could be accurately performed, the scope of application of ultrasound-guided needle therapy could be greatly expanded and the quality of treatment could be improved. For example, in liver cancer, multiple simultaneous needles are required when the tumor is large, while in pancreatic cancer there are nanoknife treatments and cryotherapy, and in prostate cancer there are systematic biopsies, which require needles to be used.

The information processing device of the present application therefore makes it possible to easily achieve accurate puncture by controlling the robot arm in puncture assistance using ultrasound images collected from the subject's position immediately before puncture.

The device configuration according to the first embodiment of the present application will be described in detail below. Note that, in the first embodiment, an example will be described in which the information processing device according to the present application is included in an ultrasound diagnostic device, but the embodiment is not limited to this. In other words, the various processes performed by the ultrasound diagnostic device described below may be performed by any device in the puncture support system 1000 described above. Furthermore, an information processing device may be further connected to the puncture support system 1000 described above, and the connected information processing device may perform the various processes performed by the ultrasound diagnostic device described below.

Figure 2 is a block diagram showing an example of the configuration of an ultrasound diagnostic device 100 according to the first embodiment. As shown in Figure 2, the ultrasound diagnostic device 100 according to this embodiment has an ultrasound probe 12, a monitor 13, an input interface 14, and a device main body 15, and the ultrasound probe 12, the monitor 13, and the input interface 14 are communicatively connected to the device main body 15. The ultrasound diagnostic device 100 according to this embodiment is further communicatively connected to a robot arm control device 21. Note that the robot arm control device 21 may be built into the device main body 15.

The robot arm control device 21 has a communication interface and a processing circuit.

The communication interface is connected to the processing circuit and transmits and receives various information to and from the ultrasound diagnostic device 100. For example, the communication interface receives control signals from the ultrasound diagnostic device 100 and outputs the received control signals to the processing circuit. The communication interface also transmits information such as pressure detected by arms 22a to 22c to the ultrasound diagnostic device 100.

The processing circuit controls the driving of arms 22a, 22b, and 22c by sending control signals to the driving mechanisms of arms 22a, 22b, and 22c. Specifically, the processing circuit moves each arm by sending a control signal to the driving mechanism based on arm movement information (movement direction and amount) received from ultrasound diagnostic device 100. The processing circuit is a processor that reads programs corresponding to each process from memory (not shown) and executes them to realize the function corresponding to each program.

The arm 22a holds the puncture needle 31 and moves it under the control of the robot arm control device 21. Specifically, the arm 22a has a holding section that holds the puncture needle 31 and a drive mechanism for driving the arm. The arm 22a has multiple movable sections equipped with drive mechanisms such as motors and actuators that move in accordance with control received from the robot arm control device 21, thereby moving the puncture needle 31 held at its tip to the desired position. Here, the arm 22a is configured to be able to position the puncture needle 31 at any three-dimensional position relative to the subject.

The arm 22b holds the puncture needle 32 and moves it under the control of the robot arm control device 21. Specifically, the arm 22b has a holding section that holds the puncture needle 32 and a drive mechanism for driving the arm. The arm 22b has multiple movable sections equipped with drive mechanisms such as motors and actuators that move in accordance with control received from the robot arm control device 21, thereby moving the puncture needle 32 held at its tip to the desired position. Here, the arm 22b is configured to be able to position the puncture needle 32 at any three-dimensional position relative to the subject.

Here, arms 22a and 22b can also move puncture needles 31 and 32 in the longitudinal direction of the needles. That is, arms 22a and 22b are configured to automatically insert and remove puncture needles under the control of robot arm control device 21. For example, arm 22a has rubber rollers or the like at the tip of the arm, is connected to a drive mechanism such as a motor or actuator, holds puncture needle 31 between multiple rollers, and moves puncture needle 31 in the longitudinal direction of the needle by rotating the multiple rollers.

The arm 22a also has a pressure sensor that detects the pressure applied to the puncture needle 31. The arm 22a then transmits information about the detected pressure to the robot arm control device 21. The arm 22a can also manually hold the puncture needle 31 so that it can be moved.

As with the example of arm 22a described above, arm 22b also has a rubber roller or the like at the tip of the arm, allowing the puncture needle to be inserted and removed automatically. Arm 22b, like arm 22a, also has a pressure sensor that detects the pressure applied to the puncture needle 32 and transmits information about the detected pressure to the robot arm control device 21. Arm 22b can also manually hold the puncture needle 32 so that it can be moved.

The arm 22c holds the ultrasound probe 12 and moves it under the control of the robot arm control device 21. Specifically, the arm 22c has a holding section that holds the ultrasound probe 12 and a drive mechanism for driving the arm. The arm 22c has multiple movable sections equipped with drive mechanisms such as motors and actuators that move in accordance with control received from the robot arm control device 21, thereby moving the ultrasound probe 12 held at its tip to the desired position. Here, the arm 22c is configured to enable the ultrasound probe 12 to be positioned at any three-dimensional position relative to the subject.

The puncture needles 31 and 32 are, for example, electromagnetic needles that generate radio waves, and are connected to a treatment device that controls the output of the radio waves generated by the puncture needles 31 and 32. This treatment device can monitor the temperature of the puncture needles 31 and 32, the output of the radio waves, and the impedance of the ablation area, and the surgeon operates the treatment device to proceed with RFA using the puncture needles 31 and 32.

In addition, puncture needles 31 and puncture needles 32 are, for example, electrode needles that pass an electric current through the tissue to be treated, and are connected to a treatment device that controls the output of the electric current generated by puncture needles 31 and puncture needles 32. Here, this treatment device passes an electric current between puncture needles 31 and puncture needles 32, thereby passing an electric current through the tissue to be treated that is located between puncture needles 31 and puncture needles 32, thereby performing treatment.

The ultrasonic probe 12 is connected to a transmission/reception circuit 151 included in the device main body 15. The ultrasonic probe 12 has, for example, multiple piezoelectric vibrators in the probe main body, which generate ultrasonic waves based on drive signals supplied from the transmission/reception circuit 151. The ultrasonic probe 12 also receives reflected waves from the subject and converts them into electrical signals. The ultrasonic probe 12 also has, in the probe main body, matching layers provided on the piezoelectric vibrators and backing material that prevents ultrasonic waves from propagating backward from the piezoelectric vibrators. The ultrasonic probe 12 is detachably connected to the device main body 15. For example, the ultrasonic probe 12 is a sector-type, linear-type, or convex-type ultrasonic probe.

When ultrasound waves are transmitted from the ultrasound probe 12 to the subject, they are reflected successively by discontinuous surfaces of acoustic impedance in the subject's internal tissues and received as reflected wave signals by the multiple piezoelectric transducers of the ultrasound probe 12. The amplitude of the received reflected wave signal depends on the difference in acoustic impedance at the discontinuous surfaces from which the ultrasound waves are reflected. When the transmitted ultrasound pulse is reflected by the surface of a moving blood flow, heart wall, or the like, the reflected wave signal undergoes a frequency shift due to the Doppler effect, depending on the velocity component of the moving object relative to the direction of ultrasound transmission.

This embodiment is applicable to cases where a subject is scanned two-dimensionally using an ultrasound probe 12, which is a one-dimensional ultrasound probe in which multiple piezoelectric transducers are arranged in a row, and to cases where a subject is scanned three-dimensionally using an ultrasound probe 12 that mechanically vibrates multiple piezoelectric transducers of a one-dimensional ultrasound probe, or an ultrasound probe 12 that is a two-dimensional ultrasound probe in which multiple piezoelectric transducers are arranged two-dimensionally in a grid pattern.

The monitor 13 displays a GUI (Graphical User Interface) that allows the operator of the ultrasound diagnostic device 100 to input various setting requests using the input interface 14, as well as ultrasound images generated by the device main body 15. The monitor 13 also displays various messages and display information to notify the operator of the processing status and results of the device main body 15. The monitor 13 also has a speaker and can output audio.

The input interface 14 is operated to set a predetermined position (e.g., a puncture target), and may be realized by, for example, a trackball, switch buttons, a mouse, a keyboard, a touchpad that performs input operations by touching the operation surface, a touch monitor that integrates a display screen and touchpad, a non-contact input circuit using an optical sensor, and a voice input circuit. The input interface 14 is connected to the processing circuit 155, which will be described later, and converts input operations received from the operator into electrical signals and outputs them to the processing circuit 155. Note that, in this specification, the input interface 14 is not limited to those equipped with physical operating components such as a mouse and keyboard. For example, an electrical signal processing circuit that receives electrical signals corresponding to input operations from an external input device provided separately from the device and outputs these electrical signals to the processing circuit 155 is also included as an example of an input interface.

The device main body 15 has a transmission/reception circuitry 151, a B-mode processing circuitry 152, a Doppler processing circuitry 153, a memory 154, a processing circuitry 155, and a communication interface 156. In the ultrasound diagnostic device 1 shown in FIG. 2, each processing function is stored in the memory 154 in the form of a program executable by a computer. The transmission/reception circuitry 151, the B-mode processing circuitry 152, the Doppler processing circuitry 153, and the processing circuitry 155 are processors that realize the function corresponding to each program by reading and executing the program from the memory 154. In other words, when each program is read, each circuit has the function corresponding to the read program.

The transmission/reception circuit 151 includes a pulse generator, a transmission delay circuit, a pulser, etc., and supplies a drive signal to the ultrasonic probe 12. The pulse generator repeatedly generates rate pulses at a predetermined rate frequency to form transmitted ultrasonic waves. The transmission delay circuit focuses the ultrasonic waves generated from the ultrasonic probe 12 into a beam and provides a delay time for each piezoelectric transducer required to determine the transmission directivity to each rate pulse generated by the pulse generator. The pulser also applies a drive signal (drive pulse) to the ultrasonic probe 12 at a timing based on the rate pulse. In other words, the transmission delay circuit varies the delay time provided for each rate pulse to arbitrarily adjust the transmission direction of the ultrasonic waves transmitted from the piezoelectric transducer surface.

The transmitter/receiver circuit 151 has the ability to instantly change the transmission frequency, transmission drive voltage, etc., in order to execute a specified scan sequence based on instructions from the processing circuit 155, which will be described later. In particular, changes to the transmission drive voltage are achieved by a linear amplifier-type oscillator circuit that can instantly switch its value, or a mechanism that electrically switches between multiple power supply units.

The transmission/reception circuit 151 also has a preamplifier, A/D (Analog/Digital) converter, reception delay circuit, adder, etc., and performs various processes on the reflected wave signals received by the ultrasound probe 12 to generate reflected wave data. The preamplifier amplifies the reflected wave signals for each channel. The A/D converter A/D converts the amplified reflected wave signals. The reception delay circuit provides the delay time required to determine the reception directivity. The adder adds the reflected wave signals processed by the reception delay circuit to generate reflected wave data. The addition process by the adder emphasizes the reflected components from the direction corresponding to the reception directivity of the reflected wave signal, and an overall beam for ultrasound transmission and reception is formed based on the reception directivity and transmission directivity.

The B-mode processing circuit 152 receives reflected wave data from the transmission/reception circuit 151 and performs logarithmic amplification, envelope detection processing, etc. to generate data (B-mode data) in which signal strength is expressed as luminance brightness.

The Doppler processing circuit 153 performs frequency analysis on the velocity information from the reflected wave data received from the transmission/reception circuit 151, extracts blood flow, tissue, and contrast agent echo components due to the Doppler effect, and generates data (Doppler data) that extracts moving object information such as velocity, dispersion, and power for multiple points. For example, the moving object could be a fluid such as blood flowing through blood vessels or lymph flowing through lymphatic vessels.

The B-mode processing circuit 152 and the Doppler processing circuit 153 are capable of processing both two-dimensional reflected wave data and three-dimensional reflected wave data. That is, the B-mode processing circuit 152 generates two-dimensional B-mode data from two-dimensional reflected wave data, and generates three-dimensional B-mode data from three-dimensional reflected wave data. The Doppler processing circuit 153 generates two-dimensional Doppler data from two-dimensional reflected wave data, and generates three-dimensional Doppler data from three-dimensional reflected wave data. The three-dimensional B-mode data is data in which brightness values corresponding to the reflection intensity of the reflection source located at each of multiple points (sample points) set on each scan line of the three-dimensional scanning range are assigned. The three-dimensional Doppler data is data in which brightness values corresponding to blood flow information (velocity, dispersion, power) are assigned to each of multiple points (sample points) set on each scan line of the three-dimensional scanning range.

The B-mode processing circuit 152 and the Doppler processing circuit 153 can also combine multiple two-dimensional reflected wave data to generate three-dimensional reflected wave data, and generate three-dimensional data from the generated reflected wave data. For example, the B-mode processing circuit 152 combines multiple two-dimensional reflected wave data to generate three-dimensional reflected wave data, and generates three-dimensional B-mode data from the generated three-dimensional reflected wave data. The Doppler processing circuit 153 combines multiple two-dimensional reflected wave data to generate three-dimensional reflected wave data, and generates three-dimensional Doppler data from the generated three-dimensional reflected wave data.

Memory 154 stores image data for display generated by processing circuitry 155. Memory 154 can also store data generated by the B-mode processing circuitry 152 and Doppler processing circuitry 153. Memory 154 also stores control programs for ultrasound transmission and reception, image processing, and display processing, as well as various data such as diagnostic information (e.g., patient ID, doctor's findings, etc.), diagnostic protocols, and various body marks.

The communication interface 156 is connected to the processing circuitry 155 and controls communications between the ultrasound diagnostic apparatus 100 and each device. Specifically, the communication interface 156 receives various types of information from each device and outputs the received information to the processing circuitry 155. For example, the communication interface 156 is implemented by a network card, network adapter, NIC (Network Interface Controller), etc. For example, the communication interface 156 transmits control signals received from the processing circuitry 155 to the robot arm control device 21.

The processing circuitry 155 controls the overall processing of the ultrasound diagnostic device 100. Specifically, the processing circuitry 155 performs various processes by reading from the memory 154 and executing programs corresponding to the control function 155a, image generation function 155b, scan control function 155c, setting function 155d, puncture control function 155e, and alignment function 156f shown in FIG. 2. For example, the processing circuitry 155 is a processor that realizes the function corresponding to each program by reading and executing each program from the memory 154. In other words, when each program has been read, the processing circuitry 155 has each function shown in the processing circuitry 155 in FIG. 2.

Here, control function 155a is an example of a display control unit. Image generation function 155b is an example of an image generation unit. Scanning control function 155c is an example of a scanning control unit. Setting function 155d is an example of a setting unit. Puncture control function 155e is an example of a puncture control unit. Alignment function 156f is an example of an alignment unit. Note that in this embodiment, each of the processing functions described below is implemented by a single processing circuit 155, but a processing circuit may be configured by combining multiple independent processors, and functions may be implemented by each processor executing a program.

The control function 155a controls the processing of the transmission/reception circuit 151, B-mode processing circuit 152, and Doppler processing circuit 153 based on various setting requests input by the operator via the input interface 14, and various control programs and data read from the memory 154. The control function 155a also controls the display of ultrasound images and various information on the monitor 13.

The control function 155a also acquires information about a pre-planned puncture plan via the communication interface 156. For example, the control function 155a acquires a puncture plan that has been established based on three-dimensional medical image data collected by a CT scanner, MRI scanner, or ultrasound diagnostic scanner. In other words, the control function 155a acquires information about the puncture position for the subject that is set based on the puncture plan.

The image generation function 155b generates ultrasound image data from the data generated by the B-mode processing circuit 152 and the Doppler processing circuit 153. That is, the image generation function 155b generates B-mode image data, which represents the intensity of the reflected wave as brightness, from the two-dimensional B-mode data generated by the B-mode processing circuit 152. The B-mode image data is data that depicts the tissue shape within the ultrasonically scanned area. The image generation function 155b also generates Doppler image data that represents moving object information from the two-dimensional Doppler data generated by the Doppler processing circuit 153. The Doppler image data is velocity image data, dispersion image data, power image data, or image data that combines these. The Doppler image data is data that indicates fluid information regarding the fluid flowing within the ultrasonically scanned area.

Here, the image generation function 155b generally converts (scan converts) the scan line signal sequence of the ultrasound scan into a scan line signal sequence in a video format, such as that used on a television, to generate ultrasound image data for display. Specifically, the image generation function 155b generates ultrasound image data for display by performing coordinate conversion according to the ultrasound scanning format used by the ultrasound probe 12. In addition to scan conversion, the image generation function 155b also performs various other image processing, such as image processing (smoothing processing) that regenerates an average brightness image using multiple image frames after scan conversion, and image processing (edge enhancement processing) that uses a differential filter within the image. The image generation function 155b also combines text information of various parameters, scales, body marks, etc., with the ultrasound image data.

Furthermore, the image generation function 155b generates three-dimensional B-mode image data by performing coordinate transformation on the three-dimensional B-mode data generated by the B-mode processing circuitry 152. Also, the image generation function 155b generates three-dimensional Doppler image data by performing coordinate transformation on the three-dimensional Doppler data generated by the Doppler processing circuitry 153. Furthermore, the image generation function 155b can perform rendering processing on the volume data to generate various types of two-dimensional image data for displaying the volume data on the monitor 13.

The scanning control function 155c controls the movement of the arm 22c that holds the ultrasonic probe 12. Specifically, the scanning control function 155c controls the movement of the arm 22c by sending a control signal to the robot arm control device 21. Here, the scanning control function 155c receives manual operation of the ultrasonic probe 12 by the operator, and controls the arm 22c to assist the manual operation of the ultrasonic probe 12. Specifically, the scanning control function 155c controls the ultrasonic probe 12 to assist the force applied by the operator.

The setting function 155d sets control information for the puncture needle 31 (or/and the puncture needle 32) held by the arm 22a (or/and the arm 22b) based on puncture information input for the ultrasound image collected by the ultrasound probe 12 held by the arm 22c. Specifically, the setting function 155d sets control information for operating the arm holding the puncture needle based on puncture information (such as information about the puncture target) input for the ultrasound image collected immediately before the insertion of the puncture needle.

The puncture control function 155e controls the movement of the arm 22a (or/and arm 22b) holding the puncture needle 31 (or/and puncture needle 32). Specifically, the scanning control function 155c controls the movement of the arm 22a (or/and arm 22b) by sending control signals to the robot arm control device 21. Specifically, the puncture control function 155e controls the movement of the arm holding the puncture needle based on the control information set by the setting function 155d. More specifically, the puncture control function 155e controls the arm 22a (or/and arm 22b) based on the control information to adjust the position and angle of the puncture needle 31 (or/and puncture needle 32), move the puncture needle 31 (or/and puncture needle 32) to the insertion position on the subject's body surface, and insert the puncture needle 31 (or/and puncture needle 32) into the puncture target.

The alignment function 155f aligns the ultrasound image with the reference image. Specifically, the alignment function 155f aligns the ultrasound image collected immediately before the puncture with the CT, MRI, or other image used to create the puncture plan, so that the angles and positions of the planes of both images match. For example, the alignment function 155f aligns the ultrasound image with the reference image based on the puncture target specified in the ultrasound image and the puncture target set on the reference image. This associates the coordinate space in the robot arm device 200 with the coordinate space in the reference image.

The device configuration according to the first embodiment has been described above. Next, the information processing device (ultrasound diagnostic device 100) according to the first embodiment will be described in detail. First, the processing procedure using the information processing device of the present application will be described using FIG. 3. FIG. 3 is a flowchart for explaining the processing procedure according to the first embodiment. Steps S101 to S102 in FIG. 3 are procedures performed when planning a puncture, and steps S103 to S106 are procedures performed when actually performing a puncture treatment. Step S103 is implemented by the processing circuitry 155 reading and executing programs corresponding to the control function 155a, image generation function 155b, and scan control function 155c from memory 154. Steps S104 and S105 are implemented by the processing circuitry 155 reading and executing programs corresponding to the setting function 155d from memory 154. Furthermore, step S106 is realized by the processing circuitry 155 reading from memory 154 and executing a program corresponding to the puncture control function 155e.

As shown in FIG. 3, in a procedure using the information processing device of the present application, for example, the surgeon first collects three-dimensional image data (volume data) using a CT scanner, an MRI scanner, an ultrasound diagnostic device 100, or the like (step S101). Then, the surgeon uses the collected three-dimensional image data to plan the puncture position (step S102).

For example, the surgeon identifies the position of the target to be treated by observing three-dimensional image data collected using a CT scanner, MRI scanner, ultrasound diagnostic device 100, or the like. The surgeon then determines the positions and number of puncture needles to insert, taking into account the shape and size of the target and the positions of surrounding organs and bones. For example, the surgeon determines the insertion position and angle on the body surface of each puncture needle so that appropriate treatment can be performed on the target while avoiding blood vessels, various organs, bones, etc. In other words, the puncture plan determines the positions on the body surface where the puncture needles will be inserted and the insertion angle of the puncture needles in the coordinates of the volume data.

After the puncture plan is established as described above, the puncture is performed. Specifically, the processing circuitry 155 first controls arm 22c to collect ultrasound images immediately before the puncture (step S103). The processing circuitry 155 then receives puncture information for the collected ultrasound images (step S104) and sets control information for controlling arm 22a (or/and arm 22b) that holds the puncture needle, based on the puncture information for the ultrasound images.

Then, the processing circuitry 155 controls the arm 22a (or/and arm 22b) that holds the puncture needle based on the control information (step S106) and performs the puncture. In this way, the information processing device (ultrasound diagnostic device 100) of the present application controls the arm based on the ultrasound image collected immediately before the puncture, thereby easily achieving accurate puncture when using an arm to assist in puncture. The details of the processing by the ultrasound diagnostic device 100 are explained below.

(Acquisition of ultrasound images)
As described above, the scanning control function 155c collects ultrasound images by controlling the movement of the arm 22c that holds the ultrasound probe 12. Specifically, the scanning control function 155c controls the movement of the arm 22c by transmitting control information including coordinate information in the operational space of the robot arm device 200 to the robot arm control device 21.

Here, the scanning control function 155c receives manual operation of the ultrasound probe 12 by the operator and controls the arm 22c to assist in manual operation of the ultrasound probe. Specifically, when the scanning control function 155c receives an operation to move the ultrasound probe 12 held by the arm 22c with the operator's hand, it drives the drive mechanism of the arm 22c to assist the operator in moving the ultrasound probe 12.

In other words, the scanning control function 155c controls the ultrasound probe 12 to supplement the force applied by the operator. For example, the scanning control function 155c automatically calculates the load, such as the weight of the ultrasound probe and cable, in real time and maintains this load, thereby reducing the burden on the operator when scanning the probe. The scanning control function 155c also detects the direction of manual movement using a sensor provided on the arm 22c, and controls the arm 22c to move in the detected direction, thereby reducing the burden on the operator when scanning the probe.

This allows the operator to move the ultrasound probe 12 more easily than manually scanning with a conventional ultrasound probe, and place the ultrasound probe 12 in an appropriate position where ultrasound images including the puncture target can be collected. As a result, the scan control function 155c can determine the position (coordinates) of the arm 22c in the coordinate space of the robot arm device 200 for collecting appropriate ultrasound images when performing puncture.

Here, the scan control function 155c can further adjust the position of the ultrasound probe that has been moved manually. As described above, the ultrasound probe 12 held by the arm 22c is placed on the subject's body surface by the operator's manual scanning. However, due to the subject's body movements, etc., the position of the ultrasound probe 12 required to collect an appropriate ultrasound image may be slightly shifted.

The scanning control function 155c therefore controls the arm 22c to fine-tune the position and angle of the ultrasonic probe 12. Figure 4 is a diagram showing an example of the arm 22c that holds the ultrasonic probe 12 according to the first embodiment. For example, as shown in Figure 4, the arm 22c holds the ultrasonic probe 12 at its tip. The scanning control function 155c then drives the drive mechanism of the arm 22c to move the ultrasonic probe 12 in the X-axis, Y-axis, and Z-axis directions. Furthermore, the scanning control function 155c drives the drive mechanism of the holder of the arm 22c to rotate the ultrasonic probe 12 in the directions of arrows D1 to D3, thereby changing the orientation of the ultrasonic probe 12. The scanning control function 155c performs these movements at a fine level to fine-tune the position and angle of the ultrasonic probe 12.

This makes it easy to move the ultrasound probe 12 slightly and fix its position after movement, making it easier to collect more accurate ultrasound images.

(Control information setting)
As described above, the setting function 155d sets control information for controlling the arm 22a (and/or the arm 22b) based on the puncture information input for the ultrasound image collected in real time during the puncture. That is, the setting function 155d identifies a position corresponding to the puncture information in the spatial coordinates where the arm 22c and the arm 22a (and the arm 22b) operate, and sets the control information for the puncture needle based on the identified position.

Specifically, the setting function 155d identifies the position (coordinates) of the ultrasound image in the coordinate space of the robot arm device 200 based on the position (coordinates) of the arm 22c in that coordinate space when the ultrasound image into which the puncture information was input was collected, thereby identifying the position (coordinates) of the ultrasound image in that coordinate space.

Figure 5 is a diagram illustrating an example of processing by the setting function 155d according to the first embodiment. Here, Figure 5 shows processing in a state where an ultrasound image is displayed that appropriately shows the puncture target, puncture route, etc. For example, when a doctor specifies a target T1 on the ultrasound image as shown in the upper part of Figure 5 and then specifies a puncture route, the setting function 155d sets control information for performing puncture on the specified target T1 along the specified puncture route.

That is, the setting function 155d determines the position (coordinates) of the ultrasound image in the coordinate space S1 of the robot arm device 200, as shown in the lower diagram of Figure 5, based on the position (coordinates) of the arm 22c when the ultrasound image was collected. The setting function 155d then determines the positions (coordinates) of the target T1 and puncture path specified on the ultrasound image in the coordinate space S1.

Then, the setting function 155d sets control information for moving the arm 22a (or/and) the arm 22b holding the puncture needle based on the current position (coordinates) in the coordinate space S1 of the arm 22a (or/and) the arm 22b, and the positions (coordinates) in the coordinate space S1 of the identified target T1 and the puncture path. In other words, the setting function 155d sets control information for moving the arm 22a (or/and) the arm 22b from the current position to a position for performing the puncture.

In ultrasound-guided puncture, various techniques are used to obtain appropriate tumor images or ensure an appropriate puncture route, such as changing the patient's position, changing the respiratory phase, and adjusting the way the probe is applied, pushed, and tilted. Furthermore, in ultrasound-guided puncture, artifacts caused by gas in the lungs are also reduced by using artificial pleural effusion or artificial ascites. Even in these various situations, by setting the control information described above, it is possible to respond to changes in the needle tip position and puncture route position coordinates in the coordinate space of the robot arm device 200, allowing for easy position correction.

Furthermore, ultrasound-guided puncture may be performed while referring to a reference image, such as a CT or MRI image, which is aligned with the ultrasound image. In such procedures, a position sensor is typically attached to the ultrasound probe, and real-time ultrasound image position information is linked to the reference image. However, in the present application, a dedicated position sensor is not attached to the ultrasound probe 12, and position information of the arm 22c in the coordinate space of the robot arm device 200 is used. Specifically, the alignment function 155f reads a reference image, such as an ultrasound image, CT image, or MRI image, and aligns it with the real-time ultrasound image. This allows the position information on the reference image to be defined in terms of coordinates in the coordinate space of the robot arm device 200, synchronized with the position information of the real-time ultrasound image.

In this way, once the real-time ultrasound image and the reference image are aligned, the position on the reference image can be converted to a position in the coordinate space of the robot arm device 200, making it possible to specify the target position and puncture path during puncture on the reference image. In other words, the setting function 155d sets control information for the puncture needle held in arm a (or/and arm 22b) based on the puncture information input for the reference image that has been aligned with the ultrasound image.

Furthermore, once the real-time ultrasound image and the reference image are aligned, the control function 155a can display the real-time ultrasound image and the corresponding reference image on the monitor 13. For example, the control function 155a can display the real-time ultrasound image and the corresponding reference image side by side.

The control function 155a can also display the position of the ultrasound probe 12 and the position of the real-time ultrasound image on the reference image. This allows the operator to move the arm 22c holding the ultrasound probe 12 using the position of the ultrasound probe 12 displayed on the reference image and the position of the real-time ultrasound image.

For example, the control function 155a displays markers indicating the position of the ultrasound probe 12 and the position of the ultrasound image on the reference image. Furthermore, for example, the control function 155a displays a GUI for inputting the distance and angle to move the ultrasound probe 12. The operator sets the control information for the arm 22c by moving the markers and inputting numerical values via the input interface 14.

The scanning control function 155c controls the movement of the ultrasound probe 12 held by the arm 22c based on operation information input for the reference image that has been aligned with the ultrasound image. In other words, the scanning control function 155c moves the arm 22 based on control information set by the operator so that an ultrasound image of the specified position is acquired.

(Control of puncture by robotic arm)
As described above, the puncture control function 155e controls the movement of the arm 22a (or/and the arm 22b) that holds the puncture needle based on the control information set by the setting function 155d. Specifically, the puncture control function 155e moves the arm 22a (or/and the arm 22b) to the insertion position of the puncture needle based on the control information. That is, the puncture control function 155e changes the angle of the arm so that the tip of the puncture needle is positioned at the insertion position and the puncture needle is inserted at the angle of the puncture path based on the coordinates of the insertion position on the subject's body surface, the coordinates of the needle tip position in the target, and the coordinates of the puncture path, which are included in the control information.

Here, the puncture control function 155e can support manual manipulation of the puncture needle by accepting manual manipulation of the puncture needle by the operator and controlling arm 22a (or/and arm 22b). In other words, the operator can manually move arm 22a (or/and arm 22b).

The puncture control function 155e then controls the arm to move the puncture needle in the longitudinal direction. Here, the puncture control function 155e can control the pressure and speed when inserting and removing the puncture needle.

For example, the puncture control function 155e controls the insertion speed of the puncture needle to a constant value, and controls the insertion speed of the puncture needle in response to changes in the insertion pressure of the puncture needle. As an example, the puncture control function 155e monitors the pressure on the puncture needle while inserting the puncture needle at a constant insertion speed, and controls the insertion speed to increase if the pressure exceeds a threshold value.

Furthermore, for example, the puncture control function 155e controls the insertion pressure of the puncture needle to a constant value, and controls the insertion pressure of the puncture needle in response to changes in the insertion speed of the puncture needle. As an example, the puncture control function 155e monitors the insertion speed of the puncture needle while inserting the puncture needle at a constant insertion pressure, and increases the allowable pressure applied to the puncture needle if the insertion speed falls below a threshold.

Furthermore, for example, the puncture control function 155e can switch between a constant insertion speed and a constant insertion pressure depending on the position from the subject's body surface to the target. For example, the puncture control function 155e controls the insertion speed to a constant speed up to a position a predetermined distance from the body surface, and controls the insertion pressure to a constant speed from the position a predetermined distance from the body surface to the target.

As described above, the puncture control function 155e can control various conditions, such as the insertion speed and pressure, thresholds for changing the speed and pressure, and switching between a constant insertion speed and a constant insertion pressure, by appropriately combining these conditions. The above conditions can be set arbitrarily depending on the insertion position of the puncture needle in the subject, the target organ, and so on. In other words, the puncture control function 155e acquires information about the insertion position of the puncture needle and the target organ, reads out the conditions corresponding to the acquired information, and inserts the puncture needle.

As described above, the puncture control function 155e can monitor the actual insertion speed and pressure while controlling the insertion speed and pressure of the puncture needle. In this case, the control function 155a can display on the monitor 13 the insertion speed and pressure of the puncture needle controlled by the puncture control function 155e and the actually monitored insertion speed and pressure.

Figure 6 is a diagram illustrating an example of processing by the control function 155a according to the first embodiment. For example, as shown in Figure 6, the control function 155a displays the "speed: xxx" and "pressure: yyy" of the puncture needle that are actually being monitored during ultrasound-guided puncture, along with an ultrasound image including the puncture needle. Note that the control function 155a can also display the set insertion speed and insertion pressure in addition to the actually monitored insertion speed and insertion pressure.

(Control of procedures using multiple puncture needles)
The robotic arm can control not only single-needle procedures but also multiple-needle procedures, such as nanoknife, ablation treatment with Celon needles, and systematic prostate biopsy (a procedure in which multiple predetermined locations are punctured).

For such procedures, puncture control conditions, such as the insertion speed, insertion pressure, and position of the puncture needle, as well as a puncture protocol, are registered in advance in memory 154, and the scanning control function 155c and puncture control function 155e perform control according to the puncture control conditions and puncture protocol. That is, the scanning control function 155c and puncture control function 155e read the puncture control conditions and puncture protocol specified by the operator from memory 154 and perform control in accordance with the read puncture control conditions and puncture protocol. The puncture control conditions and puncture protocol can be edited as appropriate.

(Manual puncture)
In the above example, the puncture control function 155e controls the arm 22a (or/and the arm 22b) to automatically insert the puncture needle. However, the arms 22a and 22b can hold the puncture needle so that the puncture needle can be inserted manually. That is, the puncture control function 155e positions the tip of the puncture needle at the insertion position, changes the angle of the arm so that the puncture needle inserts at the angle of the puncture path, and then the operator (doctor) moves the puncture needle to insert it into the subject.

Alternatively, manual puncture can be performed by having arm 22a (or/and arm 22b) hold a puncture adapter that guides the insertion of the puncture needle instead of the puncture needle. In such cases, setting function 155d sets control information for moving arm 22a (or/and arm 22b) so that the attachment position of the puncture needle in the puncture adapter is aligned with the insertion position on the body surface and the angle at which the puncture needle is held by the puncture adapter matches the angle of the puncture path.

The puncture control function 155e moves arm 22a (or/and arm 22b) based on the control information set by the setting function 155d. This allows the puncture adapter to be held stably in the desired position. The operator (doctor) can attach a puncture needle to the puncture adapter held by arm 22a (or/and arm 22b) and insert it themselves.

As described above, according to the first embodiment, the setting function 155d sets control information for the puncture needle held by the arm 22a (or/and the arm 22b) based on puncture information input for the ultrasound image collected by the ultrasound probe 12 held by the arm 22c. Therefore, the ultrasound diagnostic device 100 according to the first embodiment can set control information for the puncture needle using real-time ultrasound images immediately before puncture, making it possible to easily achieve accurate puncture when puncture assistance is performed using a robotic arm.

Furthermore, according to the first embodiment, the scan control function 155c receives manual operation of the ultrasound probe 12 by the operator and controls the arm 22c to support the manual operation of the ultrasound probe 12. The setting function 155d sets control information based on puncture information input for ultrasound images collected by the ultrasound probe 12 held by the arm 22c operated by manual operation. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to easily collect ultrasound images using the ultrasound probe held by the arm.

Furthermore, according to the first embodiment, the scan control function 155c controls the ultrasound probe 12 to assist the force applied by the operator. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to reduce the burden on the operator during manual operation.

Furthermore, according to the first embodiment, the scan control function 155c adjusts the position of the ultrasound probe 12 that has been moved manually. Therefore, the ultrasound diagnostic device 100 according to the first embodiment can finely adjust the position of the ultrasound probe 12 and can stably maintain the ultrasound probe 12 in the adjusted position.

Furthermore, according to the first embodiment, the puncture control function 155e receives manual manipulation of the puncture needle by the operator and controls the arm 22a (or/and arm 22b), thereby assisting the manual manipulation of the puncture needle. Therefore, the ultrasound diagnostic device 1 according to the first embodiment makes it possible to easily move the arm 22a (or/and arm 22b).

Furthermore, according to the first embodiment, the setting function 155d identifies a position corresponding to the puncture information in the spatial coordinates where the arms 22c and 22a (or/and 22b) operate, and sets the control information for the puncture needle based on the identified position. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to accurately perform puncture at a position specified in an ultrasound image.

Furthermore, according to the first embodiment, the alignment function 155f aligns the ultrasound image with the reference image. The setting function 155d sets control information for the puncture needle held by arm 22a (or/and arm 22b) based on puncture information input for the reference image that has been aligned with the ultrasound image. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to set puncture information using the reference image, taking into account the subject's body position immediately before puncture.

Furthermore, according to the first embodiment, the alignment function 155f aligns the ultrasound image with the reference image. The scan control function 155c controls the movement of the ultrasound probe 12 held by the arm 22c based on operation information input for the reference image that has been aligned with the ultrasound image. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to determine the position of the ultrasound image by referring to information contained in the reference image.

Furthermore, according to the first embodiment, arm 22a (or/and arm 22b) holds a puncture adapter that guides the insertion of a puncture needle. Therefore, the ultrasound diagnostic device 100 according to the first embodiment allows manual insertion of a puncture needle to be performed in a more stable manner.

Furthermore, according to the first embodiment, the puncture control function 155e controls the pressure and speed when inserting and removing the puncture needle. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to perform puncture using a robot arm that is compatible with various procedures.

Furthermore, according to the first embodiment, the puncture control function 155e controls the insertion speed of the puncture needle to a constant value, and controls the insertion speed of the puncture needle in response to changes in the insertion pressure of the puncture needle. Therefore, the ultrasound diagnostic device 100 according to the first embodiment is able to detect changes in the hardness and ease of insertion inside the living body, and perform puncture accordingly.

Furthermore, according to the first embodiment, the puncture control function 155e controls the insertion pressure of the puncture needle to a constant value, and controls the insertion pressure of the puncture needle in response to changes in the insertion speed of the puncture needle. Therefore, the ultrasound diagnostic device 100 according to the first embodiment is able to detect changes in the hardness and ease of insertion inside the living body, and perform puncture accordingly.

Furthermore, according to the first embodiment, the control function 155a controls the display to display information about the pressure and speed when inserting and removing the puncture needle. Therefore, the ultrasound diagnostic device 100 according to the first embodiment makes it possible to present to the observer the detection results of changes in the hardness and ease of insertion inside the living body.

Furthermore, according to the first embodiment, the puncture control function 155e controls the movement of the second robot arm according to a puncture protocol for inserting multiple puncture needles. Therefore, the ultrasound diagnostic apparatus 100 according to the first embodiment makes it possible to stably perform difficult procedures that have traditionally required skill when performed manually. As a result, the frequency with which effective procedures can be performed can be increased.

Second Embodiment
In the first embodiment described above, a case where the puncture needle is inserted automatically or manually is described. In the second embodiment, a case where the puncture needle is inserted using a pseudo-puncture device is described. FIG. 7 is a block diagram showing an example of the configuration of an ultrasound diagnostic apparatus 100 according to the second embodiment. Here, the ultrasound diagnostic apparatus 100 according to the second embodiment differs from the first embodiment in that a pseudo-puncture device 41 is newly connected and in the processing content of the puncture control function 155e. These points will be mainly described below.

As shown in FIG. 7, the pseudo-puncture device 41 is connected to the device main body 15 via a communications interface. The pseudo-puncture device 41 is implemented by a haptic device and simulates the insertion pressure of the puncture needle and the stress of the living body. Specifically, the pseudo-puncture device 41 acquires the pressure on the puncture needle detected by arm 22a (or arm 22b) and operates the haptic device based on the acquired pressure, thereby replicating the insertion pressure of the puncture needle and the stress of the living body. The pseudo-puncture device 41 then accepts operations related to inserting the puncture needle from the operator (doctor) and notifies the puncture control function 155e of the accepted operations. The puncture control function 155e controls arm 22a (or arm 22b) in accordance with the operations notified by the pseudo-puncture device 41.

Figure 8 is a schematic diagram showing an example of a pseudo-puncture device 41 according to the second embodiment. For example, as shown in Figure 8, the pseudo-puncture device 41 has a puncture needle tool that simulates a puncture needle. The operator (doctor) controls arm 22a (or arm 22b) by grasping and manipulating the puncture needle tool in the pseudo-puncture device 41. That is, as shown in the diagram on the right side of Figure 8, the operator (doctor) controls arm 22a (or arm 22b) so that the puncture needle is inserted into the subject by pushing down the puncture needle tool. The operator (doctor) also controls arm 22a (or arm 22b) so that the puncture needle is removed from the subject by pushing up the puncture needle tool, as shown in the diagram on the left side of Figure 8.

At this time, the pseudo-puncture device 41 uses a haptic device to reproduce the insertion pressure of the puncture needle and the stress of the living body, allowing the operator (doctor) to feel the pressure from the puncture needle and the speed of the puncture needle's movement through the sense of touch with the puncture needle tool, and perform the puncture with the same sensation as a normal puncture.

The puncture control function 155e according to the second embodiment controls the puncture by the arm 22a (or arm 22b) in response to an operation input via a pseudo-puncture device that reproduces the state of a puncture needle at the time of puncture. Specifically, the puncture control function 155e controls the arm 22a (or arm 22b) in response to the movement of the puncture needle tool in the pseudo-puncture device 41, and inserts and removes the puncture needle.

Here, the puncture control function 155e can accurately reflect the operation of the puncture needle tool by the operator (doctor) on the arm 22a (or arm 22b). In other words, the puncture control function 155e can control the arm 22a (or arm 22b) so that the actual puncture needle moves by the same amount as the movement of the puncture needle tool by the operator (doctor).

Furthermore, the puncture control function 155e can make various corrections to the operation of the puncture needle tool by the operator (doctor) and control the arm 22a (or arm 22b) with the corrected movement amount. For example, the puncture control function 155e can reproduce the operation after correcting for the operator's (doctor's) hand shake. Furthermore, the puncture control function 155e can reproduce the operation with a corrected movement amount relative to the movement amount actually made by the operator (doctor) to move the puncture needle tool. As an example, the puncture control function 155e can control the arm 22a (or arm 22b) so that the actual puncture needle moves by an amount that is 1/5 (or twice) the amount actually made by the operator (doctor) to move the puncture needle tool. The movement amount correction can be set to various values, not limited to 1/5 or 2 times.

As described above, according to the second embodiment, the puncture control function 155e controls the puncture by the arm 22a (or arm 22b) in response to operations input via the pseudo-puncture device 41, which reproduces the state of a puncture needle at the time of puncture. Therefore, the ultrasound diagnostic device 100 according to the second embodiment makes it possible to perform puncture by a robot arm in a more intuitive manner.

(Other embodiments)
In the above-described embodiment, the case where the information processing device according to the present application is incorporated into the ultrasound diagnostic device 100 has been described. However, the embodiment is not limited to this, and for example, the information processing device according to the present application may be incorporated into the robot arm device 200. In such a case, the above-described functions are executed by a processing circuit of the robot arm device 200.

The term "processor" used in the above description refers to circuits such as a CPU (Central Processing Unit), GPU (Graphics Processing Unit), or an Application Specific Integrated Circuit (ASIC), programmable logic device (e.g., Simple Programmable Logic Device (SPLD), Complex Programmable Logic Device (CPLD), and Field Programmable Gate Array (FPGA)). A processor achieves its functions by reading and executing a program stored in memory. Instead of storing a program in memory, the processor may be configured so that the program is directly embedded in its circuitry. In this case, the processor achieves its functions by reading and executing the program embedded in the circuitry. Each processor in this embodiment is not limited to being configured as a single circuit, but may also be configured as a single processor by combining multiple independent circuits to achieve its functions.

Note that the components of each device illustrated in the above description of the embodiments are conceptual functional units and do not necessarily need to be physically configured as illustrated. In other words, the specific form of distribution and integration of each device is not limited to that illustrated, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions performed by each device can be realized by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

The control method described in the above-mentioned embodiment can be realized by executing a prepared control program on a computer such as a personal computer or workstation. This control program can be distributed via a network such as the Internet. This control program can also be recorded on a non-transitory computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, DVD, USB memory, or flash memory such as SD card memory, and can be executed by being read from the non-transitory recording medium by a computer.

As described above, according to the embodiment, accurate puncture can be easily achieved when puncture assistance is performed using a robotic arm.

While several embodiments have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments may be embodied in a variety of other forms, and various omissions, substitutions, modifications, and combinations of embodiments may be made without departing from the spirit of the invention. These embodiments and their variations are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims.

100 Ultrasound diagnostic device 155 Processing circuit 155a Control function 155c Scanning control function 155d Setting function 155e Puncture control function 155f Positioning function

Claims (16)

a setting unit that sets control information for a puncture needle held by a second robot arm based on puncture information input for an ultrasound image collected by an ultrasound probe held by a first robot arm;
a puncture control unit that controls the movement of the second robot arm that holds the puncture needle based on the control information;
Equipped with
the puncture control unit controls the pressure and speed when inserting and removing the puncture needle, switches between a constant insertion speed and a constant insertion pressure depending on the position from the body surface of the subject to the target , and controls the puncture by the second robot arm depending on an operation input via a pseudo puncture device that reproduces the state at the time of puncture with the puncture needle;
The pseudo-puncture device acquires the pressure applied to the puncture needle detected by the second robot arm, and reproduces the insertion pressure of the puncture needle and the stress of the living body by operating a tactile device based on the acquired pressure . a scanning control unit that receives a manual operation of the ultrasound probe by an operator and controls the first robot arm to support the manual operation of the ultrasound probe;
The information processing device according to claim 1 , wherein the setting unit sets the control information based on puncture information input for an ultrasound image collected by an ultrasound probe held by the first robot arm operated by the manual operation. The information processing device of claim 2, wherein the scanning control unit controls the ultrasound probe to assist the force applied by the operator. The information processing device according to claim 2 or 3, wherein the scanning control unit adjusts the position of the ultrasound probe moved by the manual operation. The information processing device described in any one of claims 1 to 4, wherein the puncture control unit receives manual manipulation of the puncture needle by an operator and controls the second robot arm to assist in the manual manipulation of the puncture needle. The information processing device of any one of claims 1 to 5, wherein the setting unit identifies a position corresponding to the puncture information in spatial coordinates where the first robot arm and the second robot arm operate, and sets control information for the puncture needle based on the identified position. a registration unit that performs registration between the ultrasound image and a reference image;
The information processing device according to any one of claims 1 to 6, wherein the setting unit sets control information for a puncture needle held by a second robot arm based on puncture information input for a reference image that has been aligned with the ultrasound image. a registration unit that performs registration between the ultrasound image and a reference image;
3. The information processing device according to claim 2, wherein the scan control unit controls a movement of the ultrasound probe held by a first robot arm based on operation information input for a reference image that has been aligned with the ultrasound image. The information processing device described in any one of claims 1 to 8, wherein the second robot arm holds a puncture adapter that guides the insertion of the puncture needle. The information processing device according to any one of claims 1 to 9, wherein the puncture control unit controls the puncture speed at a constant speed from the body surface of the subject to a position a predetermined distance away, and controls the puncture pressure at a constant speed from the position a predetermined distance away from the body surface to the target . The information processing device of claim 1, wherein the puncture control unit further controls the puncture needle to increase its insertion speed when the insertion pressure of the puncture needle exceeds a threshold while controlling the puncture needle at a constant insertion speed. The information processing device of claim 1, wherein the puncture control unit further controls the puncture needle to increase its insertion pressure if the insertion speed of the puncture needle falls below a threshold while controlling the puncture needle at a constant insertion pressure. The information processing device according to claim 1 , further comprising a display control unit that controls so as to display information about the pressure and speed at which the puncture needle is inserted and removed. The information processing device according to claim 1 , wherein the puncture control unit controls the movement of the second robot arm in accordance with a puncture protocol for inserting a plurality of puncture needles. the ultrasonic probe;
an image generating unit that generates the ultrasound image based on ultrasound transmitted and received by the ultrasound probe;
An information processing device according to any one of claims 1 to 14 ;
An ultrasound diagnostic device comprising: a setting function for setting control information for a puncture needle held by a second robot arm based on puncture information input for an ultrasound image collected by an ultrasound probe held by a first robot arm;
a puncture control function that controls the movement of the second robot arm holding the puncture needle based on the control information, controls the pressure and speed when inserting and removing the puncture needle, switches between a constant insertion speed and a constant insertion pressure depending on the position from the body surface of the subject to the target , and controls the puncture by the second robot arm in accordance with an operation input via a pseudo puncture device that reproduces the state at the time of puncture with the puncture needle;
The information processing device executes the above .
The pseudo-puncture device acquires the pressure applied to the puncture needle detected by the second robot arm, and reproduces the insertion pressure of the puncture needle and the stress of a living body by operating a tactile device based on the acquired pressure .