JP7822738B2 - Information processing device, information processing method, and program - Google Patents

Description

The disclosure of this specification relates to an information processing device, information processing method, and program that determine data to store in a storage device based on inference results.

Recently, improvements in the performance of medical imaging devices such as CT and MRI have made it possible to generate high-resolution tomographic image data. However, there is a limit to the capacity of storage devices that store medical image data used for interpretation, and it is not possible to generate and store high-resolution tomographic image data for every case.

Patent Document 1 discloses a method for saving high-resolution medical image data of a region of interest specified by the user during scanning, and viewing it together with a lower-resolution image.

JP 2015-84968 A

In Patent Document 1, the user had to specify the region of interest to be saved in high resolution, which was time-consuming. Furthermore, if the person who specified the region of interest during scanning was different from the person who interpreted the image, high-resolution medical image data of the region intended by the interpreter might not have been created.

The information processing device described in the present invention comprises a first image acquisition unit that acquires first medical image data based on imaging data of a subject, a second image acquisition unit that acquires second medical image data different from the first medical image data based on the imaging data, an inference unit that makes a first inference regarding the presence or absence of an abnormality in the subject for the second medical image data, and a determination unit that determines a storage format of the second medical image data to be stored in an external storage device based on the inference result by the inference unit, wherein when the inference unit infers that there is an abnormality in the subject, it further makes a second inference regarding the benign or malignant nature of the abnormality in the subject for the second medical image data, and the determination unit determines the storage format so that the second medical image data that is determined to be an abnormality by the first inference and to be malignant by the second inference, and the first medical image data, are determined to be data to be stored in the storage device .

This invention makes it possible to determine the storage format of data to be saved in a storage device while reducing the user's effort.

FIG. 1 is a functional configuration diagram of a medical image diagnostic system according to a first embodiment. FIG. 1 is a diagram showing an example of a hardware configuration of an information processing apparatus according to a first embodiment. FIG. 4 is a flowchart showing processing of the information processing apparatus according to the first embodiment. FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. FIG. 1 is a block diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. FIG. 10 is a diagram showing correspondence information that associates inference results with types of transmitted images according to the first embodiment. FIG. 10 is a functional configuration diagram of an information processing apparatus according to a second embodiment. FIG. 11 is a diagram showing an example of correspondence information that associates a medical image with an inference result according to the second embodiment. FIG. 11 is a diagram showing an example of a screen display of an image interpretation terminal according to the second embodiment.

Embodiments of the present invention will be described below with reference to the drawings.

[First embodiment]
In the first embodiment, an example in which the present invention is applied to a medical image diagnostic system will be described.

In the medical field, "imaging diagnosis" is performed based on medical image data generated from imaging data obtained by imaging devices such as X-ray CT (Computer Tomography) and MRI (Magnetic Resonance Imaging). The process of observing medical image data and arriving at a diagnosis is called "image interpretation." In image diagnosis, an image interpretation physician, a doctor specializing in image diagnosis, interprets the images at the request of, for example, the attending physician. The image interpretation physician makes a comprehensive judgment based on the findings obtained from the medical image data (hereinafter referred to as "image findings") and various measurements to identify the lesions depicted in the medical image data and the symptoms of the patient. The image interpretation physician then uses the image findings and measurements to write an interpretation report explaining how they arrived at the diagnosis and sends it back to the requesting attending physician.

The medical image diagnostic system according to the first embodiment is composed of an information processing device 100, an imaging device 200, a storage device 300, and an interpretation terminal 400. Figure 1 is a diagram showing an example of the configuration of the medical image diagnostic system. Figure 1 also shows an example of the functional configuration of the information processing device 100.

The imaging device 200 is a device that captures data (signals and images) used in medical treatment. It includes plain X-ray imaging devices (or X-ray devices), CT devices, MRI devices, PET devices, PET/CT devices, SPECT devices, ultrasound imaging devices, fundus cameras (or fundus photography devices), OCT devices, etc., as well as general cameras. The data captured by the imaging device 200 (hereinafter referred to as imaging data) is transmitted to the information processing device 100.

The information processing device 100 is communicably connected to the imaging device 200 and storage device 300, and generates medical image data to be used in image diagnosis based on the imaging data received from the imaging device 200. The generated medical image data is sent to the storage device 300.

The storage device 300 is a device for electronically storing, searching, and communicating medical image data, such as a PACS (Picture Archiving and Communication Systems). It stores medical image data received from the information processing device 100.

The image interpretation terminal 400 is a terminal used by image interpretation physicians for image interpretation. It is communicatively connected to the storage device 300, receives medical image data to be interpreted from the storage device 300, and displays it on an output device such as a monitor. Furthermore, in response to operations by the image interpretation physician, it performs various processes related to viewing and editing images and creating reports.

The configuration of the medical image diagnostic system is not limited to the above. For example, the configuration may include HIS (Hospital Information Systems), which are comprehensive systems that include a medical administrative accounting system, a medical appointment system, a medical information system, etc. Furthermore, the configuration may also include RIS (Radiology Information Systems) that handles things like imaging appointments, image reading management, and material inventory management in the radiology department, as well as a system that manages image reading reports.

Figure 2 shows an example of the hardware configuration of the information processing device 100.

The CPU 11 primarily controls the operation of each component.

Main memory 12 stores the control programs executed by CPU 11 and provides a working area when CPU 11 executes the programs.

The magnetic disk 13 stores programs for implementing various application software, including an operating system (OS), device drivers for peripheral devices, and programs for performing the processes described below. The CPU 11 executes the programs stored in the main memory 12 and the magnetic disk 13 to implement the functions (software) of the information processing device 100 described below in Figure 2.

Display memory 14 temporarily stores display data to be displayed on monitor 15, for example. Monitor 15 is, for example, a CRT monitor or LCD monitor, and displays images, text, etc. based on the data from display memory 14. Mouse 16 and keyboard 17 are used by the user to input pointing and characters, etc., respectively.

The above components are connected to each other via a common bus 18 so that they can communicate with each other.

Note that the configuration of the information processing device 100 is not limited to the above. For example, the information processing device 100 may have multiple processors. It may also have a GPU or an FPGA (Field-Programmable Gate Array) on which some of the processing is programmed.

As shown in FIG. 1, the information processing device 100 includes a photographic data storage unit 101, a first image acquisition unit 102, a second image acquisition unit 103, a transmission unit 104, an inference unit 105, and a determination unit 106.

The imaging data storage unit 101 stores imaging data captured by the imaging device 200.

The first image acquisition unit 102 generates and acquires medical image data or a group of medical image data used in interpretation by reconstruction or other means based on the imaging data. The images acquired by the first image acquisition unit 102 are hereinafter referred to as first medical image data.

The second image acquisition unit 103 generates and acquires medical image data or a group of medical image data to be used in interpretation based on the imaging data. The medical image data acquired by the second image acquisition unit 103 is different from the first medical image data acquired by the first image acquisition unit 102. The medical image data acquired by the second image acquisition unit 103 will be referred to as second medical image data hereinafter.

The transmission unit 104 transmits to the storage device 300 the data (medical image data or a group of medical image data) that has been determined by the determination unit 106 (described below) to be stored in an external storage device. For example, the transmission unit 104 transmits the first medical image data and the second medical image data to the storage device 300.

The inference unit 105 performs inference processing to infer the presence or absence of an abnormality for the second medical image data. The inference unit 105 may perform inference using a trained inference model based on machine learning or deep learning. The inference unit 105 performs inference regarding the diagnosis, findings, and benign/malignant distinction for the abnormality. The inference unit 105 also performs a first inference to infer the presence or absence of an abnormality, and a second inference to distinguish benign/malignant for the inferred abnormality.

The determination unit 106 determines the storage format of the data to be saved in the external storage device 300 based on the inference result of the inference unit 105. Here, the determination unit 106 determines the storage format of whether or not to save the second medical image data in the storage device 300. The medical image data that has been decided to be saved by the determination unit 106 is transmitted to the storage device 300 via the transmission unit 104.

That is, the information processing device 100 includes a first image acquisition unit 102 that acquires first medical image data based on imaging data of the subject, a second image acquisition unit 103 that acquires second medical image data different from the first based on the imaging data, an inference unit 105 that infers the presence or absence of an abnormality in the second medical image data, and a determination unit 106 that determines the data to be stored in the external storage device 300 based on the inference results by the inference unit 105.

In the first embodiment, the imaging data is signal data captured by a CT device, and the first medical image data and second medical image data are tomographic image data obtained by reconstructing the signal data. The first medical image data is thick slice data generated so that the slice thickness is 5 mm, and the second medical image data is thin slice data generated so that the slice thickness is 0.5 mm. The second medical image data is image data with higher resolution than the first medical image data. High-resolution image data refers to data with a larger data volume than the first medical image data due to the slice thickness and resolution.

The inference unit 105 also extracts abnormalities such as lesions from the thin slice data, which is the second medical image data, and infers whether the inferred lesions are benign or malignant. If multiple lesions are extracted, inference may be performed for each of the extracted lesions.

Figure 3 is a flowchart showing the process flow in the information processing device 100, in which the inference unit 105 performs inference processing on acquired medical image data, the determination unit 106 determines the data to be stored in an external storage device, and the transmission unit 104 transmits the data to the storage device 300.

In step S3001, the first image acquisition unit 102 generates and acquires first medical image data from the imaging data. Parameters for generating the medical image data may be set manually by a radiologist or the like.

In step S3002, the second image acquisition unit 103 generates and acquires second medical image data from the captured image data. This processing in the second image acquisition unit 103 is performed automatically by the information processing device 100, and the parameters for generation are set in advance to be optimal as input to the inference unit 105. The generated image is automatically sent to the inference unit 105.

In step S3003, the inference unit 105 applies a first inference process to the second medical image data to infer the presence or absence of an abnormality. It also executes a second inference process to infer the benign or malignant nature of the inferred abnormality. Here, the inference unit 105 does not apply an inference process to infer the presence or absence of an abnormality to the first medical image data. Because the second medical image data has higher resolution than the first medical image data, the inference unit 105 can perform inference based on more detailed features. Note that inference may be performed on medical image data that has been processed or modified from the second medical image data using a trained inference model used by the inference unit 105 or learning data for the inference model. The inference unit 105 sends the results of the inference process to the determination unit 106 and proceeds to the next step.

In step S3004, the determination unit 106 determines the data to be stored in the external storage device 300 based on the inference result of the inference unit 105. The determination unit 106 determines, for example, whether or not to store the second medical image data in the external storage device.

In this embodiment, if a malignant abnormality is present among the abnormalities inferred by the inference unit 105, the determination unit 106 determines that the second medical image data is necessary for subsequent diagnosis and determines that the second medical image data is to be stored in the external storage device 300, transmits the determined information to the transmission unit 104, and proceeds to the next step. The determination unit 106 may also determine that the medical image data (part of the second medical image data) corresponding to the region (region of interest) in the second medical image data where the malignant abnormality is present is to be stored in the external storage device 300. This configuration reduces the amount of data to be stored, thereby preventing pressure on the storage capacity of the storage device 300.

If the inference unit 105 infers that there are no abnormalities in the second medical image data, the decision unit 106 decides not to store the second medical image data in the external storage device 300.

In step S3005, the transmission unit 104 transmits the data determined by the determination unit 106 to be stored in the external storage device 300 to the storage device 300, and then ends the process.

By performing the processing described above, the information processing device 100 can store the second medical image data, which is Thin slice data, in the external storage device 300 when the inference unit 105 infers that a malignant lesion is present in the second medical image data.

Here, the first medical image data, which is thick slice data, is image data that is generally used for interpretation and is sent to the external storage device 300. On the other hand, thin slice data is an image that is used as an auxiliary during interpretation and is referenced when the radiologist wants to observe the lesion in more detail while observing the thick slice data. Thin slice data has a larger capacity than thick slice data, and it is difficult to store it in the storage device 300 for all cases.

In this embodiment, if it is inferred that a malignant lesion is present in the second medical image data, thin slice data can be transmitted to the storage device 300 prior to interpretation by the radiologist. Storing thin slice data in advance allows the radiologist to quickly refer to high-resolution medical image data when they wish to do so. Furthermore, because the determination unit 106 determines the data to be stored in the storage device 300 based on the inference results of the inference unit 105, it is possible to prevent the storage capacity of the storage device 300 from being overwhelmed by unnecessary image data being stored.

(Modification 1 of the first embodiment)
The second image acquisition unit 103 described above generates and acquires, as the second medical image data, tomographic image data with a slice thickness thinner than that of the first medical image data, but other image data may also be generated and acquired as the second medical image data.

The second image acquisition unit 103 may generate and acquire second medical image data by reconstructing imaging data such as signal data so that the slice thickness is the same but the resolution is higher than that of the first medical image data. Furthermore, the second image acquisition unit 103 does not have to generate one type of image data; multiple types of medical image data may be generated in accordance with the input from the inference unit 105. The data from which the second medical image is generated is not limited to signal data; the first medical image data may be used, or a combination of signal data and the first medical image data may be used.

This makes it possible to transmit the generated second medical image data, which is more useful as an aid to interpretation.

(Modification 2 of the First Embodiment)
The transmitting unit 104 described above transmits the first medical image data and the second medical image data to the storage device 300, but may also transmit other medical image data together.

For example, as shown in Figures 4 and 5, the information processing device 100 may further include a third image generation unit 107, and the transmission unit 104 may transmit the third medical image data generated by the third image acquisition unit 107 to the external storage device 300.

Here, Figure 4 shows the functional configuration when a radiologist predicts that third medical image data will be necessary for subsequent diagnosis and generates and acquires it via the third image acquisition unit 107.

On the other hand, Figure 5 shows a case where the third image acquisition unit 107 automatically generates and acquires third medical image data based on the inference result of the inference unit 105. When the third image acquisition unit 107 automatically generates third medical image data, it uses correspondence information such as that shown in Figure 6 (a). The third image acquisition unit 107 determines the medical image data to be generated by referencing the correspondence information based on the inference result of the inference unit 105 for the second medical image data. The third image acquisition unit 107 generates and acquires, as third medical image data, medical image data of a type that has not been acquired by the first image acquisition unit 102 or the second image acquisition unit 103.

This makes it possible to transmit third medical image data, in addition to the second medical image data, to the storage device 300 prior to interpretation.

(Modification 3 of the first embodiment)
The above describes an example in which the explanation determination unit 106 determines whether or not to store the second medical image data in the external storage device 300. However, the determination unit 106 may also determine whether or not to store the first medical image data and the third medical image data. For example, the determination unit 106 refers to the correspondence information shown in FIG. 6A and determines the type of image to store in the external storage device 300 based on the inference result of the inference unit 105. This makes it possible to selectively transmit medical image data useful as an aid to interpretation in accordance with the configuration of the medical image diagnostic system.

(Fourth Modification of the First Embodiment)
Although the inference unit 105 described above further infers whether an abnormality such as a lesion is benign or malignant, other inferences may also be performed. For example, further inferences may be performed on imaging findings for abnormalities such as lesions. Furthermore, the determination unit 106 may have a table such as that shown in FIG. 6B and determine the medical image data to be stored in the external storage device 300 based on the inference results of the inference unit 105. The inference unit 105 may perform both inferences on benign or malignant status and on imaging findings, and the determination unit 106 may determine the medical image data to be stored in the storage device 300 based on these results. For example, in pulmonary CT image data, if a lesion is determined to be malignant and fine imaging findings such as spicules are obtained, the determination unit 106 may determine thin slice data as the medical image data to be stored in the storage device 300. Furthermore, the inference unit 105 may detect lesions, and the determination unit 106 may determine the medical image data to be stored in the external storage device 300 based on the number of lesions.

Furthermore, if there is imaging data older than the imaging data currently being processed, the inference unit 105 may determine changes in the size and shape of the lesion, and the determination unit 106 may determine the medical image data to be stored in an external storage device based on the results.

The inference unit 105 may also use medical information other than the lesion to predict the medical phase, such as "pre-diagnosis," "treatment," or "follow-up observation," and the prognosis, and the determination unit 106 may determine the data to be stored in the external storage device 300 based on the results.

This allows the decision unit 106 to make a more accurate decision on whether to save to the external storage device 300.

(Fifth Modification of the First Embodiment)
The information processing device 100 may have multiple pieces of correspondence information between the inference results of the inference unit 105 and the types of data to be saved in the external storage device 300, as shown in Figures 6(a) and 6(b). For example, correspondence information may be prepared for each radiologist, and the determination unit 106 may change the type of image to be saved depending on the radiologist who will be interpreting the images. This allows the determination unit 106 to transmit medical image data that matches the preferences of the radiologist to the storage device 300 prior to interpretation. Note that a similar effect can be achieved by adding the name or ID of the radiologist to the correspondence table shown in Figures 6(a) and 6(b).

[Second embodiment]
In the first embodiment, the first medical image data and the second medical image data stored in the storage device 300 are viewed by an image-interpreting doctor via the image-interpreting terminal 400 .

In this embodiment, the inference results of the inference unit 105 are further stored in the storage device 300, making it possible to view the first medical image data and the second medical image data more efficiently. In the second embodiment, as in the first embodiment, the first medical image data will be described as thick slice data, and the second medical data as thin slice data. Furthermore, the inference unit 105 performs inference processing on the thin slice data, making a first inference to infer abnormalities such as lesions, and a second inference to infer whether the inferred abnormality is benign or malignant.

Figure 7 is a diagram showing an example of the functional configuration of the information processing device 100 in the second embodiment. In addition to the functions of the information processing device 100 in the first embodiment, it has an image correspondence generation unit 108. Based on the inference results of the inference unit 105, the image correspondence generation unit 108 creates a correspondence table that associates the slice positions of the first medical image and the second medical image, the position and size of the lesion in each slice image data, and the benign/malignant classification results for each lesion. An example of the correspondence table created by the image correspondence generation unit 108 is shown in Figure 8. Note that as long as equivalent information is saved, any storage format is acceptable, and the data may be saved in XML or the like. The correspondence table created by the image correspondence generation unit 108 is associated with the first medical image data and the second medical image data and saved in the storage device 300.

Figure 9 is an example of a screen display on the image interpretation terminal 400. In the second embodiment, the image interpretation terminal 400 acquires the first and second medical image data to be interpreted and the correspondence information shown in Figure 8 from the storage device 300, and performs the following processing by appropriately referencing them.

The image display unit 401 displays slice data of the first medical image data or the second medical image data. The slice data to be displayed can be changed by the slice position adjustment unit 402. If a lesion is present on the displayed slice data and the inferred result of the lesion is malignant, a lesion presentation frame 403 is displayed based on its position and size. Note that the lesion presentation frame 403 may also be displayed when the inferred result of the lesion is benign, in which case the malignant and benign cases can be distinguished by changing the frame color or thickness, for example.

The image selection unit 404 displays selectable previews of the first medical image data and the second medical image data. When a preview is selected, the selected medical image data is displayed on the image display unit 401. At that time, slice data corresponding to the slice position of the medical image data originally displayed is displayed.

Note that the screen display is not limited to the above. For example, a list of lesions may be displayed separately, and slice data containing the selected lesion may be displayed on the image display unit 401.

The processing described above makes it possible to utilize the inference results of the inference unit 105 to efficiently proceed with interpretation by switching between the display of the first medical image data and the second medical image data.

(Modification 1 of the second embodiment)
The image correspondence generation unit 108 described above does not have to be included in the information processing device 100. For example, the image correspondence generation unit 108 may be configured to be included in the storage device 300 or the interpretation terminal 400. In this case, the inference result of the inference unit 105 is stored as is in the storage device 300, and information equivalent to the correspondence information shown in Fig. 8 is generated during interpretation, thereby achieving the same effect.

(Modification 2 of the second embodiment)
The determining unit 106 may determine, based on the inference result by the inference unit 105, partial image data including slices or regions inferred to contain abnormalities as data to be stored in the external storage device 300.

This allows the amount of data stored in the storage device 300 to be kept to a minimum.

[Modification]
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The diagnostic support device in each of the above-described embodiments may be realized as a single device, or multiple devices may be combined to communicate with each other and execute the above-described processing; both are included in the embodiments of the present invention. The above-described processing may also be executed by a common server device or server group. The multiple devices that make up the diagnostic support device and information processing system only need to be able to communicate at a specified communication rate, and do not need to be located in the same facility or the same country.

Embodiments of the present invention include those in which a software program that realizes the functions of the above-described embodiments is supplied to a system or device, and the computer in that system or device reads and executes the code of the supplied program.

Therefore, the program code installed on a computer to implement the processing according to the embodiments is also an embodiment of the present invention. Furthermore, the OS or other software running on the computer may perform some or all of the actual processing based on instructions contained in the program read by the computer, and this processing may also implement the functions of the aforementioned embodiments.

Appropriate combinations of the above-mentioned embodiments are also included in the embodiments of the present invention.

102 First image acquisition unit 103 Second image acquisition unit 105 Inference unit 106 Determination unit

Claims (11)

a first image acquisition unit that acquires first medical image data based on imaging data obtained by imaging the subject;
a second image acquisition unit that acquires second medical image data different from the first medical image data based on the imaging data;
an inference unit that performs a first inference regarding the presence or absence of an abnormality in a subject based on the second medical image data;
a determination unit that determines a storage format of the second medical image data to be stored in an external storage device based on an inference result by the inference unit ,
When the inference unit infers that there is an abnormality in the subject, the inference unit further performs a second inference regarding whether the abnormality in the subject is benign or malignant for the second medical image data;
The information processing device is characterized in that the determination unit determines the storage format so that the second medical image data, which is determined to be abnormal by the first inference and malignant by the second inference, and the first medical image data are determined to be data to be stored in the storage device . 2. The information processing apparatus according to claim 1 , wherein the second medical image data is image data with higher resolution than the first medical image data. 3. The information processing apparatus according to claim 1 , wherein the second medical image data is tomographic image data, and the thickness of the tomographic layer is thinner than that of the first medical image data. 4. The information processing apparatus according to claim 3 , wherein the first medical image data is thick slice data, and the second medical image data is thin slice data. The information processing device according to claim 1 , wherein the inference unit performs the first inference using a trained inference model. The information processing device according to any one of claims 1 to 5, characterized in that the decision unit decides not to store the second medical image data in the external storage device when the inference unit infers that the second medical image data does not contain any abnormalities in the subject. 7. The information processing apparatus according to claim 1 , wherein the determination unit further determines the inference result by the inference unit as data to be stored in the external storage device. 8. The information processing apparatus according to claim 1, further comprising a transmitting unit that transmits the data to be stored in the external storage device determined by the determining unit to the external storage device. 9. The information processing apparatus according to claim 1, wherein the determination unit further determines the captured data as data to be stored in an external storage device. 1. A computer-implemented information processing method, comprising:
a first image acquisition step of acquiring first medical image data based on imaging data obtained by imaging the subject;
a second image acquisition step of acquiring second medical image data different from the first medical image data based on the imaging data;
an inference step of performing a first inference regarding the presence or absence of an abnormality in the subject based on the second medical image data;
a determination step of determining a storage format of the second medical image data to be stored in an external storage device based on an inference result in the inference step ,
the inference step further performs a second inference regarding whether the abnormality in the subject is benign or malignant for the second medical image data when it is inferred that there is an abnormality in the subject;
An information processing method characterized in that the determination step determines the storage format so that the second medical image data, which is determined to be abnormal by the first inference and malignant by the second inference, and the first medical image data are determined to be data to be stored in the storage device . A program for causing a computer to execute the information processing method according to claim 10 .