JP7630952B2 - Information processing system, information terminal, information processing method and program - Google Patents

Description

The present invention relates to an information processing system, an information processing device, an information processing method, and a program that perform inference using a trained model.

A Computer Aided Diagnosis (CAD) system is known that analyzes medical data acquired by a medical imaging device and presents diagnostic support information to doctors. CAD systems apply machine learning technology to medical data, such as medical image data, and output diagnostic support information.

Patent Document 1 discloses that, in order to increase the reliability of analysis based on machine learning, multiple processed medical signals related to a subject are acquired by performing at least one of different imaging methods and different signal processing, and inference is performed for each of the multiple processed medical signals using multiple trained models.

JP 2020-42810 A

In Patent Document 1, inference is performed using multiple trained models for each of multiple processed medical signals, but it is desirable to reduce the data size (amount of data) of the processed medical signals when transmitting and receiving them, and output the inference results quickly.

The present invention aims to provide an information processing system (information terminal) that can quickly output inference results obtained using a trained model.

In order to achieve the object of the present invention, an information processing system includes an information terminal that acquires medical image data from a medical imaging device, and an information processing device that performs inference on the medical image data using a trained model, the information terminal and the information processing device are connected via a network, the information terminal reduces the data size of the medical image data and transmits it to the information processing device, the information processing device performs inference on the medical image data whose data size has been reduced, and transmits a first inference result to the information terminal.

The present invention makes it possible to quickly output inference results obtained using a trained model.

FIG. 1 is a diagram showing a configuration of an information processing system according to the present invention. FIG. 1 is a diagram showing a configuration of an information processing device according to the present invention. FIG. 2 is a diagram showing the configuration of a storage unit in the information processing device of the present invention. 3A and 3B are diagrams showing an example of a screen display form in a display unit of the present invention. FIG. 1 is a diagram showing a configuration of an information terminal according to the present invention. FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. FIG. 2 is a diagram showing an example of a processing mode of a processing unit of the present invention. 4 is a flowchart showing the operation of a learning phase of the present invention. 4 is a flow chart showing the operation of the inference phase of the present invention. FIG. 11 is a diagram showing a second embodiment of the present invention.

A preferred embodiment of the present invention will now be described with reference to the accompanying drawings.

Figure 1 shows the configuration of the information processing system of the present invention.

The information processing system of the present invention comprises an information terminal 110, a network 120, and an information processing device 130. The information terminal 110 may be a single information terminal or multiple information terminals. The medical imaging device 100 images a subject and generates medical image data.

An operation unit 112 and a display unit 114 are connected to the information terminal 110. The operation unit 112 accepts various instructions from an operator and transmits the various instructions to the information terminal 110 and the medical imaging device 100. The operation unit 112 is composed of, for example, a mouse, a keyboard, a button, a panel switch, a foot switch, a trackball, a joystick, etc. The display unit 114 displays a GUI for inputting various instructions in the operation unit 112, and displays medical image data acquired from the medical imaging device 100.

In FIG. 1, the operation unit 112 and the display unit 114 are provided separately for the information terminal 110, but the information terminal 110 may have the functions of the operation unit 112 and the display unit 114 internally.

The medical imaging device 100 is a device that generates medical image data of a subject, such as an X-ray CT (Computed Tomography) device, an MRI (Magnetic Resonance Imaging) device, or an ultrasound diagnostic device.

An X-ray CT device is equipped with an X-ray source and an X-ray detector. While rotating the X-ray source and the X-ray detector around the subject, the X-ray source irradiates the subject with X-rays and the data detected by the X-ray detector is projected to generate CT image data.

The MRI device generates a specific magnetic field for a subject placed in a static magnetic field, and generates MRI image data by performing a Fourier transform on the acquired data.

An ultrasound diagnostic device transmits ultrasound waves to a subject, receives ultrasound waves reflected from the subject, and generates ultrasound image data.

The medical image data (CT image data, MRI image data, ultrasound image data, etc.) generated by the medical imaging device 100 is three-dimensional image data (volume data) or two-dimensional image data. The medical image data also includes raw data. The medical image data may be video image data consisting of multiple frames of data. The medical image data may also include measurement data obtained by performing various measurements using the medical image data.

The medical imaging device 100 is connected to an information terminal 110. The information terminal 110 may be a PC terminal, a mobile phone such as a smartphone, a notebook terminal, a tablet terminal, or the like. The information terminal 110 can also set subject information and can associate medical data acquired from the medical imaging device 100 with the subject information. The information terminal 110 can also display various types of data, such as medical image data and measurement data acquired from the medical imaging device 100.

The information terminal 110 and the information processing device 130 are connected to a network 120. The network 120 includes a communication network outside the hospital, such as wireless communication (Wi-Fi), the Internet, a wireless base station, a provider, a communication line, etc. The network 120 may also include an intranet, which is a communication network within the hospital. The information terminal 110 can communicate with the information processing device 130 via the network 120. The information terminal 110 can transmit medical image data to the information processing device 130. The information processing device 130 can transmit the inference results obtained by performing inference on the medical image data to the information terminal 110.

Figure 2 shows the configuration of the information processing device 130 of the present invention. The information processing device 130 includes a teacher data generation unit 200 that generates teacher data using medical image data, a learning unit 202 that uses the teacher data generated by the teacher data generation unit 200 to learn about the medical image data, a storage unit 204 that stores the trained model generated by the learning unit 202, and an inference unit 206 that uses the trained model to perform inference on the medical image data.

The components (functions) of the information processing device 130 are realized by, for example, a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) executing a program (software) stored in memory. The information processing device 130 has a processor and memory inside. The processor can execute each process of the information processing device 130 according to the program stored in the memory, and can function as a teacher data generation unit 200, a learning unit 202, a memory unit 204, an inference unit 206, etc.

The teacher data generating unit 200 is connected to the network 120 and can acquire medical image data. The teacher data generating unit 200 generates teacher data using the medical image data. The teacher data generated by the teacher data generating unit 200 is determined according to the inference task performed by the neural network and the target of classification.

Inference tasks performed by neural networks include classification tasks to classify the classes of medical image data, detection tasks to detect what is captured in what position in the medical image data, and segmentation tasks to extract target areas from medical image data.

When training a neural network to perform a classification task, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct answer label, which is a label indicating an object that appears in the medical image data.

On the other hand, when performing a detection task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct answer image to which a ROI (Region of Interest) indicating the position of an object reflected in the medical image data and a correct answer label that is a label indicating the object have been assigned.

When performing a segmentation task using a neural network, the teacher data generation unit 200 generates teacher data that pairs medical image data with a correct image to which position information of pixels of a target area reflected in the medical image data and a correct label that indicates the target area have been assigned.

For example, a neural network is trained to perform the task of segmenting the presence or absence, type, and area of a lesion from medical image data acquired from the information terminal 110. In this case, the training data generating unit 200 generates training data that pairs medical image data having a lesion area with information on a correct answer label indicating the type of lesion for the medical image data and a correct answer image with position information of the pixels of the lesion.

The teacher data generating unit 200 may also perform preprocessing of medical image data depending on the neural network that is learning in the learning unit 202. For example, when the subject of inference by the neural network is medical image data, the acquired medical image data is subjected to processes such as noise removal, filtering, image cropping, and resolution change. When the subject of inference is natural language such as text, the teacher data generating unit 200 performs preprocessing of the data to be processed, such as performing morphological analysis and applying vector conversion technology, depending on the subject of inference by the neural network and the task.

In FIG. 2, the teacher data generating unit 200 is shown to be included inside the information processing device 130, but the information terminal 110 may also include the teacher data generating unit 200. In other words, the teacher data generating unit 200 may be included as part of the information terminal 110. For example, after the teacher data as described above is generated by the information terminal 110, the inference device may be trained by the learning unit 202 of the information processing device 130 via the network 120.

The learning unit 202 is connected to the teacher data generation unit 200. The learning unit 202 generates a trained model by using a neural network to learn by associating medical image data with teacher data. Here, the trained model indicates parameters determined by performing a learning process up to a predetermined standard and information on a model corresponding to the parameters. Note that the trained model may be used to train another model as transfer learning, or further learning processes may be performed on the trained model.

A neural network includes multiple layers. Among neural networks, CNN (Convolutional Neural Network), which is a type of deep learning technology, has multiple intermediate layers between the input layer and the output layer (not shown). The multiple intermediate layers include a convolutional layer, a pooling layer, an upsampling layer, and a synthesis layer. The convolutional layer is a layer that performs convolution processing on a group of input values. In the convolutional layer, the input medical image data is convolved to extract features of the medical image data.

The pooling layer is a layer that performs processing to make the number of output value groups less than the number of input value groups by thinning out or combining input value groups. The upsampling layer is a layer that performs processing to make the number of output value groups greater than the number of input value groups by duplicating input value groups or adding values interpolated from the input value groups. The composition layer is a layer that inputs value groups, such as the output value groups of a layer or pixel values that make up medical image data, from multiple sources and combines them by concatenating or adding them. The number of intermediate layers can be changed at any time depending on the learning content.

The memory unit 204 is connected to the learning unit 202. The memory unit 204 stores a trained model that has been trained to extract the type and area of a lesion in medical image data, for example. The trained model is generated using a neural network, for example. However, in addition to CNN and RNN (Recurrent Neural Network), which are deep learning technologies among neural network technologies, and models derived from CNN and RNN, other machine learning technologies such as support vector machines, logistic regression, and random forests may also be used, or a rule-based method may also be used.

The inference unit 206 is connected to the network 120 and can acquire medical image data and measurement data. The inference unit 206 is connected to the memory unit 204 and can perform inference using the trained model stored in the memory unit 204.

The inference unit 206 performs inference on the newly generated medical image data, for example, using a trained model that has been trained to extract the type and area of a lesion in the medical image data. If the newly generated medical image data contains a lesion, the inference unit 206 can output the type and area of the lesion.

Figure 3 shows the configuration of the storage unit 204 of the information processing device 130 of the present invention. The storage unit 204 associates and stores the trained model trained in the learning unit 202 with the incidental information that specifies the type of trained model. Specifically, the storage unit 204 assigns the first incidental information 310 to the first trained model 300 trained in the learning unit 202, and stores the first trained model 300. The storage unit 204 assigns the second incidental information 312 to the second trained model 302 trained in the learning unit 202, and stores the second trained model 302. The storage unit 204 assigns the third incidental information 314 to the third trained model 304 trained in the learning unit 202, and stores the third trained model 304.

In FIG. 3, three trained models are stored in the storage unit 204 together with the accompanying information, but the storage unit 204 can also store four or more trained models together with the accompanying information. In addition, each of the multiple trained models differs in at least one of the following: characteristics of trained medical image data, characteristics of teacher data, diagnostic purpose, class indicating classification target, model structure, etc. Based on the accompanying information assigned to each trained model, it is possible to specify each trained model or identify the trained model. In addition, the inference unit 206 in the information processing device 130 can select a trained model according to the characteristics of medical image data acquired from the information terminal 110, and use the selected trained model to perform inference on the medical image data acquired from the information terminal 110.

The additional information includes the type of trained model. The inference unit 206 can identify the type of trained model by referring to the additional information. The additional information includes information that identifies at least one of the characteristics of trained medical image data, the characteristics of training data, the diagnostic purpose, the class indicating the classification target, and the model structure.

Figure 4 shows an example of a screen display form in the display unit 114 of the present invention. The display unit 114 displays a selection screen 400 for selecting the type of trained model to be used by the inference unit 206. In other words, the selection screen 400 is a selection unit for selecting the trained model to be used by the inference unit 206. Icons suitable for diagnostic purposes are displayed on the selection screen. By selecting each icon, the operator can select a trained model for detecting pneumonia, a trained model for detecting pulmonary nodules, a trained model for detecting lung cancer, a trained model for detecting emphysema, a trained model for detecting pleural effusion, a trained model for detecting pneumothorax, and the like. In addition, the operator can select all trained models by selecting the ALL icon.

Here, icons corresponding to pneumonia, pulmonary nodules, and pneumothorax are selected. The inference unit 206 can identify trained models corresponding to pneumonia, pulmonary nodules, and pneumothorax by referring to the additional information (diagnostic purpose). In this way, a trained model for detecting pneumonia, pulmonary nodules, and pneumothorax is selected.

The display unit 114 also displays a medical image 402 based on the medical image data. The medical image 402 is a plurality of slice images of a CT image (three-dimensional volume data). The inference unit 206 performs inference using a trained model corresponding to pneumonia, pulmonary nodules, and pneumothorax. As shown in FIG. 4, for example, when the inference unit 206 detects a pneumothorax in a CT image, the information processing device 130 transmits detection information and an area in which the pneumothorax is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the pneumothorax is detected. The display unit 114 also displays an area 404 in which the pneumothorax is detected. That is, when the inference unit 206 in the information processing device 130 detects a lesion in the medical image data, the information processing device 130 transmits detection information in which the lesion is detected to the information terminal 110. The display unit 114 displays detection information 406 in which the lesion is detected.

Figure 5 shows the configuration of an information terminal 110 of the present invention. The information terminal 110 has functions such as thinning medical image data, performing pre-processing for inference, and transmitting and receiving information.

The information terminal 110 has a function of reducing the data size of medical image data. The information terminal 110 includes an image acquisition unit 502 that acquires medical image data from the medical imaging device 100, a processing unit 504 that performs thinning processing on the medical image data, a transmission unit 506 that transmits the medical image data processed by the processing unit 504 to an external device such as the information processing device 130 via the network 120, and a reception unit 508 that receives information transmitted from the external device such as the information processing device 130 via the network 120. The display unit 114 displays a medical image based on the medical image data and information transmitted from the external device such as the information processing device 130.

The processing unit 504 performs thinning processing on the first medical image data acquired by the image acquisition unit 502, and generates second medical image data having a data size smaller than the data size (data amount) of the first medical image data. The processing unit 504 generates third medical image data, which is the remaining medical image data obtained by subtracting the second medical image data from the first medical image data. Here, the processing unit 504 generates multiple medical image data (second medical image data, third medical image data) having a data size smaller than the first medical image data from the first medical image data.

The processing unit 504 outputs second medical image data generated by performing a thinning process on the first medical image data to the transmitting unit 506. The processing unit 504 then outputs third medical image data, which is the remaining medical image data obtained by subtracting the second medical image data from the first medical image data, to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data to an external device such as the information processing device 130, and then transmits the third medical image data to the external device such as the information processing device 130. Note that the transmitting unit 506 may transmit the second medical image data to an external device such as the information processing device 130, and then transmit the first medical image data to an external device such as the information processing device 130.

As shown in FIG. 2, the information processing device 130 (inference unit 206) receives the second medical image data transmitted from the information terminal 110 (transmission unit 506) via the network 120, and performs inference on the second medical image data. The inference unit 206 performs inference on the second medical image data using the trained model stored in the memory unit 204.

The inference unit 206 can identify the type of trained model by referring to the supplementary information described above. For example, the inference unit 206 identifies the type of trained model using characteristics of trained medical image data. The inference unit 206 can identify a trained model suitable for thinned medical image data. The inference unit 206 performs inference on the second medical image data using the trained model identified by the supplementary information.

The inference unit 206 extracts lesion information (lesion type and area) from the second medical image data. The inference unit 206 transmits the inference result obtained by performing inference on the second medical image data to the information terminal 110 via the network 120. If the second medical image data contains a lesion, the inference unit 206 transmits the lesion information (lesion type and area). If the second medical image data does not contain a lesion, the inference unit 206 transmits information indicating that the second medical image data does not contain a lesion. The information terminal 110 (receiving unit 508) receives the inference result obtained by performing inference on the second medical image data (lesion information).

The display unit 114 displays the inference results obtained by performing inference on the thinned second medical image data together with the medical image based on the unthinned first medical image data. Here, the display unit 114 does not need to display the medical image based on the thinned second medical image data.

For example, assume that the first medical image data is a 1024 x 1024 matrix with 1000 slice images, and the thinned second medical image data is a 1024 x 1024 matrix with 500 slice images. Since the second medical image data has about half the data size (amount of data) of the first medical image data, the transfer time of the second medical image data is shorter than the transfer time of the first medical image data. By transmitting the second medical image data, which has a data size smaller than the data size (amount of data) of the first medical image data, to the information processing device 130 (inference unit 206) and performing inference, the inference result can be output quickly.

The inference unit 206 also identifies a trained model suitable for medical image data obtained by integrating the second medical image data and the third medical image data. The inference unit 206 performs inference on the medical image data obtained by integrating the second medical image data and the third medical image data using the trained model suitable for unthinned medical image data. The display unit 114 displays a first inference result obtained by inference on the second medical image data, and then displays a second inference result obtained by inference on the medical image data obtained by integrating the second medical image data and the third medical image data. This makes it possible to output accurate inference results.

Figures 6 and 7 show the processing form of the processing unit 504 of the present invention. Here, it is assumed that the medical image data is three-dimensional volume data. The three-dimensional volume data is made up of multiple slice images (frame images). Here, for simplicity of illustration, the explanation will be given assuming eight slice images.

As shown in FIG. 6(a), the processing unit 504 performs thinning processing on the first medical image data 600 to generate second medical image data 602 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects odd-numbered slice images and generates second medical image data 602. In FIG. 6(a), the selected slice images are shown in white, and unselected slice images are shown in gray. The processing unit 504 generates third medical image data 604, which is the remaining medical image data obtained by subtracting the second medical image data 602 from the first medical image data 600. The processing unit 504 generates the third medical image data 604 from the even-numbered slice images.

The processing unit 504 outputs second medical image data 602 consisting of odd-numbered slice images to the transmission unit 506. The transmission unit 506 transmits the second medical image data 602 to the information processing device 130.

The inference unit 206 performs inference on the second medical image data 602 using the trained model stored in the memory unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for the thinned medical image data.

Here, the supplementary information of the trained model stored in the memory unit 204 includes the characteristics of the trained medical image data, and the characteristics of the medical image data include at least one of data reduction information, thinning information, data size, matrix size, number of slice images, area information, etc.

The data size reduction information is information that indicates the extent to which the data size has been reduced compared to the original medical image data. For example, if the original medical image data has been reduced to 1/2 size, the data size reduction information is 1/2. If the original medical image data has been reduced to 1/3 size, the data size reduction information is 1/3. Furthermore, the thinning information is information that indicates the extent to which the original medical image data has been thinned. For example, if frames of the original medical image data are selected alternately, the thinning information is 1/2.

For example, the learning unit 202 generates a trained model by previously learning the thinned medical image data in association with the teacher data. The memory unit 204 then stores the trained model together with the accompanying information. The accompanying information is information indicating that the training was performed using the thinned medical image data. Since the second medical image data 602 is thinned medical image data, the inference unit 206 performs inference on the second medical image data 602 using the trained model identified by the accompanying information.

The inference unit 206 transmits the inference results obtained by performing inference on the second medical image data 602 to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the thinned second medical image data 602 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference results for the medical image data.

Then, the transmission unit 506 transmits the thinned third medical image data 604 to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 obtained by integrating the second medical image data 602 and the third medical image data 604. The information processing device 130 performs processing to return the thinned second medical image data 602 and the third medical image data 604 to the original first medical image data 600.

The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since the inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

In this way, the first inference result obtained by performing inference on the thinned second medical image data 602 and the second inference result obtained by performing inference on the unthinned first medical image data 600 are displayed on the display unit 114 in two stages. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

As shown in FIG. 6B, the processing unit 504 performs thinning processing on the first medical image data 600 to generate second medical image data 612 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects slice images in a predetermined range (near the center) and generates second medical image data 612. The area near the center of the medical image data is an area where lesions tend to be concentrated in the medical image data. In FIG. 6B, the selected slice images are shown in white, and the unselected slice images are shown in gray. The processing unit 504 generates third medical image data 614, which is the remaining medical image data obtained by subtracting the second medical image data 612 from the first medical image data 600. The processing unit 504 generates the third medical image data 614 from slice images near the edge.

The processing unit 504 outputs second medical image data 612 consisting of slice images in a predetermined range (near the center) to the transmitting unit 506. The processing unit 504 then outputs third medical image data 614 consisting of slice images outside the predetermined range (near the end) to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 612 to the information processing device 130.

The inference unit 206 performs inference on the second medical image data 612 using the trained model stored in the memory unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information.

The second medical image data 612 is thinned medical image data, but the slice interval is the same as that of the first medical image data 600, so the inference unit 206 performs inference on the second medical image data 612 using a trained model suitable for unthinned medical image data.

The inference unit 206 transmits the inference result obtained by performing inference on the second medical image data 612 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the second medical image data 612 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference result.

Then, the transmission unit 506 transmits the third medical image data 614 to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 that is the combination of the second medical image data 612 and the third medical image data 614.

The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since the inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

In this way, the first inference result obtained by performing inference on the second medical image data 612 in a predetermined range (near the center) and the second inference result obtained by performing inference on the first medical image data 600 are displayed on the display unit 114 in two stages. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

Furthermore, as shown in FIG. 7(a), the processing unit 504 performs a matrix size reduction process on the first medical image data 600. The processing unit 504 generates second medical image data 622 having slice images with a matrix size smaller than the matrix size of the slice images constituting the first medical image data 600. The processing unit 504 generates third medical image data 624 having slice images with a matrix size larger than the matrix size of the slice images constituting the second medical image data 622. For example, if the slice images constituting the first medical image data 600 have a size of 1024×1024 matrix, the slice images constituting the second medical image data 622 have a size of 512×512 matrix. The slice images constituting the third medical image data 624 have a size of 768×768 matrix.

Figure 7 (a) shows second medical image data 622 and third medical image data 624, which are slice images that have undergone matrix size reduction processing.

The processing unit 504 outputs the second medical image data 622 to the transmission unit 506. Then, the processing unit 504 outputs the third medical image data 624 to the transmission unit 506. The transmission unit 506 transmits the second medical image data 622 to the information processing device 130.

The inference unit 206 performs inference on the second medical image data 622 using the trained model stored in the memory unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model that is suitable for the reduced matrix size.

The memory unit 204 stores the trained model together with additional information (matrix size). The additional information is information indicating that the model has been trained using medical image data composed of slice images of a specified matrix size. The inference unit 206 performs inference on the second medical image data 622 using the trained model identified by the additional information.

The inference unit 206 transmits the inference results obtained by performing inference on the second medical image data 622 consisting of slice images that have been subjected to a matrix size reduction process to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the second medical image data 622 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference results.

Then, the transmitting unit 506 transmits to the information processing device 130 third medical image data 624 having a slice image with a matrix size different from the matrix size of the slice image of the second medical image data 622. The matrix size of the slice image of the third medical image data 624 is larger than the matrix size of the slice image of the second medical image data 622. The inference unit 206 performs inference on the third medical image data 624. The inference unit 206 performs inference on the third medical image data 624 using a trained model identified by the supplementary information (matrix size).

The inference unit 206 transmits the inference result obtained by performing inference on the third medical image data 624 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the third medical image data 624 together with the medical image based on the first medical image data 600. Since inference is performed on the third medical image data 624 having a slice image with a matrix size larger than the matrix size of the slice images constituting the second medical image data 622, a more accurate inference result can be displayed. The operator can recognize the more accurate inference result.

In this way, a first inference result obtained by performing inference on second medical image data 622 having slice images of a second matrix size (512 x 512) and a second inference result obtained by performing inference on third medical image data 624 having slice images of a third matrix size (768 x 768) are displayed in two stages on the display unit 114. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

Also, as shown in FIG. 7(b), the processing unit 504 performs area extraction processing on the first medical image data 600 to generate second medical image data 632 having a data size smaller than that of the first medical image data 600. The processing unit 504 selects a predetermined area (central area) of each slice image to generate the second medical image data 632. In FIG. 7(b), the selected image area is shown in white, and the unselected image area is shown in gray. The processing unit 504 generates third medical image data 634, which is the remaining medical image data obtained by subtracting the second medical image data 632 from the first medical image data 600. The processing unit 504 generates the third medical image data 634 consisting of slice images excluding the predetermined area (central area).

The processing unit 504 outputs second medical image data 632 consisting of slice images of the specified region (central region) to the transmitting unit 506. The processing unit 504 then outputs third medical image data 634 consisting of slice images excluding the specified region (central region) to the transmitting unit 506. The transmitting unit 506 transmits the second medical image data 632 to the information processing device 130.

The inference unit 206 performs inference on the second medical image data 632 using the trained model stored in the memory unit 204. The inference unit 206 identifies the type of trained model by referring to the above-mentioned supplementary information. The inference unit 206 identifies a trained model suitable for a specified area.

The memory unit 204 stores the trained model together with the additional information (region information). The additional information is information indicating that training has been performed using medical image data consisting of slice images of a specified region. Since the second medical image data 632 is medical image data consisting of slice images of a specified region (central region), the inference unit 206 performs inference on the second medical image data 632 using the trained model identified by the additional information.

The inference unit 206 transmits the inference results obtained by performing inference on the second medical image data 632 to the information terminal 110 via the network 120. The display unit 114 displays the inference results obtained by performing inference on the second medical image data 632 together with the medical image based on the first medical image data 600. The operator can quickly recognize the inference results.

Then, the transmission unit 506 transmits the third medical image data 634 consisting of slice images excluding the specified region (central region) to the information processing device 130. The inference unit 206 performs inference on the first medical image data 600 obtained by integrating the second medical image data 632 and the third medical image data 634.

The inference unit 206 identifies a trained model suitable for unthinned medical image data. The inference unit 206 performs inference on the first medical image data 600 using the trained model suitable for unthinned medical image data.

The inference unit 206 transmits the inference result obtained by performing inference on the first medical image data 600 to the information terminal 110 via the network 120. The display unit 114 displays the inference result obtained by performing inference on the first medical image data 600 together with a medical image based on the first medical image data 600. Since the inference is performed on the first medical image data 600 that has not been thinned out, it is possible to display a more accurate inference result. The operator can recognize the more accurate inference result.

In this way, the first inference result obtained by performing inference on the second medical image data 632 consisting of slice images of a specified region (central region), and the second inference result obtained by performing inference on the first medical image data 600 are displayed on the display unit 114 in two stages. The operator can quickly recognize the inference result in the initial stage, and can recognize the accurate inference result in the next stage.

The operation of the learning phase of the present invention will be explained using Figure 8.

S800: The medical imaging device 100 captures an image of the subject and acquires medical image data. The information terminal 110 (image acquisition unit 502) acquires the medical image data from the medical imaging device 100.

S802: The operator selects whether or not to reduce the data size (amount of data) of the medical image data via the operation unit 112. If the data size of the medical image data is not to be reduced, proceed to S804. If the data size of the medical image data is to be reduced, proceed to S810.

S804: The information terminal 110 (transmission unit 506) transmits the medical image data to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs the medical image data with a correct answer label. The information processing device 130 (learning unit 202) generates a trained model by learning the medical image data in association with the teacher data using a neural network.

S806: The information processing device 130 (storage unit 204) stores the trained model trained by the training unit 202 and the associated information.

S810: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical image data acquired by the image acquisition unit 502. The processing unit 504 generates medical image data with a data size smaller than the data size (data amount) of the medical image data. This processing generates medical image data with multiple data sizes.

S812: The information terminal 110 (transmission unit 506) transmits medical image data of multiple data sizes to the information processing device 130. The information processing device 130 (teacher data generation unit 200) generates teacher data that pairs each medical image data with a correct answer label. The information processing device 130 (learning unit 202) uses a neural network to learn by associating each medical image data with the teacher data, thereby generating multiple trained models.

S814: The information processing device 130 (storage unit 204) stores multiple trained models and associated information trained by the training unit 202, as shown in FIG. 3.

The operation of the inference phase of the present invention will be explained using Figure 9.

S900: The medical imaging device 100 captures an image of the subject and acquires medical image data. The information terminal 110 (image acquisition unit 502) acquires the medical image data from the medical imaging device 100.

S902: The operator selects whether or not to reduce the data size (amount of data) of the medical image data via the operation unit 112. If the data size of the medical image data is not to be reduced, proceed to S904. If the data size of the medical image data is to be reduced, proceed to S910.

S904: The information terminal 110 (transmission unit 506) transmits the medical image data to the information processing device 130. The information processing device 130 (inference unit 206) performs inference on the medical image data using the trained model stored in the memory unit 204.

S906: The information processing device 130 (inference unit 206) transmits the inference result obtained by performing inference on the medical image data to the information terminal 110. The display unit 114 displays the inference result obtained by performing inference on the medical image data together with the medical image based on the medical image data.

S910: The information terminal 110 (processing unit 504) performs data size reduction processing on the medical image data acquired by the image acquisition unit 502. The processing unit 504 generates medical image data with a data size smaller than the data size (data amount) of the medical image data. This processing generates medical image data with multiple data sizes.

S912: The information terminal 110 (the inference unit 206) performs multiple inferences on the multiple medical image data using the multiple trained models stored in the memory unit 204. The inference unit 206 identifies a trained model suitable for medical image data whose data size has been reduced. The inference unit 206 also identifies a trained model suitable for medical image data whose data size has not been reduced. The inference unit 206 performs inferences on the multiple medical image data using the trained models suitable for the multiple medical image data.

S914: The display unit 114 displays a first inference result obtained by performing inference on the medical image data whose data size has been reduced, together with a medical image based on the medical image data. Then, the display unit 114 displays a second inference result obtained by performing inference on the medical image data whose data size has not been reduced. The display unit 114 displays the inference results in the order in which they were transmitted from the information terminal 110 (inference unit 206).

As described above, the information processing system of the present invention includes an information terminal 110 that acquires medical image data from a medical imaging device 100, and an information processing device 130 that performs inference on the medical image data using a trained model, the information terminal 110 and the information processing device 130 being connected via a network 120, the information terminal 110 reducing the data size of the medical image data and transmitting it to the information processing device 130, the information processing device 130 performing inference on the medical image data with the reduced data size and transmitting a first inference result to the information terminal 110. Thus, the operator can quickly recognize the inference result at an early stage.

The information terminal 110 transmits the remaining medical image data that has not been transmitted to the information processing device 130 to the information processing device 130, and the information processing device 130 performs inference on the medical image data that is an integration of the medical image data with the reduced data size and the remaining medical image data, and transmits the second inference result to the information terminal 110. Therefore, the operator can recognize the accurate inference result in the next stage.

An information processing system (information processing device 130) according to a second embodiment of the present invention will be described with reference to FIG. 10. The difference from the first embodiment is that the information processing system includes an integration processing unit 210 that integrates multiple inference results. The integration processing unit 210 includes an inference result storage unit 800 that stores multiple inference results inferred by the inference unit 206, and an addition processing unit 802 that adds up the multiple inference results.

The inference unit 206 performs inference on the second medical image data with reduced data size using the trained model stored in the memory unit 204. The inference unit 206 transmits a first inference result (initial inference result) obtained by performing inference on the second medical image data to the information terminal 110 via the network 120. The display unit 114 displays the first inference result obtained by performing inference on the second medical image data with reduced data size.

The inference unit 206 also transmits the first inference result obtained by performing inference on the second medical image data to the integration processing unit 210. The inference result storage unit 800 stores the first inference result.

The inference unit 206 performs inference on the first medical image data whose data size has not been reduced, using the trained model stored in the memory unit 204. The inference unit 206 transmits a second inference result (next inference result) obtained by performing inference on the first medical image data to the integrated processing unit 210. The inference result memory unit 800 stores the second inference result. Here, the inference result memory unit 800 stores the first inference result and the second inference result, but may store three or more inference results.

The addition processing unit 802 performs an addition process on multiple inference results (first inference result and second inference result) stored in the inference result storage unit 800. At this time, the objects of addition in the addition processing unit 802 are second medical image data generated from first medical image data obtained by imaging the subject, and multiple inference results obtained by inferring the first medical image data. In other words, the objects of addition in the addition processing unit 802 are multiple inference results obtained by inferring medical image data (multiple items) whose original medical image data is the same.

The addition processing unit 802 can adjust the addition weight. The addition processing unit 802 performs the addition process by weighting the second inference result heavier than the weighting of the first inference result. This is because the second inference result is inferred from the first medical image data whose data size has not been reduced.

The integration processing unit 210 transmits the inference result, in which the multiple inference results are integrated, to the information terminal 110 via the network 120. The display unit 114 displays the integrated inference result.

As described above, the information processing system of the present invention includes an integration processing unit 210 that integrates multiple inference results. By integrating multiple inference results, the inference accuracy can be improved.

A computer program that realizes the functions of the above-mentioned embodiments can be supplied to a computer via a network or memory (not shown), and the computer program can be executed by a processor (not shown). This is a computer program that causes a computer to execute the above-mentioned information processing method. In other words, the computer program is a program that causes a computer to realize the functions of an information processing device. The memory stores the computer program.

REFERENCE SIGNS LIST 100 Medical imaging device 110 Information terminal 112 Operation unit 114 Display unit 120 Network 130 Information processing device 200 Teacher data generation unit 202 Learning unit 204 Storage unit 206 Inference unit 502 Image acquisition unit 504 Processing unit 506 Transmission unit 508 Reception unit

Claims (18)

an information terminal for acquiring medical image data from a medical imaging device;
an information processing device having an inference unit that performs inference on the medical image data using a trained model;
the information terminal and the information processing device are connected via a network,
the information terminal reduces the data size of the medical image data and transmits the data to the information processing device;
the inference unit performs inference on the medical image data whose data size has been reduced, and transmits a first inference result to the information terminal;
An information processing system characterized in that the information terminal transmits remaining medical image data that has not been transmitted to the information processing device to the information processing device, the inference unit performs inference on the medical image data, and the information processing device transmits a second inference result to the information terminal. The information processing system according to claim 1, characterized in that the inference unit selects a trained model according to the characteristics of the medical image data acquired from the information terminal, and performs inference on the medical image data using the selected trained model. The information processing system according to claim 1, further comprising a storage unit that stores the trained model in association with additional information that identifies the type of the trained model, and the inference unit identifies the type of the trained model by referring to the additional information. The information processing system according to claim 3, characterized in that the supplementary information includes information that identifies at least one of the characteristics of the learned medical image data, the characteristics of the training data, the diagnostic purpose, the class indicating the classification target, and the model structure. The information processing system according to claim 1, characterized in that, when the inference unit detects a lesion in the medical image data, the information processing device transmits detection information of the detected lesion to the information terminal. The information processing system according to claim 5, further comprising a display unit that displays the detection information transmitted to the information terminal. The information processing system according to claim 1, characterized in that the information terminal includes an image acquisition unit that acquires the medical image data from the medical imaging device, and a processing unit that performs thinning processing on the medical image data. The information processing system according to claim 7, characterized in that the processing unit performs a thinning process on the first medical image data acquired by the image acquisition unit to generate second medical image data having a data size smaller than the data size of the first medical image data, generates third medical image data which is the remaining medical image data obtained by subtracting the second medical image data from the first medical image data, and transmits the second medical image data and the third medical image data to the information processing device. The information processing system according to claim 8, characterized in that the inference unit uses the trained model to perform inference on the second medical image data to generate a first inference result, and to perform inference on the first medical image data that is an integration of the second medical image data and the third medical image data to generate a second inference result. The information processing system according to claim 8, further comprising a display unit that displays a first inference result obtained by performing inference on the second medical image data together with a medical image based on the first medical image data. The information processing system according to claim 10, characterized in that the display unit displays a first inference result obtained by performing inference on the second medical image data, and then displays a second inference result obtained by performing inference on the first medical image data that is an integrated version of the second medical image data and the third medical image data. The information processing system according to claim 2 or 9, further comprising an integration processing unit that integrates the first inference result and the second inference result. The information processing system according to claim 12, characterized in that the integration processing unit includes an inference result storage unit that stores multiple inference results inferred by the inference unit, and an addition processing unit that adds up the multiple inference results. an information terminal for acquiring medical image data from a medical imaging device;
an information processing device having an inference unit that performs inference on the medical image data using a trained model;
the information terminal and the information processing device are connected via a network,
the information terminal reduces the data size of the medical image data and transmits the data to the information processing device;
the information processing device performs inference on the medical image data whose data size has been reduced, and transmits a first inference result to the information terminal;
An information processing system characterized in that the information terminal transmits other medical image data contained in the medical image data that has not been transmitted to the information processing device to the information processing device, and the information processing device performs inference on the other medical image data and transmits a second inference result to the information terminal. an information terminal for acquiring medical image data from a medical imaging device;
an information processing device having an inference unit that performs inference on the medical image data using a trained model;
the information terminal and the information processing device are connected via a network,
the information terminal reduces the data size of the medical image data and transmits the data to the information processing device;
The information processing device is an information processing system that performs inference on the medical image data whose data size has been reduced and transmits a first inference result to the information terminal,
the information terminal includes an image acquisition unit that acquires first medical image data from the medical imaging device, and a processing unit that performs a thinning process on the first medical image data;
An information processing system characterized in that the processing unit performs a thinning process on the first medical image data acquired by the image acquisition unit, generates second medical image data having a data size smaller than the data size of the first medical image data, and generates third medical image data which is the remaining medical image data by subtracting the second medical image data from the first medical image data, and the second medical image data and the third medical image data are transmitted to the information processing device. An information terminal connected via a network to an information processing device that performs inference on medical image data,
an acquisition unit for acquiring medical image data from a medical imaging device;
A processing unit that reduces a data size of the medical image data and generates the medical image data having the reduced data size;
a transmission unit that transmits the medical image data whose data size has been reduced to the information processing device;
a receiving unit that receives, via the network, from the information processing device, a first inference result in which inference has been performed by the information processing device on the medical image data whose data size has been reduced;
The transmission unit transmits remaining medical image data that has not been transmitted to the information processing device to the information processing device,
The information terminal is characterized in that the receiving unit receives a second inference result, which is an inference performed by the information processing device on the medical image data, from the information processing device via the network. acquiring medical image data from a medical imaging device;
and performing inference on the medical image data using the trained model,
Reducing the data size of the medical image data;
transmitting the medical image data whose data size has been reduced from the information terminal to the information processing device;
A step of performing inference on the medical image data whose data size has been reduced by the information processing device, and transmitting a first inference result to the information terminal;
transmitting the remaining medical image data that has not been transmitted to the information processing device to the information processing device, causing the information processing device to perform inference on the medical image data, and transmitting a second inference result to the information terminal;
13. An information processing method comprising:
An information processing method comprising the steps of: A program causing a computer to execute the information processing method according to claim 17.

Images