JP7737694B2 - Information processing device, information processing method, and computer program - Google Patents

Description

The technology disclosed in this specification relates to an information processing device for detecting pulmonary leakage from lung images.

For example, one of the complications of pneumothorax or lung resection is pulmonary leakage (pulmonary fistula). Pneumonorrhea occurs when the surface of the lung is damaged, causing air to leak from the lung parenchyma into the thoracic cavity. There are various methods for treating pneumonorrhea, including surgery, but the first step in treatment is to identify the site of the lung leakage.

Traditionally, the location of a lung leak has been identified using a leak test. In this test, water (saline) is injected into the thoracic cavity, air is then pumped into the lungs to inflate them, and the location of air bubbles leaking from the lung surface is identified (see, for example, Non-Patent Document 1).

Kazuhiro Ueda, "Effectiveness of pulmonary fistula closure using absorbable mesh and fibrin adhesive in thoracoscopically assisted lung cancer surgery," Yamaguchi Medical Journal, 2010, Vol. 59, No. 4, pp. 161-165

Leak testing to identify the site of a lung leak must be performed under general anesthesia, placing a significant burden on the patient. Furthermore, lung leaks may not be detected depending on how the pressure is applied, leaving room for improvement in the accuracy of lung leak detection. Furthermore, because the lungs must be inflated, it is difficult to maintain a clear view during laparoscopic surgery.

This specification discloses technology that can solve the above-mentioned problems.

The technology disclosed in this specification can be realized, for example, in the following forms:

(1) The information processing device disclosed in this specification is an information processing device for detecting pulmonary leaks from lung images, and includes a target image acquisition unit, a model acquisition unit, and a detection execution unit. The target image acquisition unit acquires a target image that is an image of the lungs. The model acquisition unit acquires a pulmonary leak detection model. The pulmonary leak detection model is a model generated by deep learning using training data that associates multiple lung images with pulmonary leak information that indicates the pulmonary leak site in each of the multiple lung images. The detection execution unit uses the target image and the pulmonary leak detection model to perform pulmonary leak detection that identifies the pulmonary leak site in the target image, and outputs the results of the pulmonary leak detection.

This information processing device uses a target image, which is an image of the lungs, and a lung leak detection model generated by deep learning to perform lung leak detection, identifying the lung leak site in the target image, and output the lung leak detection results. In other words, this information processing device can achieve lung leak detection simply by acquiring a target image, which is an image of the lungs, without performing any other procedures on the lungs. Therefore, this information processing device can detect lung leaks using target images generated and acquired by imaging with a thoracoscope under local anesthesia, without requiring general anesthesia as in conventional lung leak detection using leak tests, thereby reducing the burden on the patient. Furthermore, this information processing device does not require the application of pressure to the lungs as in conventional lung leak detection using leak tests, thereby preventing missed lung leak detections due to the way pressure is applied and improving lung leak detection accuracy. Furthermore, this information processing device does not require the lungs to be inflated as in conventional lung leak detection using leak tests, and can detect lung leaks using target images generated and acquired by imaging the lungs in a collapsed state, thereby preventing the field of view from becoming narrower during endoscopic surgery.

Machine learning is sometimes used to perform diagnosis in pathological screening using X-ray images and in detecting lesions in endoscopic camera images of the digestive tract. However, such machine learning-based diagnosis automates diagnosis that was previously performed by experts visually inspecting images. Meanwhile, as noted above, lung leak detection has consistently been performed through leak testing, and the technical concept of detecting lung leaks from lung images is unknown. Through extensive research, the inventors of the present application focused on the technical concept of detecting lung leaks from lung images and further developed specific means for detecting lung leaks from lung images, thereby completing the present invention. Thus, the present invention utilizes the technical concept of detecting lung leaks from lung images, which was not anticipated in prior art and could not have been easily conceived from prior art.

(2) The above information processing device may further include a training data acquisition unit that acquires the training data, and the model acquisition unit may acquire the pulmonary leak detection model by creating the pulmonary leak detection model through deep learning using the training data. This information processing device can acquire a pulmonary leak detection model without using another device, and can use the model to perform pulmonary leak detection from a target image.

(3) In the above-described information processing device, the pulmonary leakage information may include information indicating the degree of pulmonary leakage, and the pulmonary leakage detection performed by the detection execution unit may include processing to identify the degree of pulmonary leakage at each pulmonary leakage site in the target image. According to this information processing device, the degree of pulmonary leakage at the pulmonary leakage site can be determined simply by acquiring the target image, and more useful information for treating pulmonary leakage can be obtained.

(4) In the above-described information processing device, the target image acquisition unit may be configured to acquire, in real time, a moving image of the lungs captured by an imaging device as the target image. This information processing device can perform pulmonary leak detection in real time while capturing moving images of the patient's lungs, thereby enabling efficient diagnosis and treatment of pulmonary leaks.

The technology disclosed in this specification can be realized in various forms, such as an information processing device, an information processing method, a computer program that realizes those methods, or a non-transitory recording medium on which that computer program is recorded.

FIG. 1 is an explanatory diagram showing a schematic configuration of an information processing apparatus 100 according to an embodiment of the present invention. 1 is a flowchart showing a pulmonary leak detection model acquisition process according to an embodiment of the present invention; FIG. 1 is an explanatory diagram illustrating an example of training data TD. An explanatory diagram conceptually illustrating an example of a lung leak detection model MO. 1 is a flowchart showing a pulmonary leakage detection process according to an embodiment of the present invention;

A. Embodiments:
A-1. Configuration of information processing device 100:
1 is an explanatory diagram showing a schematic configuration of an information processing device 100 according to this embodiment. The information processing device 100 is a computer (PC, server, etc.) for detecting pulmonary leakage from lung images. In this specification, detecting pulmonary leakage from lung images means identifying the location of pulmonary leakage in the lungs represented by the images. In this embodiment, detecting pulmonary leakage from lung images also includes identifying the extent of pulmonary leakage.

The information processing device 100 includes a control unit 110, a memory unit 120, a display unit 130, an operation input unit 140, and an interface unit 150. These units are connected to each other via a bus 190 so that they can communicate with each other.

The display unit 130 of the information processing device 100 is configured, for example, by an LCD display, and displays various images and information. The operation input unit 140 is configured, for example, by a keyboard, mouse, buttons, microphone, etc., and accepts operations and instructions from the administrator. The display unit 130 may also function as the operation input unit 140 by being equipped with a touch panel. The interface unit 150 is configured, for example, by a LAN interface or USB interface, and communicates with other devices via wired or wireless connections.

The storage unit 120 of the information processing device 100 is composed of, for example, ROM, RAM, a hard disk drive (HDD), etc., and stores various programs and data, and is used as a work area when executing various programs, and as a temporary storage area for data. For example, the storage unit 120 stores a pulmonary leak detection program CP, which is a computer program for executing the pulmonary leak detection model acquisition process and pulmonary leak detection process described below. The pulmonary leak detection program CP is provided in a state stored on a computer-readable recording medium (not shown), such as a CD-ROM, DVD-ROM, or USB memory, and is stored in the storage unit 120 by installing it on the information processing device 100.

In addition, the storage unit 120 of the information processing device 100 stores training data TD, pulmonary leak detection model MO, target image Io, and pulmonary leak detection result data RD during the pulmonary leak detection model acquisition process and pulmonary leak detection process described below. The contents of this information will be explained in conjunction with the explanation of the pulmonary leak detection model acquisition process and pulmonary leak detection process described below.

The control unit 110 of the information processing device 100 is configured, for example, by a CPU, and controls the operation of the information processing device 100 by executing computer programs read from the storage unit 120. For example, the control unit 110 reads and executes the pulmonary leak detection program CP from the storage unit 120, thereby functioning as a training data acquisition unit 112, a model acquisition unit 114, a target image acquisition unit 115, and a detection execution unit 116 for executing the pulmonary leak detection model acquisition process and pulmonary leak detection process described below. The functions of each of these units will be explained in conjunction with the explanation of the pulmonary leak detection model acquisition process and pulmonary leak detection process described below.

A-2. Lung leak detection model acquisition process:
Next, a pulmonary leak detection model acquisition process executed by information processing device 100 of this embodiment will be described. FIG. 2 is a flowchart showing the pulmonary leak detection model acquisition process of this embodiment. The pulmonary leak detection model acquisition process is a process for acquiring a pulmonary leak detection model MO, which is a learning model used to detect pulmonary leaks from lung images. In this embodiment, information processing device 100 acquires the pulmonary leak detection model MO by creating the pulmonary leak detection model MO through predetermined deep learning. The pulmonary leak detection model acquisition process is started in response to a start command being input by the user operating operation input unit 140 of information processing device 100.

In the pulmonary leak detection model acquisition process, first, the training data acquisition unit 112 (Figure 1) of the information processing device 100 acquires training data TD (S110). The training data TD is acquired via the interface unit 150 and stored in the memory unit 120.

FIG. 3 is a diagram illustrating an example of training data TD. The training data TD is data that associates training image data It(N), which are multiple images of the lungs generated by photographing the lungs, with lung leakage information indicating the location and severity of lung leakage in the lungs represented by each training image data It(N). That is, as shown in FIG. 3, the training data TD includes, for each training image data It(N), information identifying a rectangular region surrounding the lung leakage location (hereinafter referred to as the "lung leakage region Rx") (e.g., information identifying the coordinates of the upper left and lower right points of the lung leakage region Rx) and information identifying one of multiple preset levels of lung leakage severity (e.g., three levels: "heavy," the most severe; "medium," the intermediate level; and "mild," the mildest level). In the example of Figure 3, of the multiple training image data It(N) that make up the training data TD, the first training image data It(1) is an image of lungs with pulmonary leakage, and a pulmonary leakage region Rx is identified on the training image data It(1), and the degree of pulmonary leakage is identified as "severe." Note that for images of lungs without pulmonary leakage, such as the fourth training image data It(4), the pulmonary leakage location and degree of pulmonary leakage are identified as "none" in the pulmonary leakage information.

Of the training data TD, images of lungs with lung leakage can be obtained, for example, by photographing lungs that have been confirmed to have lung leakage by a conventional leak test. Furthermore, lung leakage information (information indicating the location and extent of lung leakage) for images of lungs with lung leakage can be identified based on the results of the leak test. Furthermore, images of lungs without lung leakage can be obtained, for example, by photographing lungs that have been confirmed to have no lung leakage by a conventional leak test. The lung images (training image data It(N)) that make up the training data TD include multiple images generated by photographing the lungs of multiple people. Furthermore, the lung images that make up the training data TD may also include multiple images generated by photographing the lungs of the same person multiple times.

Next, the model acquisition unit 114 (Figure 1) of the information processing device 100 creates a pulmonary leak detection model MO through a predetermined deep learning process using the training data TD (S120). Figure 4 is an explanatory diagram conceptually showing an example of a pulmonary leak detection model MO. As shown in Figure 4, the pulmonary leak detection model MO is a model for detecting pulmonary leaks from a target image Io generated by photographing the lungs. In other words, the pulmonary leak detection model MO is a model for identifying the pulmonary leak region Rx, which is the site of pulmonary leak, from the target image Io, and for identifying the degree of pulmonary leak.

The model acquisition unit 114 uses the training image data It(N) included in the training data TD as input data, the lung leak information (information indicating the lung leak location and degree of lung leak) associated with each training image data It(N) as the objective variable, and performs well-known deep learning using predetermined evaluation indices (e.g., root mean square error (RMSE), mean absolute error (MAE), coefficient of determination ( ^R2 )) to create a lung leak detection model MO. Various well-known deep learning algorithms can be used to create the lung leak detection model MO, but for example, YOLO, a real-time object detection algorithm, can be used. The created lung leak detection model MO is stored in the memory unit 120 of the information processing device 100. The above steps complete the lung leak detection model acquisition process for acquiring the lung leak detection model MO.

A-3. Lung leak detection process:
Next, the pulmonary leak detection process executed by the information processing device 100 of this embodiment will be described. FIG. 5 is a flowchart showing the pulmonary leak detection process in this embodiment. The pulmonary leak detection process is a process for detecting pulmonary leaks from an image of the lungs. More specifically, the pulmonary leak detection process is a process for detecting pulmonary leaks from the target image Io using the target image Io generated by photographing the lungs and the above-mentioned pulmonary leak detection model MO, that is, a process for identifying the pulmonary leak region Rx, which is the pulmonary leak site, and identifying the level of pulmonary leak, and outputting the detection results. The pulmonary leak detection process is started in response to a start command being input by the user operating the operation input unit 140 of the information processing device 100.

In the pulmonary leak detection process, the target image acquisition unit 115 (Figure 1) of the information processing device 100 first acquires a target image Io, which is an image generated by imaging the lungs (S210). The target image Io is, for example, an image generated by imaging using a thoracoscope under local anesthesia, and is acquired via the interface unit 150 and stored in the memory unit 120. The target image Io may be one or more still images, or a moving image composed of multiple frames. The target image acquisition unit 115 may perform predetermined processing (e.g., trimming) on the acquired target image Io so that it is in a format suitable for input to the pulmonary leak detection model MO.

Next, the detection execution unit 116 (FIG. 1) of the information processing device 100 uses the target image Io acquired in S210 and the above-mentioned pulmonary leak detection model MO to perform pulmonary leak detection from the target image Io (S220). As shown in FIG. 4, the detection execution unit 116 inputs the data of the target image Io into the pulmonary leak detection model MO and acquires pulmonary leak information output from the pulmonary leak detection model MO (information identifying the pulmonary leak region Rx, which is the pulmonary leak site, and information identifying the level of pulmonary leak), thereby achieving pulmonary leak detection. For example, in the example shown in FIG. 4, the pulmonary leak region Rx is identified in the target image Io, and the level of pulmonary leak is identified as "medium." The detection execution unit 116 generates pulmonary leak detection result data RD, which is information indicating the pulmonary leak detection result, and stores it in the memory unit 120 of the information processing device 100.

Next, the detection execution unit 116 of the information processing device 100 outputs the pulmonary leak detection result based on the pulmonary leak detection result data RD (S230). For example, the detection execution unit 116 displays the pulmonary leak detection result on the display unit 130. At this time, for example, the target image Io is displayed on the display unit 130, and a rectangular line indicating the pulmonary leak region Rx is displayed on the target image Io. The level of pulmonary leak at each pulmonary leak site is also displayed using text. Note that the method of displaying the level of pulmonary leak is not limited to displaying using text, and other display methods such as displaying using color can also be used. Furthermore, if multiple pulmonary leak regions Rx are detected, the level of pulmonary leak at each pulmonary leak site is displayed. This display allows the user of the information processing device 100 to understand the pulmonary leak detection result for the target image Io.

In the pulmonary leak detection process, the target image acquisition unit 115 may acquire a moving image of the lungs captured by an imaging device as the target image Io in real time. The detection execution unit 116 may also perform pulmonary leak detection in real time using the target image Io and output the pulmonary leak detection result.

A-4. Effects of this embodiment:
As described above, the information processing device 100 of this embodiment includes a target image acquisition unit 115, a model acquisition unit 114, and a detection execution unit 116. The target image acquisition unit 115 acquires a target image Io, which is an image of the lungs. The model acquisition unit 114 acquires a pulmonary leak detection model MO. The pulmonary leak detection model MO is a model generated by deep learning using training data TD in which training image data It(N), which are multiple images of the lungs, are associated with pulmonary leak information indicating the pulmonary leak site in each training image data It(N). The detection execution unit 116 uses the target image Io and the pulmonary leak detection model MO to perform pulmonary leak detection to identify the pulmonary leak site in the target image Io and output the pulmonary leak detection result.

As described above, the information processing device 100 of this embodiment can perform lung leak detection by identifying the lung leak site in the target image Io using the target image Io, which is an image of the lungs, and the lung leak detection model MO generated by deep learning, and output the lung leak detection results. In other words, the information processing device 100 of this embodiment can achieve lung leak detection simply by acquiring the target image Io, which is an image of the lungs, without performing any other procedures on the lungs. Therefore, the information processing device 100 of this embodiment can detect lung leaks using the target image Io generated and acquired by imaging using a thoracoscope under local anesthesia, without requiring general anesthesia as in conventional lung leak detection using leak testing, thereby reducing the burden on the patient. Furthermore, the information processing device 100 of this embodiment does not require the application of pressure to the lungs as in conventional lung leak detection using leak testing, thereby preventing missed lung leak detections due to the way pressure is applied, and improving lung leak detection accuracy. Furthermore, with the information processing device 100 of this embodiment, there is no need to inflate the lungs as with conventional leak testing to detect lung leaks; instead, lung leaks can be detected using target images Io that are generated and acquired by photographing the lungs in a collapsed state, which prevents the field of view from becoming narrower during endoscopic surgery.

The information processing device 100 of this embodiment also includes a training data acquisition unit 112 that acquires training data TD. The model acquisition unit 114 acquires the pulmonary leak detection model MO by creating the pulmonary leak detection model MO through deep learning using the training data TD. Therefore, with the information processing device 100 of this embodiment, it is possible to acquire the pulmonary leak detection model MO without using any other device, and to perform pulmonary leak detection from the target image Io using this model.

Furthermore, in the information processing device 100 of this embodiment, the pulmonary leakage information includes information indicating the degree of pulmonary leakage, and the pulmonary leakage detection performed by the detection execution unit 116 includes processing to identify the degree of pulmonary leakage at each pulmonary leakage site in the target image Io. Therefore, with the information processing device 100 of this embodiment, it is possible to determine the degree of pulmonary leakage at the pulmonary leakage site simply by acquiring the target image Io, and to obtain more useful information for treating pulmonary leakage.

Furthermore, in the information processing device 100 of this embodiment, the target image acquisition unit 115 may be configured to acquire a moving image of the lungs captured by an imaging device as the target image Io in real time. With this configuration, pulmonary leakage detection can be performed in real time while capturing moving images of the patient's lungs, allowing for efficient diagnosis and treatment of pulmonary leakage.

A-5. Example:
An example conducted to verify the accuracy of pulmonary leak detection by the information processing device 100 of the above-described embodiment will be described below. First, 110 images of lungs confirmed to have pulmonary leaks by a conventional leak test were prepared as training data TD, and used as training image data It. Each image was acquired by extracting a 416 x 416 pixel still image from a moving image generated by thoracoscope imaging. Furthermore, for each image containing pulmonary leaks, the coordinates of a rectangular pulmonary leak region Rx indicating the pulmonary leak site were identified based on the results of the leak test.

In addition, in this embodiment, the lung leak detection model MO was created using the above-mentioned training data TD and YOLO, a real-time object detection algorithm.

Next, 17 images of lungs with lung leakage and 10 images of lungs without lung leakage were prepared as test data. Lung leakage detection was performed on each image constituting the test data using the lung leakage detection model MO. As a result, lung leakage sites were detected in the correct position in 15 of the 17 images of lungs with lung leakage. In other words, the detection rate was 88.2% (= 15/17), confirming that there were very few missed detections. Furthermore, lung leakage sites were not detected in any of the 10 images of lungs without lung leakage. In other words, the false detection rate was 0% (= 0/10), confirming that there were very few false detections. Thus, according to this example, it was confirmed that lung leakage detection can be achieved with extremely high accuracy by detecting lung leakage from lung images using the information processing device 100 of this embodiment.

B. Variations:
The technology disclosed in this specification is not limited to the above-described embodiments, and can be modified in various forms without departing from the spirit thereof, for example, the following modifications are also possible.

The configuration of the information processing device 100 in the above embodiment is merely an example and can be modified in various ways. Furthermore, the contents of the pulmonary leak detection model acquisition process and pulmonary leak detection process in the above embodiment are merely an example and can be modified in various ways. For example, in the above embodiment, the information processing device 100 acquires the pulmonary leak detection model MO by creating the pulmonary leak detection model MO, but the information processing device 100 may also acquire a pulmonary leak detection model MO generated by another device. In this case, the information processing device 100 does not need to have a training data acquisition unit 112.

In the above embodiment, the pulmonary leakage information includes information indicating the degree of pulmonary leakage, but the pulmonary leakage information does not necessarily have to include information indicating the degree of pulmonary leakage.

In the above embodiment, some of the configurations implemented by hardware may be replaced with software, and conversely, some of the configurations implemented by software may be replaced with hardware.

100: Information Processing Device 110: Control Unit 112: Training Data Acquisition Unit 114: Model Acquisition Unit 115: Target Image Acquisition Unit 116: Detection Execution Unit 120: Memory Unit 130: Display Unit 140: Operation Input Unit 150: Interface Unit 190: Bus CP: Lung Leak Detection Program Io: Target Image It: Training Image Data MO: Lung Leak Detection Model RD: Lung Leak Detection Result Data Rx: Lung Leak Area TD: Training Data

Claims (6)

An information processing device for detecting pulmonary leakage from an image of the lungs,
a target image acquisition unit that acquires a target image, which is an image of the lungs;
a model acquisition unit that acquires a pulmonary leak detection model, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images;
a detection execution unit that executes pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputs the result of the pulmonary leak detection;
An information processing device comprising: The information processing device according to claim 1, further comprising:
a training data acquisition unit that acquires the training data,
The information processing device wherein the model acquisition unit acquires the pulmonary leak detection model by creating the pulmonary leak detection model through the deep learning using the training data. 3. The information processing device according to claim 1,
the pulmonary leakage information includes information indicating the degree of pulmonary leakage;
An information processing device, wherein the pulmonary leakage detection performed by the detection execution unit includes a process of identifying the degree of pulmonary leakage at each pulmonary leakage site in the target image. 4. The information processing device according to claim 1,
The target image acquisition unit acquires, in real time, a moving image of the lungs captured by an imaging device as the target image. 1. An information processing method for detecting pulmonary leakage from lung images using a computer, comprising :
acquiring, by the computer, a target image , the target image being an image of the lung;
a step of acquiring a pulmonary leak detection model by the computer, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images;
a step in which the computer performs pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputs a result of the pulmonary leak detection;
An information processing method comprising: 1. A computer program for detecting pulmonary leaks from lung images, comprising:
On the computer,
obtaining a target image, the target image being an image of the lungs;
A process for acquiring a pulmonary leak detection model, the pulmonary leak detection model being a model generated by deep learning using training data in which a plurality of lung images are associated with pulmonary leak information indicating a pulmonary leak site in each of the plurality of lung images;
A process of performing pulmonary leak detection to identify a pulmonary leak site in the target image using the target image and the pulmonary leak detection model, and outputting the result of the pulmonary leak detection;
A computer program that executes