JP7561382B2 - Colonoscopic observation support device, operation method, and program - Google Patents

Description

The present invention relates to a colonoscopic observation support device, a colonoscopic observation support method, and a program.

Conventionally, image processing devices have been proposed that perform image recognition processing on images captured by an endoscope of hollow organs such as the digestive system, and automatically detect tumors, polyps, and other lesions using learning functions such as artificial intelligence (AI) (see, for example, Patent Document 1).

In addition, a technology has been disclosed that supports pathological diagnosis by alerting a doctor or other operator in real time (e.g., presenting an image of the lesion location, displaying the probability of scattering, or issuing an audio warning) when a lesion is automatically detected based on captured images in order to prevent doctors from overlooking lesions (see, for example, Patent Document 2).

Advances in artificial intelligence (AI) have improved the accuracy and speed of automatic lesion detection by computers, contributing greatly to the observation and diagnosis of lesions by doctors. Non-Patent Document 1 describes how more accurate image analysis is possible by analyzing multiple frames with AI. There is a strong need for technology to be developed that can assist doctors in more accurately observing and diagnosing lesions by referring to the detection results of artificial intelligence, without being unnecessarily influenced by those detection results.

JP 2006-153742 A JP 2011-104016 A

Tran, Du, et al. "Learning spatiotemporal features with 3d convolutional networks." Computer Vision (ICCV), 2015 IEEE International Conference on. IEEE, 2015.

The present invention was made in consideration of the above circumstances, and aims to improve the performance of endoscopic observation support, allowing doctors to more easily and accurately observe lesions, etc.

In order to achieve the above object, the colonoscopic observation support device according to the present application is a colonoscopic observation support device that supports observation of the colon using a colonoscope, and is characterized by comprising: a vibrator provided in a gripping section held by an operator of the colonoscope; an image information acquisition section that acquires images of the colon captured in real time by the colonoscope and displays them in real time on a display section; a lesion information acquisition section that detects a predetermined lesion based on the captured images and acquires lesion information related to the lesion; and a notification section that notifies the user of the detected lesion by vibrating the vibrator with a wavelength and/or intensity changed according to the lesion information.

The present invention can improve the performance of endoscopic observation support, allowing doctors to observe lesions more easily and accurately.

1 is a schematic diagram showing a hardware configuration of an endoscopic observation support device according to an embodiment of the present application. 2 is a functional block diagram of the endoscopic observation support device of FIG. 1 . 4 is a flowchart for explaining an example of an operation of an endoscopic observation support process executed by the endoscopic observation support device of FIG. 1 . FIG. 11 shows an example of an endoscopic image displayed on the display unit and the state of the sound output unit and vibration unit at that time, where (a) shows the state when a lesion appears on the endoscopic image, and (b) shows the state when the lesion has disappeared from the endoscopic image. 13 shows examples of the display of the display unit when an alarm is displayed, where (a) shows an example of the display before a lesion is detected (no lesion), (b) shows an example of the display when the lesion has been detected and disappeared from the display unit, (c) shows a modified example of the display when the lesion has been detected and disappeared from the display unit, (d) shows another example of the display before a lesion is detected (no lesion), (e) shows another example of the display when the lesion has been detected and disappeared from the display unit, and (f) shows a modified example of the display when the lesion has been detected and disappeared from the display unit.

First Embodiment
Hereinafter, an endoscopic observation support device according to a first embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic diagram showing a hardware configuration of an endoscopic observation support device 100 according to this embodiment. As shown in Fig. 1, the endoscopic observation support device 100 mainly includes a support device main body 10 and an endoscope device 40.

As shown in FIG. 1, the support device main body 10 is configured with an information processing section 20 having a CPU 11, a ROM 12, a RAM 13, an input I/F (interface) 14, an output I/F 15, an external device I/F 16, a memory section 17, etc., an input section 21, a display section 22, a sound output section 23, a vibration section 24, etc. In addition to the above, the support device main body 10 may also be equipped with a printer as an output destination for printing endoscopic images and support results, a communication device for communicating with the outside, etc.

The information processing unit 20 can be configured, for example, from a personal computer (PC) or a microcomputer. The CPU 11 is a central processing unit that controls the operation of the entire endoscopic observation support device 100. The ROM 12 is a read-only memory that stores the control program and the endoscopic observation support program executed by the CPU 11. The RAM 13 is a write/read memory that is used as the work area of the CPU 11. That is, the CPU 11 uses the RAM 13 as a work area to execute the endoscopic observation support program stored in the ROM 12, thereby operating the endoscopic observation support device 100.

The storage unit 17 is composed of a recording medium such as a hard disk or a flash memory, and stores various information required for endoscopic observation support processing, calculation results, etc. The input I/F 14 is an interface for connecting the input unit 21 and the CPU 11. The output I/F 15 is an interface for connecting the display unit 22 and the CPU 11. The external device I/F 16 is an interface for connecting an external device and the CPU 11, and in this embodiment, the endoscope device 40, the sound output unit 23, and the vibration unit 24 are connected to the CPU 11 via the external device I/F 16 corresponding to each of them.

The input unit 21 is a device for inputting letters, numbers, various instructions, etc., and is operated by a doctor or the like as an operator who operates the endoscope device 40. Examples of the input unit 21 include a keyboard, mouse, numeric keypad, and touch panel. In addition, the input unit 21 is used to input instructions for the doctor to take and record still images of the lesion, and instructions to confirm an automatically detected lesion.

The display unit 22 is a device that displays a menu screen, an operation screen, and endoscopic images (captured images) such as moving images and still images captured by the endoscope device 40. Examples of the display unit 22 include a liquid crystal display (monitor).

The sound output unit 23 and the vibration unit 24 are used to notify the doctor (alarm notification). The sound output unit 23 is a device that outputs a notification sound (alarm sound) or a notification message (alarm message) to attract the doctor's attention, and examples of such devices include a speaker or buzzer that outputs sound or voice. The vibration unit 24 is a vibrating body that generates vibrations to be applied to the doctor's body, and examples of such devices include a vibration device that generates vibrations using a motor or the like. It is desirable for the vibration unit 24 to be configured to be worn on the doctor's body so that vibrations can be directly applied to the doctor's body. Alternatively, if there is no interference with the operation of the endoscope device 40, the vibration unit 24 may be arranged on the endoscope device 40 and applied to the hand of the doctor holding the endoscope device 40.

In this embodiment, both the sound output unit 23 and the vibration unit 24 are provided, but only one of them may be provided as long as the alarm notification can attract the doctor's attention. For example, if the alarm notification is provided only by vibration, it is possible to prevent the patient from realizing that a lesion has been detected. In addition, the display unit 22 may display an alarm, and the doctor's attention can be attracted visually as well as by sound and vibration.

The endoscope device 40 is an instrument used for observing and treating tubular organs, and is composed of an electronic scope, a fiberscope, etc. The endoscope device 40 of this embodiment is composed of an imaging section 41, an insertion section 42, an operation section 43, etc. In addition to these, the endoscope device 40 is also equipped with components that may be included in known endoscope devices, such as water and air supply means, suction means, and treatment tools such as forceps for tissue sampling and other treatments.

The imaging unit 41 is a device that captures endoscopic images such as moving and still images inside a living body, and is configured with a CCD camera, a condenser lens, an eyepiece, a light source, a light-guiding optical system, an imaging element, etc. The insertion section 42 is configured from a flexible tubular member that can be bent freely, and is provided with the CCD camera and condenser lens of the imaging section 41 at its tip, as well as a nozzle for releasing air and water and a hole through which a treatment tool enters and exits. The operation section 43 is equipped with various operation buttons and switches for bending the insertion section 42, supplying air, supplying water, suction, etc.

The endoscope device 40 is electrically connected to the support device main body 10 via a connection cable or the like. The image signal of the endoscopic image captured by the imaging unit 41 is output to the support device main body 10 in real time.

The endoscopic device 40 is not particularly limited, and any known device can be used. For example, major examples include laryngeal endoscopes for observing the nasal cavity, pharynx, larynx, and esophagus, bronchoscopes for observing the trachea and bronchi, upper gastrointestinal endoscopes for observing the esophagus, stomach, and duodenum, small intestinal endoscopes for observing the small intestine, large intestinal endoscopes for observing the large intestine, cholangioendoscopes for observing the bile duct, pancreatic endoscopes for observing the pancreatic duct, thoracoscopes for observing the thoracic cavity, laparoscopes for observing the abdominal cavity, cystoscopes for observing the urethra and bladder, arthroscopes for observing joints, and vascular endoscopes for observing coronary arteries, but are not limited to these.

In addition, the endoscope device 40 is preferably, for example, a high-definition endoscope, but is not limited to this.

Next, the functions of the information processing unit 20 that controls the operation of the support device main body 10 will be described using the functional block diagram of FIG. 2. As shown in FIG. 2, the information processing unit 20 functions as an image information acquisition unit 31, a lesion information acquisition unit 32, a confirmation information acquisition unit 33, a lesion disappearance determination unit 34, a notification unit 35, and a lesion information output unit 36.

The image information acquisition unit 31 converts the image signal input from the imaging unit 41 of the endoscope device 40 into a digital signal, performs appropriate image processing, and displays it in real time on the display unit 22 (see FIG. 4(a)). In this embodiment, moving images and still images captured by the imaging unit 41 are displayed on the display unit 22, but this is not limited to this, and may be 3D images generated based on moving images, etc. The acquired endoscopic image is passed to the lesion information acquisition unit 32.

The lesion information acquisition unit 32 performs image recognition processing such as feature extraction and pattern matching based on the endoscopic image acquired by the image information acquisition unit 31 to automatically detect lesions and acquire various information related to the lesion (lesion information).

More specifically, the lesion information acquisition unit 32 automatically detects tumors, polyps, bleeding, diverticula, stool, vascular dysplasia, peritoneal seeding, pleural seeding, and other lesions by image recognition processing based on, for example, laryngeal endoscopy video images of the nasal cavity, pharynx, larynx, and esophagus, bronchoscope video images of the trachea and bronchi, upper gastrointestinal endoscopy video images of the esophagus, stomach, and duodenum, small intestine endoscopy video images of the small intestine, large intestine endoscopy video images of the large intestine, biliary duct endoscopy video images of the bile duct, pancreatic duct endoscopy video images of the pancreatic duct, thoracoscope video images of the small thoracic cavity, laparoscope video images of the abdominal cavity, cystoscope video images of the urethra and bladder, arthroscope video images of the inside of a joint, and vascular endoscopy images of blood vessels such as coronary arteries.

The lesion detection using the image recognition processing performed by the lesion information acquisition unit 32 may be performed using any known technology. For example, a method of analyzing multiple frames using artificial intelligence, such as that described in the paper "Learning Spatiotemporal Features with 3D Convolutional Networks" cited as Non-Patent Document 1, can be suitably used for lesion detection. By using this method, lesion detection can be performed with higher accuracy. Here, the detection of a lesion performed by the lesion information acquisition unit 32 is the result of an artificial intelligence (computer) mechanically estimating that a lesion is a lesion using image recognition technology, learning functions, etc. With the development of artificial intelligence, it has become possible to automatically detect lesions at higher speeds and with higher accuracy, and to obtain more detailed information such as the type and condition of the lesion. In the endoscopic observation support device 100 of this embodiment, the detection results by this artificial intelligence are effectively used to provide support that can contribute to pathological diagnosis by doctors.

The lesion information acquired by the lesion information acquisition unit 32 is associated with the endoscopic image and output to the lesion disappearance determination unit 34. The lesion information includes, for example, flag information indicating that a lesion has been detected, and the position, size, and condition of the detected lesion on the endoscopic image. Furthermore, it is more desirable to include the classification results of the lesion by image recognition, more specifically, the classification results of the lesion such as the aforementioned tumor, polyp, bleeding, diverticulum, stool, vascular dysplasia, peritoneal dissemination, and pleural dissemination.

The confirmation information acquisition unit 33 acquires the confirmation information input from the input unit 21. More specifically, for example, a capture button for capturing still images and a lesion confirmation button are displayed on the display screen of the display unit 22. Alternatively, these buttons may be set as keyboard keys. Then, the doctor performs observations inside the living body while visually checking the endoscopic image displayed on the display unit 22 or directly viewing the inside of the body with a microscope, and when the doctor confirms the presence of a lesion, he operates the capture button and the lesion confirmation button of the input unit 21. This operation signal is input to the confirmation information acquisition unit 33 as confirmation information. In this embodiment, the confirmation information acquisition unit 33 is provided to provide support more efficiently, but it is not necessarily required to be provided.

The lesion disappearance determination unit 34 determines whether or not the lesion has disappeared from the endoscopic image based on the endoscopic image acquired by the image information acquisition unit 31 and the lesion information acquired by the lesion information acquisition unit 32. Furthermore, in this embodiment, the lesion disappearance determination unit 34 determines whether or not the lesion has disappeared from the endoscopic image based on the confirmation information acquired by the confirmation information acquisition unit 33, without the lesion being confirmed by a doctor. The result of this determination is output to the notification unit 35.

The notification unit 35 issues an alarm notification based on the result of the determination by the lesion disappearance determination unit 34. When the lesion disappearance determination unit 34 determines that the lesion has disappeared from the endoscopic image without the doctor confirming the lesion, the notification unit 35 controls the sound output unit 23 and the vibration unit 24 to generate an alarm sound from the sound output unit 23 and vibrate the vibration unit 24. This alarm sound and vibration notify the doctor that the automatically detected lesion has disappeared from the endoscopic image without the doctor having to confirm the detection of the lesion, and can attract the doctor's attention.

An alarm sound may be output with different tone, message content, volume, and pitch depending on the lesion classification result, and vibrations may be generated with different wavelengths and strength depending on the lesion classification result. Weighting may also be applied depending on the lesion classification result and the degree of condition, and the alarm sound or vibration may be changed according to the weighting. This can attract the doctor's attention more strongly and encourage more careful observation.

The lesion information output unit 36 displays the endoscopic image corresponding to the lesion detected by the lesion information acquisition unit 32 on the display unit 22 as the output destination and stores it in the memory unit 17. This allows the doctor to check the endoscopic image and lesion information at any time after the examination is completed and use it as a reference for pathological diagnosis, improving the support performance of endoscopic observation and reducing memory capacity.

It is also desirable to display the lesion information on the display unit 22 in association with the endoscopic image, or to store it in the storage unit 17. For example, if the lesion site on the endoscopic image is displayed on the display unit 22 surrounded by a line, it becomes easier for the doctor to confirm the position of the automatically detected lesion, and the doctor's attention can be attracted.

Also, an alarm may be displayed on the image displayed on the display unit 22. In this case, it is preferable to avoid displaying lesion information within the endoscopic image in order to suppress any effect on the visibility of the endoscopic image. For example, the color of the area other than the endoscopic image is changed to attract the doctor's attention. Specifically, as shown in FIG. 5(a) and the like, the area (four corners) outside the display area 22a of the endoscopic image on the display screen of the display unit 22 is used as an alarm display area 22b. In FIG. 5(a), before a lesion such as a polyp is detected, the color of the alarm display area 22b is not changed as shown in FIG. 5(a) and is displayed in a normal color (e.g., black). Also, it remains black immediately after a lesion is detected.

When the lesion has been detected and disappeared from the endoscopic image, the alarm display area 22b is displayed in a conspicuous color (e.g., red) as shown in FIG. 5(b). It may also be made to blink. FIG. 5(c) is a modified example of the display example when the lesion has disappeared, in which the outer periphery of the display screen is used as the alarm display area 22b, and when the lesion has disappeared, it is displayed in red. In this way, by changing the color of the alarm display area 22b, which is provided on the outside separately from the display area 22a of the endoscopic image, the doctor or the like can clearly view the endoscopic image and recognize that the lesion has disappeared not only by sound and vibration but also by sight.

Figures 5(d), (e), and (f) are other display examples of the display unit 22 when an alarm is notified by color change. These are displayed by the same processing as in Figures 5(a), (b), and (c), and provide the same effect, except that the display area 22a of the endoscopic image is approximately elliptical and the inner periphery of the alarm display area 22b is also arc-shaped.

In addition, by storing endoscopic images and lesion information in the memory unit 17, more detailed information about the lesion can be provided when a doctor makes a pathological diagnosis after the examination.

An example of the endoscopic observation support process executed by the endoscopic observation support device 100 configured as described above will be described with reference to the flowchart in FIG. 3. First, the doctor inserts the insertion portion 42 into the body of the patient to be observed while operating the operation portion 43 of the endoscopic device 40. Following the insertion of the insertion portion 42, the imaging portion 41 captures endoscopic images of the inside of the body in real time.

In step S1, the image information acquisition unit 31 acquires an endoscopic image (frame) captured by the imaging unit 41, processes the image, and displays it on the display unit 22. Then, based on the acquired endoscopic image, the processes from step S2 onwards are performed, and these processes may be performed for each frame or every few frames.

In step S2, based on the endoscopic image acquired by the image information acquisition unit 31, the lesion information acquisition unit 32 performs automatic detection of a lesion by the image recognition processing and learning function described above. FIG. 4(a) shows an example of an endoscopic image in which a lesion appears. At this time, even if a lesion is automatically detected, no notification is given from the sound output unit 23 or the vibration unit 24. When a lesion is detected, the lesion information acquisition unit 32 acquires the position, size, condition, classification, etc. of the lesion on the endoscopic image and generates lesion information. Note that when multiple lesions are detected in one endoscopic image, lesion information is generated for each of them.

In the next step S3, the lesion information acquisition unit 32 determines whether or not a lesion has been detected, and if no lesion has been detected (NO), the process skips steps S4 to S9 and proceeds to step S10.

On the other hand, if a lesion is detected in step S3 (YES), the process proceeds to step S4, where the confirmation information acquisition unit 33 acquires confirmation information to detect whether or not the doctor has performed a lesion confirmation operation. Next, the process proceeds to step S5, where the image information acquisition unit 31 acquires the next frame of the endoscopic image and displays it on the display unit 22, and the lesion information acquisition unit 32 detects the lesion, in order to track the lesion on the confirmation information endoscopic image. Next, in step S6, it is determined whether the lesion being tracked has disappeared from the endoscopic image.

If it is determined in step S6 that the lesion has not disappeared (NO), the process returns to step S4, confirmation information is acquired, and the process proceeds to step S5 to continue tracking the lesion. In this way, while the lesion is present on the endoscopic image, no notification is given that the lesion has been automatically detected, and the doctor can perform pathological diagnosis, etc. while observing the endoscopic image displayed on the display unit 22, without being unnecessarily influenced by the automatic detection results.

On the other hand, if it is determined in step S6 that the lesion has disappeared (YES), the process proceeds to step S7, where it is determined whether confirmation information has been acquired for the endoscopic image before the lesion disappeared. If the confirmation information has been acquired (YES), the process proceeds to step S9. In step S9, the lesion information output unit 36 displays the endoscopic image before the lesion disappeared and the lesion information on the display unit 22, and stores the endoscopic image and the lesion information in the memory unit 17.

In this way, when confirmation information is acquired, it can be determined that the lesion automatically detected by the lesion information acquisition unit 32 has also been confirmed by the doctor. In other words, when a doctor who observes the endoscopic image or microscope image on the display unit 22 detects a lesion, the doctor issues an instruction to capture a still image including the lesion from the input unit 21 or issues an instruction to confirm, and a confirmation information signal is input to the support device main body 10. Note that if an instruction to capture a still image has been issued, this still image is displayed and stored, and if only a confirmation instruction has been issued, an endoscopic image (frame) before the lesion disappeared is displayed and stored.

In contrast, if it is determined in step S7 that the lesion has disappeared but confirmation information has not been obtained from the doctor immediately before the disappearance (NO), the process proceeds to step S8 and an alarm is issued. This means that the automatic lesion detection result by the lesion information acquisition unit 32 and the doctor's lesion judgment result do not match, or the judgment matches and the lesion is confirmed but no confirmation operation is performed, and the process proceeds to imaging another area.

In the alarm notification process of step S8, the notification unit 35 controls the sound output unit 23 to generate an alarm sound, and controls the vibration unit 24 to vibrate. FIG. 4(b) shows an example of an endoscopic image in which a lesion has disappeared, and also shows a schematic diagram of a state in which an alarm sound is output from the sound output unit 23 and the vibration unit 24 is vibrating. In addition, as shown in FIGS. 5(b), (c), (e), and (f), an alarm may be notified by displaying or blinking a conspicuous color such as red in the alarm display area 22b on the display screen of the display unit 22.

The alarm sound, vibration, and alarm display on the display unit 22 allow the doctor to realize that he has not yet confirmed the lesion that was automatically detected by the computer. In other words, it is possible that the doctor did not determine that there was a lesion, or that he overlooked the lesion. It is desirable to notify the alarm within one second after the lesion disappears. This makes it possible to prevent the doctor from continuing the endoscopic examination without confirming the lesion. It is also possible to operate the operation unit 43 in the reverse direction to return the imaging area of the imaging unit 41 to the area before the lesion disappeared, allowing the doctor to reconfirm the presence or absence of a lesion and its condition, thereby improving the accuracy of the doctor's pathological diagnosis.

Then, proceed to step S9, where the lesion information output unit 36 displays the endoscopic image and lesion information of the automatically detected lesion before it disappeared on the display unit 22 (or another display unit), and stores this endoscopic image and lesion information in the memory unit 17. The doctor can also observe areas suspected of being a lesion using the endoscopic image displayed on the display unit 22, etc. In addition, the doctor can reconfirm the endoscopic image of the lesion and the lesion information stored in the memory unit 17 after the examination. Then, proceed to step S10.

In step S10, it is determined whether or not endoscopic image capture by the endoscope device 40 has been completed. If it is determined that capture has been completed (YES), the examination has ended and the endoscopic observation support process is terminated. On the other hand, if it is determined that capture is continuing (NO), the process returns to step S1 and the endoscopic observation support process is continued for the next endoscopic image.

As described above, the endoscopic observation support device 100 of this embodiment is configured to include an image information acquisition unit 31 that acquires an image of a tubular organ captured by the endoscopic device 40 and displays it on the display unit 22, a lesion information acquisition unit 32 that detects a predetermined lesion based on the captured image and acquires lesion information related to the lesion, a lesion disappearance determination unit 34 that tracks the lesion based on the captured image and the lesion information and determines whether the lesion has disappeared from the captured image, and a notification unit 35 that notifies the determination result when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the captured image.

The endoscopic observation support method of this embodiment also includes a step of acquiring an image of a tubular organ captured by the endoscopic device 40, a step of detecting a predetermined lesion based on the captured image and acquiring lesion information related to the lesion, a step of tracking the lesion based on the captured image and the lesion information and judging whether the lesion has disappeared from the captured image, and a step of notifying the judgment result when it is judged that the lesion has disappeared from the captured image.

The program for executing the endoscopic observation support method of this embodiment is a program for causing a computer to function as a means for acquiring an image of a tubular organ captured by the endoscopic device 40, a means for detecting a predetermined lesion based on the captured image and acquiring lesion information related to the lesion, a means for tracking the lesion based on the captured image and the lesion information and determining whether the lesion has disappeared from the captured image, and a means for notifying the determination result when it is determined that the lesion has disappeared from the captured image.

Therefore, in this embodiment, an operator such as a doctor can observe the tubular organ and perform pathological diagnosis and treatment by visually checking the captured image (endoscopic image) displayed on the display unit 22. At the same time, lesions can be automatically detected with high accuracy based on the captured image. And while a lesion appears on the captured image, the doctor is not notified of the automatic detection result prior to his or her judgment, so the doctor can perform a pathological diagnosis without being unnecessarily influenced by the automatic detection result.

In contrast, when a lesion disappears from the captured image, a notification is given, so the doctor can know that the lesion has been automatically detected and that the detected lesion has disappeared from the captured image. At this time, if the doctor himself has also confirmed the lesion in the captured image, he or she will know that his or her own diagnosis result matches the automatic detection result, which can be used as a valuable piece of evidence to support the reliability of the doctor's diagnosis. Furthermore, this notification can encourage the doctor to re-observe the area near the lesion or to observe more carefully. Therefore, it is possible to provide an endoscopic observation support device 100, endoscopic observation support processing, and program that can improve the performance of endoscopic observation support and enable doctors to observe lesions more easily and accurately.

In addition, in this embodiment, as described above, lesions can be automatically detected with high accuracy, and a notification is given when an automatically detected lesion disappears from the captured image, so that the system can be suitably used for training in pathological diagnosis using an endoscope, for example. In other words, when performing a pathological diagnosis based on an image, even if an inexperienced doctor continues the examination while overlooking a lesion, the doctor is promptly notified that the lesion has disappeared from the captured image, so that the doctor can notice the oversight and easily recheck, etc. Furthermore, since no notification is given while the lesion is still present in the captured image, the doctor can perform a pathological diagnosis without being unnecessarily influenced by the automatic detection result. As a result, the doctor's pathological diagnosis ability can be improved.

In addition, in this embodiment, the notification unit 35 is configured to notify the determination result by outputting a predetermined notification sound from the sound output unit 23. The notification unit 35 is also configured to notify the determination result by vibrating the vibrator (vibration unit 24). The notification sound or vibration can more strongly attract the doctor's attention and more clearly notify the doctor that the lesion has disappeared from the captured image.

In addition, in this embodiment, a lesion information output unit 36 is provided that outputs an image of the lesion before it disappears to a specified output destination (e.g., the display unit 22, the memory unit 17) when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the captured image. This allows the doctor to check the endoscopic image and lesion information at any time after the examination is completed and use them as a reference for pathological diagnosis, thereby improving the support performance of endoscopic observation.

In addition, in this embodiment, an input unit 21 is provided that accepts input of a lesion confirmation instruction from the operator, and the notification unit 35 is configured to notify the determination result when the lesion disappearance determination unit 34 determines that the lesion has disappeared from the captured image and a confirmation instruction has not been input from the input unit 21. As a result, even if the lesion has disappeared, no notification is made as long as the doctor who is the operator has confirmed the lesion, so notifications can be kept to a minimum and endoscopic observation can be supported more efficiently.

Second Embodiment
Next, an endoscopic observation support device according to a second embodiment will be described. In the above-described first embodiment, when it is determined that a lesion has disappeared from the endoscopic image, an alarm is issued to inform the user that the lesion has disappeared. In contrast, in the second embodiment, an alarm is issued to inform the user that a lesion has been detected a short time after the lesion has been detected on the endoscopic image. In other words, an alarm is issued with a delay from the time of detection, regardless of whether the lesion has disappeared from the endoscopic image.

The endoscopic observation support device of the second embodiment has the same basic configuration as the first embodiment, except that the lesion disappearance determination unit 34 is not provided, so detailed description will be omitted. The confirmation information acquisition unit 33 may or may not be provided. The endoscopic observation support method of the second embodiment includes a step of acquiring an image of a tubular organ photographed by the endoscopic device 40, a step of detecting a predetermined lesion based on the image and acquiring lesion information related to the lesion, and a step of notifying the detection of the lesion a short time after the lesion is detected. The program for executing the endoscopic observation support method of the second embodiment is a program for causing a computer to function as a means for acquiring an image of a tubular organ photographed by the endoscopic device 40, a means for detecting a predetermined lesion based on the image and acquiring lesion information related to the lesion, and a means for notifying the detection of the lesion a short time after the lesion is detected.

In the second embodiment of the endoscopic observation support device, when the lesion information acquisition unit 32 detects a lesion such as a polyp, it notifies the notification unit 35. The notification unit 35 does not notify in real time when the lesion is detected, but notifies that the lesion has been automatically detected a short time after the lesion has been detected, that is, after a predetermined time has passed. As in the first embodiment, this notification can be made by the generation of an alarm sound from the sound output unit 23, the vibration of the vibration unit 24, or even a change in color of the alarm display area 22b as shown in FIG. 5.

Here, "a short period of time" refers to the time required for a doctor or the like who visually recognizes a lesion in an endoscopic image to determine that it is a lesion. More specifically, a "short period of time" is preferably 30 ms to 2,000 ms. It is also more preferable to be able to arbitrarily set or change the "short period of time" within this range. This allows a suitable period of time to be set according to the doctor or the like, even if there is variation in the time required to make a determination depending on the doctor's level of proficiency in endoscopic examinations, age, etc. This allows for more efficient examinations and improves the usability of the endoscope device 40.

In this way, by notifying the detection of a lesion after a short delay, it is possible to secure time for the doctor or other medical professional to determine whether or not there is a lesion based on the endoscopic image. Therefore, the doctor or other medical professional can determine for himself or herself whether or not there is a lesion, without being influenced by the results of the automatic lesion detection by the lesion information acquisition unit 32.

After a short period of time has passed, the lesion information acquisition unit 32 notifies the doctor that a lesion has been detected, allowing the doctor or other medical professional to check whether his or her own judgment is the same as the result of automatic detection by artificial intelligence or the like, thereby further improving the accuracy of the doctor or other medical professional's judgment.

The second embodiment can also be suitably used when training in pathological diagnosis. For example, when an inexperienced doctor is performing endoscopic observation in the early stages of training, the "short period of time" can be set long, allowing the doctor more time to make a judgment. By gradually shortening the "short period of time" each time the training is repeated, the doctor will be able to make judgments gradually faster.

As described above, by analyzing multiple image frames of a captured image using artificial intelligence, it is possible to diagnose a lesion with higher accuracy than by analyzing a single frame. In conventional methods of notifying the detection of a lesion in real time, there is a limit to the number of image frames that can be analyzed because speed is prioritized. In contrast, when notifying a lesion a predetermined time after detection as in this embodiment, or when notifying after the lesion has disappeared as in the first embodiment, a predetermined analysis time can be secured, and more frames can be analyzed. Therefore, more detailed lesion information, such as the type and degree of the lesion, can be obtained, and the accuracy of lesion detection and lesion information by artificial intelligence can be improved.

The above describes the embodiment of the present invention in detail with reference to the drawings, but the above embodiment is merely an example of the present invention, and the present invention is not limited to the configuration of the above embodiment. Of course, even if there are design changes within the scope of the present invention, they are included in the present invention.

For example, in each of the above embodiments, an endoscope device 40 having an imaging section 41, an insertion section 42, and an operation section 43 is connected to the support device main body 10, and the support device main body 10 processes the image captured by the imaging section 41 and displays it on the display section 22. Alternatively, an endoscope unit having a display function for captured images, such as a video tuner or a display section, in addition to the imaging section 41, the insertion section 42, and the operation section 43 may be used. Furthermore, the display section of such an endoscope unit may also be used as the display section 22 of the endoscopic observation support device 100, or a display section 22 may be provided separately from this.

Furthermore, the endoscope unit may be configured to include an information processing device having an image processing unit that automatically detects lesions based on captured images. In this case, if the results of automatic lesion detection by the image processing unit are acquired by the lesion information acquisition unit 32 of the support device main body 10 and used for endoscopic observation support processing, the automatic detection results of the endoscope unit can be effectively used, reducing the processing load on the support device main body 10 and increasing the calculation speed.

The endoscopic observation support device 100 of each embodiment may also be built into an endoscope unit equipped with an information processing device or the like. That is, the endoscope unit is provided with a sound output unit 23 and a vibration unit 24, and the information processing device of the endoscope unit is also used as the information processing unit 20 to install the endoscopic observation support program of each embodiment. This makes it possible to provide an endoscope unit equipped with an endoscopic observation support device. Therefore, it is possible to sell the endoscope unit equipped with the endoscopic observation support device, or to sell the endoscopic observation support program separately and install it in an existing endoscope unit for use. It is also possible to make each device more compact, simple, and easier to handle.

In addition, in each of the above embodiments, the doctor is notified that the lesion has disappeared from the captured image by a notification sound or vibration, but in addition to this, for example, a light emitting unit such as an LED may be provided and the error may be communicated by blinking light, etc. In this case, too, the light color and blinking time may be appropriately adjusted according to the weighting of the lesion and the type of lesion, making it possible to provide a more understandable notification.

A time measuring unit may be provided that measures the time taken to insert the insertion unit 42 into the body, and the insertion speed of the insertion unit 42 (which may be measured with a speedometer, or an average insertion speed may be used), the insertion time, and the captured image may be used to calculate the distance from the anal edge, for example, in the case of a colonoscope, and displayed on the display unit 22. If the distance from the anal edge of the lesion is displayed when the lesion disappears from the captured image and an image of the lesion is displayed on the display unit 22, etc., the doctor can more easily and accurately determine how many centimeters the lesion needs to be moved back to capture an image, and the performance of lesion observation support can be further improved.

In addition, in the first embodiment, an alarm is issued when the lesion disappears, and in the second embodiment, an alarm is issued a short time after the lesion is detected, but this is not limited to this. Depending on the purpose of use, etc., an alarm may be issued both when the lesion disappears and when a short time has passed since the lesion was detected, which can more strongly attract the doctor's attention.

17 Memory unit (output destination) 20 Information processing unit 21 Input unit 22 Display unit (output destination) 23 Sound output unit 24 Vibration unit (vibrator)
31 Image information acquisition unit 32 Lesion information acquisition unit 34 Lesion disappearance determination unit (determination unit)
35 Notification unit 36 Lesion information output unit 40 Endoscope device (endoscope)
100 Endoscopic observation support device

Claims (4)

A colonoscopic observation support device for supporting observation of the large intestine by a colonoscope, comprising:
a vibrator provided in a gripping portion that is gripped by an operator of the colonoscope;
an input unit that receives an input of a lesion confirmation instruction from the operator;
an image information acquisition unit that acquires images of the large intestine captured in real time by the colonoscope and displays the images on a display unit in real time;
a lesion information acquisition unit that detects a predetermined lesion based on the captured image and acquires lesion information related to the lesion;
a confirmation information acquisition unit that acquires confirmation information input from the input unit;
a determination unit that tracks the lesion based on the captured image and the lesion information and determines whether or not the lesion has disappeared from the captured image without the operator confirming the lesion;
a notification unit that issues an alarm notifying that the lesion has disappeared by vibrating the vibrator when the determination unit determines that the lesion has disappeared from the captured image without the operator confirming the lesion,
The colonoscopic observation support device is characterized in that the notification unit issues the alarm notification by vibrating the vibrator with varying strength depending on a weighting applied depending on the state of the lesion and the classification result of the lesion by image recognition. The colonoscopic observation support device according to claim 1, wherein the notification unit issues an alarm by vibrating the vibrating body to notify that the lesion has been detected after a time period within a range of 30 ms to 2,000 ms from the detection of the lesion and set according to the skill level or age of the operator has elapsed. A method for operating the colonoscopic observation support device according to claim 1 or 2 , comprising the steps of:
a step of the image information acquisition unit acquiring an image of the large intestine captured in real time by the large intestine endoscope and displaying the image on a display unit in real time;
a step of the lesion information acquisition unit detecting a predetermined lesion based on the captured image and acquiring lesion information regarding the lesion;
a step of the confirmation information acquisition unit acquiring confirmation information input from the input unit;
a step of the determination unit tracking the lesion based on the captured image and the lesion information, and determining whether or not the lesion has disappeared from the captured image without the operator confirming the lesion;
a step of, when the determination unit determines that the lesion has disappeared from the captured image without the operator confirming the lesion, the notification unit issues an alarm notification that the lesion has disappeared by vibrating a vibrator provided in a gripping unit that is held by an operator of the colonoscope with a strength changed according to a weighting performed according to the state of the lesion and the classification result of the lesion by image recognition;
23. A method of operating comprising: A program for causing a computer to execute each of the steps according to claim 3 .