JP7542585B2 - Image processing device, endoscope system, and method of operating the image processing device - Google Patents

Description

The present invention relates to an image processing device, an endoscope system, and a method for operating the image processing device, and in particular to technology that can be used to assist doctors in endoscopic examinations.

In the medical field, examinations are performed using endoscopic devices. In recent years, it has become known to support examinations by using image analysis to recognize the location and type of lesions contained in endoscopic images and reporting the recognition results.

In image analysis for recognition, machine learning techniques for images, including deep learning, are widely used.

Patent Document 1 proposes an information processing device that includes an acquisition unit that acquires multiple images of cells captured in chronological order, an assignment unit that assigns evaluation values in chronological order to the multiple acquired images for each of one or more predetermined evaluation items, and an evaluation unit that evaluates the cells based on the change over time in the assigned evaluation values in chronological order. Here, the evaluation unit assigns evaluation values in chronological order to the multiple images of cells captured in chronological order according to a machine learning algorithm, and evaluates the cells being observed based on the change over time in the assigned evaluation values. This enables evaluation that takes into account the chronological evaluation of the cells in a comprehensive manner.

Patent Literature 2 also proposes a processing device that acquires time-series input data, which is a data sequence in a video, supplies multiple input values corresponding to input data at a certain time in the time-series input data to multiple nodes of a trained model (a model constituting a Boltzmann machine) corresponding to the time-series input data, calculates the conditional probability of each input value corresponding to the prediction target time under the conditions in which the input data series occurred based on the input data series prior to the prediction target time in the time-series input data and weight parameters between each of the multiple input values corresponding to the input data in the input data series in the model and each of the multiple nodes, and calculates the conditional probability that the next input data will be a predetermined value under the conditions in which the time-series input data occurred based on the conditional probability of each input value corresponding to the prediction target time.

As an example, this processing device can predict one image data item to be arranged at the next time based on T-1 image data items arranged in a chronological order, and generate a video containing a total of T images.

JP 2018-22216 A JP 2016-71697 A

The information processing device described in Patent Document 1 evaluates various changes during the culture process of a cell (fertilized egg) that is the subject of the imaging process using multiple images captured in chronological order, and the multiple images are captured under the same imaging conditions. This is because if the images are not captured under the same imaging conditions, it is not possible to evaluate changes in the fertilized egg from the multiple images obtained. In other words, the multiple images are not captured sequentially using different observation light.

The processing device described in Patent Document 2 makes it possible to predict image data for the next time using a trained model that inputs time-series input data, and the time-series input data is captured under the same imaging conditions. This is because image data for the next time cannot be predicted from the input time-series input data unless the data is captured under the same imaging conditions. In other words, the time-series input data is not input data that is acquired sequentially using different observation lights.

In addition, the inventions described in Patent Documents 1 and 2 both input multiple images in a time series to predict objects (cells, future videos) that change over time, but do not input multiple images for the purpose of improving the recognition accuracy of a recognizer.

The present invention has been made in consideration of these circumstances, and aims to provide an image processing device, an endoscope system, and an operating method of an image processing device that can improve recognition accuracy based on multiple images and present good observation images and recognition results.

In order to achieve the above object, an image processing device according to one aspect of the present invention includes a recognizer that receives an image set consisting of a plurality of images acquired sequentially using a plurality of different observation lights and outputs a recognition result for the image set, and a display control unit that displays an observation image and the recognition result calculated using a part of the plurality of images or the plurality of images on a display unit.

According to one aspect of the present invention, a set of images acquired sequentially using multiple different observation lights is input, and recognition results for the image set are obtained, thereby improving recognition accuracy compared to when an image is recognized based on a single image acquired using a single observation light. In addition, the recognition results can be appropriately presented by displaying the recognition results on the display unit together with the observation image obtained from the multiple images.

In an image processing device according to another aspect of the present invention, it is preferable that the recognizer has a trained model trained on a set of multiple images for training and correct answer data, and outputs a recognition result based on the trained model each time multiple images for recognition are received.

In an image processing device according to yet another aspect of the present invention, the trained model is preferably configured as a convolutional neural network. Convolutional neural networks excel at recognizing images.

In an image processing device according to yet another aspect of the present invention, it is preferable that the multiple images include a first endoscopic image and a second endoscopic image acquired using an observation light different from that of the first endoscopic image. In an endoscopic examination, multiple images may be acquired using multiple different observation lights, and the present invention can be applied to such an endoscopic examination.

In an image processing device according to yet another aspect of the present invention, it is preferable that the first endoscopic image is a normal light image captured with normal light, and the second endoscopic image is a special light image captured with special light. In general, the normal light image is used as an observation image, and the special light image is used when it is desired to observe a surface structure.

In an image processing device according to yet another aspect of the present invention, the special light image includes two or more special light images captured using two or more different special lights. Two or more special light images can be captured depending on the observation purpose, such as when the depth of the surface structure to be observed is different.

In an image processing device according to yet another aspect of the present invention, the first endoscopic image is a first special light image captured with a first special light, and the second endoscopic image is a second special light image captured with a second special light different from the first special light. In other words, the multiple endoscopic images may not include a normal light image.

In an image processing device according to yet another aspect of the present invention, it is preferable that the display control unit causes the display unit to display, as a moving image, an observation image calculated using a part of the multiple images or the multiple images. This makes it possible to carry out inspection in real time while viewing the observation image and the recognition result displayed as a moving image.

In an image processing device according to yet another aspect of the present invention, it is preferable that the recognizer recognizes an area of interest contained in the multiple images, and the display control unit displays an indicator indicating the recognized area of interest superimposed on the image displayed on the display unit. This makes it possible to support the inspection so that an area of interest in the observation image is not overlooked.

In an image processing device according to yet another aspect of the present invention, it is preferable that the recognizer recognizes an area of interest contained in a plurality of images, and the display control unit displays information indicating the presence or absence of an area of interest so as not to overlap with the image displayed on the display unit. This makes it possible to notify the user that an area of interest exists in the image for observation, and also makes it possible to prevent the information displayed on the display unit from impeding the observation of the observation image.

In an image processing device according to yet another aspect of the present invention, it is preferable that the recognizer performs lesion discrimination based on the multiple images and outputs the discrimination results, and the display control unit displays the discrimination results on the display unit. This makes it possible to visually inspect the observation images while referring to the discrimination results by the recognizer.

An endoscope system according to yet another aspect of the present invention includes a light source device that sequentially generates a first observation light and a second observation light different from the first observation light, an endoscope scope that captures multiple images by sequentially capturing images of an observation target that is sequentially illuminated by the first observation light and the second observation light, a display unit, and the above-mentioned image processing device, and a recognizer that receives an image set consisting of the multiple images captured by the endoscope scope.

In an endoscope system according to yet another aspect of the present invention, it is preferable that the endoscope system includes an endoscope processor that receives multiple images captured by the endoscope scope and performs image processing on the multiple images, and the recognizer receives the multiple images after image processing by the endoscope processor. The endoscope processor has a function of performing image processing on the multiple images captured by the endoscope scope, and the recognizer can detect and identify diseased areas using the multiple images after image processing. The recognizer may be separate from the endoscope processor, or may be built into the endoscope processor.

An image processing method according to yet another aspect of the present invention includes a first step of receiving an image set consisting of a plurality of images acquired using a plurality of different observation lights, a second step of a recognizer outputting a recognition result for the image set, and a third step of a display control unit causing a display unit to display an observation image and the recognition result calculated using a portion of the plurality of images or the plurality of images, and repeatedly executes the processing from the first step to the third step.

In an image processing method according to yet another aspect of the present invention, the second step preferably comprises a recognizer having a trained model trained from a training image set and ground truth data, which outputs a recognition result based on the trained model each time the recognizer receives an image set for recognition.

In an image processing method according to yet another aspect of the present invention, it is preferable that the trained model is composed of a convolutional neural network.

In an image processing method according to yet another aspect of the present invention, it is preferable that the multiple images include a first endoscopic image and a second endoscopic image acquired using an observation light different from that of the first endoscopic image.

In an image processing method according to yet another aspect of the present invention, it is preferable that the first endoscopic image is a normal light image captured with normal light, and the second endoscopic image is a special light image captured with special light.

According to the present invention, recognition of an image set is performed based on an image set consisting of a plurality of images acquired sequentially using a plurality of different observation lights, so that recognition accuracy can be improved. In addition, the recognition result can be appropriately presented by displaying the recognition result on the display unit together with the observation image obtained from the plurality of images.

FIG. 1 is a perspective view showing the appearance of an endoscope system 10 according to the present invention. FIG. 2 is a block diagram showing the electrical configuration of the endoscope system 10. As shown in FIG. FIG. 3 is a diagram showing an example of a multi-frame image and an image set captured primarily in the multi-frame shooting mode. FIG. 4 is a schematic diagram showing a typical example of the configuration of a convolutional neural network, which is one of the learning models constituting the recognizer 15. FIG. 5 is a schematic diagram showing an example of the configuration of the intermediate layer 15B of the CNN 15 shown in FIG. FIG. 6 is a block diagram showing the main configuration used to explain the operation of the endoscope system 10 according to the present invention. FIG. 7 is a diagram showing an example of an R image, a G image, a B image, and a V image captured frame-sequentially and an image set. FIG. 8 is a flow chart showing an embodiment of an image processing method according to the present invention.

Below, preferred embodiments of the image processing device, endoscope system, and method of operating the image processing device according to the present invention will be described with reference to the attached drawings.

[Overall configuration of endoscope system]
FIG. 1 is a perspective view showing the appearance of an endoscope system 10 according to the present invention.

As shown in FIG. 1, the endoscope system 10 mainly comprises an endoscope scope (here, a flexible endoscope) 11 that captures images of an object to be observed inside a subject, a light source device 12, an endoscope processor 13, a display unit (display device) 14 such as an LCD monitor, and a recognition device 15.

The light source device 12 supplies various types of observation light to the endoscope 11, such as white light for capturing normal light images and light in a specific wavelength band for capturing special light images.

The endoscope processor 13 has an image processing function for generating image data of a normal light image, a special light image, or an observation image for display/recording based on the image signal obtained by the endoscope scope 11, a function for controlling the light source device 12, and a function for displaying the normal image or the observation image and the recognition result by the recognizer 15 on the display 14. The recognizer 15 will be described in detail later, but it is the part that receives the endoscopic image from the endoscope processor 13 and recognizes the position detection of the area of interest in the endoscopic image (lesion, surgical scar, treatment scar, treatment tool, etc.) and the differentiation of the type of lesion, etc.

The display 14 displays a normal image, a special light image or an observation image based on the display image data input from the endoscope processor 13, and the recognition results by the recognizer 15.

The endoscope 11 is equipped with a flexible insertion section 16 that is inserted into the subject, a handheld operation section 17 that is connected to the base end of the insertion section 16 and is used to grasp the endoscope 11 and operate the insertion section 16, and a universal cord 18 that connects the handheld operation section 17 to the light source device 12 and the endoscope processor 13.

The distal end 16a of the insertion section 16 contains an illumination lens 42, an objective lens 44, an image sensor 45, and the like (see FIG. 2). A freely bendable curved section 16b is connected to the rear end of the insertion section distal end 16a. A flexible tube section 16c having flexibility is connected to the rear end of the curved section 16b.

The handheld operation unit 17 is provided with an angle knob 21, an operation button 22, a forceps inlet 23, and the like. The angle knob 21 is rotated to adjust the bending direction and amount of bending of the bending portion 16b. The operation button 22 is used for various operations such as air supply, water supply, and suction. The forceps inlet 23 is connected to the forceps channel in the insertion section 16. The handheld operation unit 17 is also provided with an endoscope operation unit 46 (see FIG. 2) for performing various settings, and the like.

The universal cord 18 incorporates air and water supply channels, signal cables, and light guides. At the tip of the universal cord 18, a connector portion 25a that is connected to the light source device 12 and a connector portion 25b that is connected to the endoscope processor 13 are provided. As a result, observation light is supplied from the light source device 12 to the endoscope scope 11 via the connector portion 25a, and an image signal obtained by the endoscope scope 11 is input to the endoscope processor 13 via the connector portion 25b.

The light source device 12 is provided with a light source operation section 12a including a power button, a light button for turning on the light source, and a brightness adjustment button, and the endoscope processor 13 is provided with a processor operation section 13a including a power button and an input section for receiving input from a pointing device such as a mouse (not shown). In this example, the endoscope processor 13 and the light source device 12 are separate, but the endoscope processor may be built into the light source device.

[Electrical configuration of the endoscope system]
FIG. 2 is a block diagram showing the electrical configuration of the endoscope system 10. As shown in FIG.

As shown in FIG. 2, the endoscope 11 has, broadly speaking, a light guide 40, an illumination lens 42, an objective lens 44, an image sensor 45, an endoscope operation unit 46, an endoscope control unit 47, and a ROM (Read Only Memory) 48.

The light guide 40 may be a large-diameter optical fiber, a bundle fiber, or the like. The light guide 40 has an input end inserted into the light source device 12 via the connector portion 25a, and an output end passing through the insertion portion 16 and facing an illumination lens 42 provided in the tip portion 16a of the insertion portion. The illumination light supplied from the light source device 12 to the light guide 40 is irradiated onto the observation object through the illumination lens 42. The illumination light reflected and/or scattered by the observation object is then incident on the objective lens 44.

The objective lens 44 forms an image of the reflected or scattered light of the incident illumination light (i.e., an optical image of the object being observed) on the imaging surface of the imaging element 45.

The imaging element 45 is a CMOS (complementary metal oxide semiconductor) type or CCD (charge coupled device) type imaging element, and is positioned and fixed relative to the objective lens 44 at a position behind the objective lens 44. A plurality of pixels, each composed of a plurality of photoelectric conversion elements (photodiodes) that photoelectrically convert an optical image, are two-dimensionally arranged on the imaging surface of the imaging element 45. In addition, red (R), green (G), and blue (B) color filters are arranged for each pixel on the incident surface side of the plurality of pixels of the imaging element 45 in this example, thereby forming R pixels, G pixels, and B pixels. The filter arrangement of the RGB color filters is generally a Bayer arrangement, but is not limited to this.

The imaging element 45 converts the optical image formed by the objective lens 44 into an electrical image signal and outputs it to the endoscope processor 13.

When the imaging element 45 is a CMOS type, an A/D (Analog/Digital) converter is built in, and a digital image signal is directly output from the imaging element 45 to the endoscope processor 13. When the imaging element 45 is a CCD type, the image signal output from the imaging element 45 is converted into a digital image signal by an A/D converter (not shown) and then output to the endoscope processor 13.

The endoscope operation unit 46 has a still image capture button (not shown), and an imaging mode setting unit that sets one of the imaging modes: normal light image capture mode, special light image capture mode, and multi-frame imaging mode. The imaging mode setting unit may be provided in the processor operation unit 13a of the endoscope processor 13.

The endoscope control unit 47 sequentially executes various programs and data read from the ROM 48 etc. in response to the operation of the endoscope operation unit 46, and mainly controls the operation of the image sensor 45. For example, in the normal light image shooting mode, the endoscope control unit 47 controls the image sensor 45 to read signals of the R pixel, G pixel, and B pixel of the image sensor 45, and in the special light image shooting mode or multi-frame shooting mode, when purple light is emitted from the V-LED 32a or blue light is emitted from the B-LED 32b as observation light to obtain a specific special light image, the endoscope control unit 47 controls the image sensor 45 to read only the signal of the B pixel of the image sensor 45 that has spectral sensitivity in the wavelength band of the purple light or blue light, or to read one or two of the three color pixels of the R pixel, G pixel, and B pixel.

The endoscope control unit 47 also communicates with the processor control unit 61 of the endoscope processor 13, and transmits to the endoscope processor 13 operation information from the endoscope operation unit 46 and identification information for identifying the type of endoscope scope 11 stored in the ROM 48.

The light source device 12 has a light source control unit 31 and a light source unit 32. The light source control unit 31 controls the light source unit 32 and communicates with the processor control unit 61 of the endoscope processor 13 to exchange various information.

The light source unit 32 has, for example, a plurality of semiconductor light sources. In this embodiment, the light source unit 32 has four color LEDs: a V-LED (Violet Light Emitting Diode) 32a, a B-LED (Blue Light Emitting Diode) 32b, a G-LED (Green Light Emitting Diode) 32c, and an R-LED (Red Light Emitting Diode) 32d. The V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d are semiconductor light sources that emit violet (V) light, blue (B) light, green (G) light, and red (R) light, which are observation light having peak wavelengths of, for example, 410 nm, 450 nm, 530 nm, and 615 nm, respectively.

The light source control unit 31 individually controls the turning on and off of the four LEDs of the light source unit 32, the amount of light emitted when turned on, etc., for each LED according to the shooting mode set by the shooting mode setting unit. In the normal light image shooting mode, the light source control unit 31 turns on all of the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d. Therefore, in the normal light image shooting mode, white light including V light, B light, G light, and R light is used as the observation light.

On the other hand, in the special light image capture mode, the light source control unit 31 turns on one of the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d light sources, or an appropriate combination of multiple light sources, or when multiple light sources are turned on, controls the light emission amount (light amount ratio) of each light source, thereby making it possible to capture images of multiple layers at different depths of the subject.

The multi-frame shooting mode is a shooting mode in which a normal light image and one or more special light images are switched for each frame, or two or more special light images are switched for each frame. In the multi-frame shooting mode, the light source control unit 31 causes the light source unit 32 to emit a different observation light for each frame.

The light of each color emitted by each LED 32a to 32d is incident on a light guide 40 inserted into the endoscope 11 via an optical path combination section formed by a dichroic mirror, lens, etc., and an aperture mechanism (not shown).

The observation light from the light source device 12 is selected from various wavelength bands according to the observation purpose, such as white light (light in a white wavelength band or light in multiple wavelength bands), light having a peak in one or multiple specific wavelength bands (special light), or a combination of these.

A first example of a specific wavelength band is, for example, the blue band or the green band in the visible range. This first example wavelength band includes a wavelength band of 390 nm to 450 nm or 530 nm to 550 nm, and the light of the first example has a peak wavelength within the wavelength band of 390 nm to 450 nm or 530 nm to 550 nm.

A second example of a specific wavelength band is, for example, the red band in the visible range. The wavelength band of this second example includes a wavelength band of 585 nm to 615 nm or 610 nm to 730 nm, and the light of the second example has a peak wavelength within the wavelength band of 585 nm to 615 nm or 610 nm to 730 nm.

A third example of the specific wavelength band includes a wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different, and the light of the third example has a peak wavelength in the wavelength band in which the absorption coefficients of oxygenated hemoglobin and reduced hemoglobin are different. The wavelength band of this third example includes a wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less, and the light of the third example has a peak wavelength in the above-mentioned wavelength band of 400±10 nm, 440±10 nm, 470±10 nm, or a wavelength band of 600 nm or more and 750 nm or less.

A fourth example of a specific wavelength band is the wavelength band (390 nm to 470 nm) of excitation light used to observe the fluorescence emitted by fluorescent substances in living organisms (fluorescence observation) and to excite these fluorescent substances.

A fifth example of a specific wavelength band is a wavelength band of infrared light. This fifth example wavelength band includes a wavelength band of 790 nm to 820 nm or 905 nm to 970 nm, and the fifth example light has a peak wavelength in the wavelength band of 790 nm to 820 nm or 905 nm to 970 nm.

The endoscope processor 13 includes a processor operation unit 13a, a processor control unit 61, a ROM 62, a digital signal processing circuit (DSP: Digital Signal Processor) 63, an image processing unit 65, a display control unit 66, and a memory unit 67.

The processor operation unit 13a includes a power button, an input unit that accepts inputs such as a coordinate position indicated on the screen of the display 14 by a mouse, and clicks (execution instructions).

The processor control unit 61 reads out necessary programs and data from the ROM 62 in response to operation information from the processor operation unit 13a and operation information from the endoscope operation unit 46 received via the endoscope control unit 47, and processes them sequentially to control each part of the endoscope processor 13 and to control the light source device 12. The processor control unit 61 may also receive necessary instruction inputs from other external devices, such as a keyboard connected via an interface (not shown).

The DSP 63, which functions as one form of image acquisition unit that acquires image data for each frame of the video output from the endoscope 11 (image sensor 45), performs various signal processing such as defect correction, offset processing, white balance correction, gamma correction, and demosaic processing (also called "synchronization processing") on the image data for one frame of the video input from the endoscope 11 under the control of the processor control unit 61, to generate image data for one frame.

The image processing unit 65 inputs image data from the DSP 63, and performs image processing such as color conversion, color enhancement, and structure enhancement on the input image data as necessary to generate image data showing an endoscopic image showing the observation target. Color conversion is a process in which the image data is converted into colors using 3x3 matrix processing, tone conversion, and 3D lookup table processing. Color enhancement is a process in which the image data that has already been color converted is enhanced in a direction that creates a difference in color between blood vessels and mucous membranes, for example. Structure enhancement is a process in which specific tissues and structures contained in the observation target, such as blood vessels and pit patterns, are emphasized, and is performed on the image data after color enhancement.

When an instruction is given to capture a still image or video, the image data for each frame of the video processed by the image processing unit 65 is recorded in the memory unit 67 as the still image or video that has been instructed to be captured.

The display control unit 66 generates display data for displaying a normal light image or a special light image on the display unit 14 based on image data input from the image processing unit 65, outputs the generated display data to the display unit 14, and causes the display unit 14 to display a display image (such as a video captured by the endoscope scope 11).

In the multi-frame shooting mode, if multiple images acquired sequentially using different observation lights are displayed sequentially as is, the appearance will change and flicker, so the display control unit 66 displays one of the multiple images (a part of an image) on the display 14, or displays an observation image calculated by the image processing unit 65 using the multiple images on the display 14.

The display control unit 66 also causes the display unit 14 to display the recognition results input from the recognizer 15 via the image processing unit 65, or the recognition results input from the recognizer 15.

When the recognition unit 15 detects an area of interest, the display control unit 66 displays an indicator indicating the area of interest by superimposing it on the image displayed on the display unit 14. For example, possible indicators include highlighting the area of interest in the displayed image by changing its color, displaying a marker, or displaying a bounding box.

The display control unit 66 can also display information indicating the presence or absence of an area of interest based on the result of the detection of the area of interest by the recognizer 15 so as not to overlap with the image displayed on the display unit 14. Possible forms of the information indicating the presence or absence of an area of interest include, for example, changing the color of the frame of the endoscopic image depending on whether an area of interest is detected or not, or displaying the text "Area of interest present!" in a display area different from the endoscopic image.

When the recognizer 15 performs a lesion discrimination, the display control unit 66 causes the display unit 14 to display the discrimination result. The discrimination result can be displayed, for example, by displaying text indicating the detection result on the display image of the display unit 14. The text does not have to be displayed on the display image, and is not particularly limited as long as the correspondence with the display image is clear.

[Recognizer 15]
Next, the recognizer 15 according to the present invention will be described.

The recognizer 15 accepts images after image processing by the endoscope processor 13, but first we will explain the images for recognition that the recognizer 15 accepts.

The recognizer 15 in this example is applied when the multi-frame shooting mode is set.

When the multi-frame shooting mode is set, the light source device 12 sequentially generates white light including purple light, blue light, green light, and red light, and light of one or more specific wavelength bands (special light) in which the illumination of the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d is controlled, and the endoscope processor 13 sequentially acquires an image under white light (normal light image) and an image under special light (special light image) from the endoscope scope 11.

In the multi-frame shooting mode of this example, as shown in FIG. 3, a normal light image (WL (White Light) image) which is the first endoscopic image, and two types of special light images (BLI (Blue Light Imaging or Blue LASER Imaging) image and LCI (Linked Color Imaging) image) which are the second endoscopic images are repeatedly acquired by sequentially switching between them for each frame.

Here, the BLI image and the LCI image are images captured using observation light for BLI and observation light for LCI, respectively.

The observation light used for BLI has a high ratio of V light, which has a high absorption rate in superficial blood vessels, and a low ratio of G light, which has a high absorption rate in mid-layer blood vessels, making it suitable for generating images (BLI images) that highlight the blood vessels and structures in the mucosal surface of the subject.

In addition, the observation light for LCI has a higher ratio of V light than the observation light for WL, and is more suitable for capturing subtle changes in color tone than the observation light for WL. The LCI image is an image that has undergone color enhancement processing, using the R component signal as well, to make reddish colors redder and whitish colors whiter, focusing on the colors near the mucous membrane.

The recognizer 15 accepts an image set Sa consisting of multiple images (in this example, a WL image, a BLI image, and an LCI image) acquired sequentially by the endoscope processor 13 as images for recognition.

The WL image, BLI image, and LCI image are each color images, and therefore have an R image, a G image, and a B image (three color channels). Therefore, the image set Sa input to the recognizer 15 is an image with 9 channels (= 3 x 3).

The recognizer 15 sequentially inputs the image sets Sa, but since the image set Sa is composed of three consecutive frames (WL image, BLI image, and LCI image) in chronological order, the time interval for inputting each image set Sa corresponds to the time for three frames captured in the multi-frame shooting mode. In other words, the time interval between the image set Sa at time t _n received by the recognizer 15 and the image set Sa at the previous time t _n-1 corresponds to the time for three frames captured in the multi-frame shooting mode.

Figure 4 is a schematic diagram showing a typical example of the configuration of a convolutional neural network (CNN), which is one of the learning models that constitute the recognizer 15.

CNN15 is a learning model that, for example, detects the position of areas of interest (lesions, surgical scars, treatment scars, treatment tools, etc.) in endoscopic images and distinguishes the type of lesion, and has a multiple layer structure and holds multiple weight parameters. CNN15 becomes a trained model and functions as a recognizer by setting the weight parameters to optimal values.

As shown in FIG. 4, CNN 15 includes an input layer 15A, an intermediate layer 15B having multiple convolutional layers and multiple pooling layers, and an output layer 15C, and each layer has a structure in which multiple "nodes" are connected by "edges."

The CNN 15 in this example is a learning model that performs segmentation to recognize the position of an area of interest in an endoscopic image, and applies a full convolutional network (FCN), a type of CNN, to it, making it possible to grasp the position of an area of interest in an endoscopic image at the pixel level.

The image set Sa for recognition (Figure 3) is input to the input layer 15A.

The intermediate layer 15B is a part that extracts features from the image set Sa input from the input layer 15A. The convolutional layer in the intermediate layer 15B filters the image set Sa or nearby nodes in the previous layer (performing a convolution operation using a filter) to obtain a "feature map." The pooling layer shrinks (or enlarges) the feature map output from the convolutional layer to create a new feature map. The "convolutional layer" is responsible for extracting features such as edge extraction from an image, and the "pooling layer" is responsible for providing robustness to the extracted features so that they are not affected by parallel movement, etc. Note that the intermediate layer 15B is not limited to cases where a convolutional layer and a pooling layer are one set, but may also include cases where convolutional layers are consecutive or a normalization layer.

The output layer 15C is a part that outputs the recognition results that detect the position of the area of interest in the endoscopic image and classify (differentiate) the type of lesion based on the features extracted by the intermediate layer 15B.

This CNN 15 is trained using a large number of sets of training image sets Sa and correct answer data for the image sets Sa, and the filter coefficients and offset values applied to each convolutional layer of the CNN 15 are set to optimal values using the training data sets. Here, the correct answer data is preferably a region of interest or a discrimination result designated by a doctor for an endoscopic image (in this example, at least one image in the image set Sa).

Figure 5 is a schematic diagram showing an example configuration of the intermediate layer 15B of the CNN 15 shown in Figure 4.

In the first (first) convolution layer of the region of interest, a convolution operation is performed between the image set Sa for recognition and the filter _F1 . Here, the image set Sa is N images (N channels) having an image size of H vertically and W horizontally. In this example, as shown in FIG. 3, the image set Sa is a 9-channel image.

The filter _F1 to be convoluted with this image set Sa has N channels (N images), so in the case of a filter of size 5, for example, the filter size is 5×5×N.

A convolution operation using this filter F ₁ generates a one-channel (one-sheet) "feature map" for one filter F _1. In the example shown in Fig. 5, M filters F ₁ are used to generate M-channel "feature maps."

If the filter _F1 used in the second convolutional layer is, for example, a filter of size 3, then the filter size will be a 3×3×M filter.

The size of the "feature map" in the nth convolutional layer is smaller than the size of the "feature map" in the second convolutional layer because it has been downscaled by the previous convolutional layers.

The convolutional layers in the first half of the intermediate layer 15B are responsible for extracting features, while the convolutional layers in the second half are responsible for segmenting the object (area of interest). The convolutional layers in the second half perform upscaling, and the final convolutional layer obtains a "feature map" of the same size as the input image set Sa. The output layer 15C (Figure 4) of CNN 15 uses the "feature map" obtained from the intermediate layer 15B to grasp the position of the area of interest in the images of image set Sa at the pixel level. In other words, it is possible to detect whether each pixel of the endoscopic image belongs to the area of interest or not, and output the detection result.

According to this embodiment, since recognition is performed using multiple images (a set of WL images, BLI images, and LCI images) acquired sequentially in multi-frame shooting mode, recognition accuracy can be improved compared to when recognition is performed using one (one type) of image among WL images, BLI images, and LCI images.

Although the CNN 15 in this example recognizes the position of the region of interest in the endoscopic image, the recognizer (CNN) according to the present invention is not limited to this, and may perform lesion discrimination and output the discrimination result. For example, the recognizer may classify the endoscopic image into three categories, "neoplastic", "non-neoplastic", and "other", and output three scores (the sum of the three scores is 100%) corresponding to "neoplastic", "non-neoplastic", and "other" as the discrimination result, or output the classification result if a clear classification can be made from the three scores. In addition, in the case of a CNN that outputs such a discrimination result, it is preferable to have a fully connected layer as the last layer or layers of the intermediate layer instead of a full-layer convolutional network (FCN).

[Operation of the endoscope system]
FIG. 6 is a block diagram showing the main configuration used to explain the operation of the endoscope system 10 according to the present invention.

From the V-LED 32a, B-LED 32b, G-LED 32c, and R-LED 32d of the light source unit 32, observation light (V light, B light, G light, and R light) having different peak wavelengths is irradiated onto the subject 20 via the light guide 40. Since the V light, B light, G light, and R light each reach multiple layers at different depths of the subject 20, it is possible to capture images of the subject 20 at different depths using these observation lights.

As explained in FIG. 3, in the multi-frame shooting mode, WL images, BLI images, and LCI images are acquired sequentially using multiple different observation lights (e.g., a first observation light for WL, a second observation light for BLI, and a third observation light for LCI), but the observation lights for WL, BLI, and LCI have different light intensity ratios of V light, B light, G light, and R light, as mentioned above.

In the endoscope scope 11, WL images, BLI images, and LCI images are captured in sequence and repeatedly by irradiating multiple different observation lights. Since the WL images, BLI images, and LCI images are each color images, the endoscope processor 13 generates WL images, BLI images, and LCI images in three channels of RGB.

The recognizer 15 accepts an image set Sa consisting of a WL image, a BLI image, and an LCI image (a total of nine channel images) as images for recognition.

The recognizer 15 detects the position of the area of interest (in this example, the lesion area) in the endoscopic image, and outputs position information (recognition result) indicating the lesion area to the endoscope processor 13.

The image processing unit 65 of the endoscope processor 13 generates a WL image, a BLI image, and an LCI image from the image signal input from the endoscope scope 11, and also generates an observation image. The observation image may be a part of a plurality of images (for example, the WL image of the WL image, the BLI image, and the LCI image), or may be an image calculated using a plurality of images (an image obtained by synthesizing two or more images of the WL image, the BLI image, and the LCI image). Note that if a plurality of images obtained in sequence using different observation lights are displayed in sequence as observation images, the appearance will change and flicker, so it is preferable that the observation image is one type of image.

The display control unit 66 inputs the observation image from the image processing unit 65 and the position information indicating the lesion area from the recognition unit 15, and displays the observation image and the recognition result on the display unit 14.

In this example, the display control unit 66 displays the observation image 26 on the display 14 and performs an emphasis process to emphasize the recognized area of interest (lesion area). The emphasis process by the display control unit 66 highlights the lesion area by superimposing an indicator 28 indicating the lesion area on the observation image 26 displayed on the display 14. Here, the display of the indicator 28 may be a highlighting display such as changing the color of the lesion area, a display of a boundary line showing the outline of the lesion area, a display of a marker indicating the lesion area, or a display of a bounding box.

In this way, by superimposing an indicator 28 indicating the area of interest on the observation image 26 displayed on the display 14, it is possible to assist in the examination so that the area of interest is not overlooked.

In this example, the recognizer 15 recognizes the position of the region of interest in the endoscopic image, but is not limited to this and may also perform a lesion discrimination and output the discrimination result. One possible method for displaying the discrimination result is to display text indicating the discrimination result on the image on the display 14. The display position of the text does not have to be on the image, and may be a window different from the image as long as the correspondence with the image is clear, and is not particularly limited.

[Another embodiment of multi-frame photography]
When color endoscopic images are obtained using an endoscope equipped with a monochrome imaging element without a color filter instead of the imaging element 45 (color imaging element), the subject is illuminated sequentially with observation light of different colors, and an image is captured for each observation light (images are captured in frame sequence).

For example, by sequentially emitting observation light of different colors (R light, G light, B light, and V light) from the light source unit 32, R images, G images, B images, and V images of colors corresponding to the R light, G light, B light, and V light are captured in frame sequence by the monochrome imaging element.

Figure 7 shows an example of an R image, a G image, a B image, and a V image captured frame-sequentially, and an image set.

The endoscope processor 13 can generate observation images such as WL images, BLI images, and LCI images based on a plurality of images (R image, G image, B image, and V image) acquired sequentially using a plurality of different observation lights (R light, G light, B light, and V light). These observation images can be generated by adjusting the composition ratio of the R image, G image, B image, and V image.

Also, an image obtained by multiplying at least two of the R, G, B, and V images by a preset coefficient and synthesizing them (by performing arithmetic operations) may be included in the image set. For example, an image obtained by dividing each pixel of an image with a central wavelength of 410 nm (V image) by an image with a central wavelength of 450 nm (B image), or an image obtained by multiplying each pixel of an image with a central wavelength of 410 nm (V image) by an image with a central wavelength of 450 nm (B image) may be used.

The recognizer 15 can accept the WL image, BLI image, and LCI image generated by the endoscope processor 13 as an image set Sb, and return the recognition results for the endoscopic image to the endoscope processor 13.

The recognizer 15 in this example accepts an image set Sa consisting of a WL image, a BLI image, and an LCI image (a total of nine channel images) as images for recognition, but is not limited to this. For example, the recognizer 15 may accept an image set consisting of the above-mentioned R image, G image, B image, and V image, and output the recognition result for the endoscopic image.

[Image processing method]
FIG. 8 is a flowchart showing an embodiment of an image processing method according to the present invention, showing the processing procedures of each part of the endoscope system 10 shown in FIG.

In FIG. 8, the multi-frame shooting mode is set, and the endoscope 11 sequentially captures multi-frame images using multiple different observation lights (step S10).

The endoscope processor 13 acquires a set of images constituting a multiframe image captured by the endoscope scope 11 (step S12, first step).

The image set may be a WL image, a BLI image, and an LCI image captured by the endoscope scope 11 using WL, BLI, and LCI observation lights, or a WL image, a BLI image, and an LCI image generated from R, G, B, and V images captured in frame sequence, but the special light image may be one of the BLI and LCI images, or a special light image captured with another special light. The image set may not include a WL image (normal light image), and may be two or more special light images including a first special light image captured with a first special light and a second special light image captured with a second special light. In short, any image set may be used as long as it is an image set consisting of a plurality of images captured sequentially using a plurality of different observation lights.

The image processing unit 65 of the endoscope processor 13 generates an image for observation based on the acquired image set (step S14). The image for observation is a part of the multiple images (for example, the WL image of the WL image, the BLI image, and the LCI image) or an image calculated using the multiple images.

On the other hand, the recognizer 15 performs operations such as detecting the position of the area of interest in the endoscopic image and differentiating the type of lesion based on the image set received via the endoscope processor 13, and outputs the recognition result (step S16, second step).

Then, the display control unit 66 causes the display 14 to display the generated observation image and the recognition result by the recognizer 15 (step S18, third step).

Next, it is determined whether or not to end the capture of the multi-frame image (step S20), and if the capture of the multi-frame image is to be continued (if "No"), the process transitions to step S10, and the processes from step S10 to step S20 are repeated. As a result, the observation image is displayed as a video, and the recognition results of the recognizer 15 are also displayed continuously.

When multi-frame image capture is complete (if "Yes"), this process ends.

[others]
In this embodiment, an endoscope system 10 including an endoscope scope 11 and the like has been described, but the present invention is not limited to the endoscope system 10 and may be an image processing device including an endoscope processor 13 and a recognizer 15. In this case, the endoscope processor 13 and the recognizer 15 may be integrated or separate.

The different observation lights are not limited to those emitted from four-color LEDs, but may be, for example, light sources that include a blue laser diode that emits blue laser light with a central wavelength of 445 nm and a blue-violet laser diode that emits blue-violet laser light with a central wavelength of 405 nm, and may be emitted by irradiating the laser light of these blue and blue-violet laser diodes on a YAG (Yttrium Aluminum Garnet) phosphor. When the blue laser light is irradiated on this phosphor, the phosphor is excited and a wide band of fluorescence is emitted, and some of the blue laser light passes through the phosphor as is. The blue-violet laser light passes through the phosphor without exciting it. Therefore, by adjusting the intensity of the blue laser light and the blue-violet laser light, it is possible to irradiate observation light for WL, observation light for BLI, and observation light for LCI, and by emitting only the blue-violet laser light, it is possible to irradiate observation light with a central wavelength of 405 nm.

In addition, the observation images according to the present invention are not limited to video images, but may be still images stored in the memory unit 67 or the like, and the recognizer may output recognition results based on an image set of still images.

Furthermore, the recognizer is not limited to CNN, but may be a machine learning model other than CNN, such as DBN (Deep Belief Network) or SVM (Support Vector Machine).

The hardware structure of the endoscope processor 13 and/or the recognizer 15 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various control units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), which is a processor whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically to execute specific processing.

A processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Multiple control units may also be composed of one processor. As an example of multiple control units being composed of one processor, first, as represented by computers such as clients and servers, there is a form in which one processor is composed of a combination of one or more CPUs and software, and this processor functions as multiple control units. Second, as represented by system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple control units in a single IC (Integrated Circuit) chip. In this way, the various control units are composed of one or more of the above various processors as a hardware structure.

Furthermore, the present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present invention.

10 Endoscope system 11 Endoscope scope 12 Light source device 12a Light source operation unit 13 Endoscope processor 13a Processor operation unit 14 Display 15 Recognizer (CNN)
15A Input layer 15B Intermediate layer 15C Output layer 16 Insertion section 16a Insertion section tip 16b Bending section 16c Flexible tube section 17 Hand operation section 18 Universal cord 20 Subject 21 Angle knob 22 Operation button 23 Forceps inlet 25a Connector section 25b Connector section 26 Observation image 28 Index 31 Light source control section 32 Light source unit 32a V-LED
32b B-LED
32c G-LED
32d R-LED
40 Light guide 42 Illumination lens 44 Objective lens 45 Image pickup element 46 Endoscope operation unit 47 Endoscope control unit 48, 62 ROM
61 Processor control unit 65 Image processing unit 66 Display control unit 67 Memory unit F1 Filter S Image set S10 Step S12 Step S14 Step S16 Step S18 Step S20 Step

Claims (10)

a recognizer that receives an image set including a normal light image and a special light image, which are endoscopic images acquired using normal light and special light, and outputs a recognition result for the image set;
a display control unit that causes an observation image composed of the normal light image or the special light image that is a part of the image set and the recognition result, or an observation image calculated using the image set and the recognition result to be displayed on a display unit,
the image set includes a normal light image captured by illuminating an observation target with the normal light and the special light, and a special light image captured by illuminating the observation target with the normal light and the special light between the normal light and the special light,
The recognizer has a trained model trained with a set of the image set for learning and ground truth data indicating the position of a lesion, and outputs the recognition result indicating the position of a lesion included in the image set for recognition based on the trained model every time the image set for recognition is received.
Image processing device. The image processing device according to claim 1, wherein the trained model is composed of a convolutional neural network. The image processing device according to claim 1 or 2, wherein the special light image includes two or more special light images captured using two or more different special lights. The image processing device according to any one of claims 1 to 3, wherein the display control unit causes the observation image to be displayed on the display unit as a video. The recognizer recognizes a location of a lesion area included in the image set for the recognition;
The image processing device according to claim 1 , wherein the display control unit causes an index indicating the recognized lesion area to be superimposed on the observation image displayed on the display unit. The recognizer recognizes a location of a lesion area included in the image set for recognition;
The image processing device according to claim 1 , wherein the display control unit displays the information indicating the presence or absence of a lesion area so as not to overlap with the observation image displayed on the display unit. a light source device that sequentially generates the normal light and the special light;
an endoscope that sequentially captures normal light images and special light images by sequentially capturing images of an observation target that is sequentially illuminated with the normal light and the special light;
The display unit;
The image processing device according to any one of claims 1 to 6 ,
The recognizer is an endoscope system that receives the image set consisting of the normal light image and the special light image captured by the endoscope scope. an endoscope processor that receives the normal light image and the special light image captured by the endoscope and performs image processing of the normal light image and the special light image,
The endoscope system according to claim 7 , wherein the recognizer accepts the image set consisting of the normal light image and the special light image after image processing by the endoscope processor. A method for operating an image processing device having one recognizer and a display control unit, comprising the steps of:
A first step in which the recognizer receives an image set including a normal light image and a special light image, which are endoscopic images acquired using normal light and special light;
a second step in which the recognizer outputs a recognition result for the set of images;
a third step in which the display control unit receives the image set for recognition, and causes a display unit to display an observation image consisting of the normal light image or the special light image that is a part of the image set and the recognition result, or an observation image calculated using the image set and the recognition result,
Repeating the processes from the first step to the third step;
the image set includes a normal light image captured by illuminating an observation target with the normal light and the special light, and a special light image captured by illuminating the observation target with the normal light and the special light between the normal light and the special light,
The second step has a trained model trained on the image set for learning and ground truth data indicating the position of a lesion as a set, and outputs the recognition result indicating the position of a lesion included in the image set for recognition based on the trained model every time the image set for recognition is received.
A method for operating an image processing device. The method of claim 9 , wherein the trained model is composed of a convolutional neural network.

Images