JP7653838B2 - Endoscope system and method of operation thereof - Google Patents

Description

The present invention relates to an endoscope system having multiple types of image analysis means and its operating method.

Swallowing disorders are a condition in which food and drink cannot be swallowed properly. It is said that people with swallowing disorders are more likely to choke on food accidentally entering the airway, or develop aspiration pneumonia. Because swallowing disorders occur in conjunction with aging and neurological diseases, it is becoming increasingly important to test swallowing function in today's aging society. Swallowing function tests are performed to properly treat and prevent swallowing disorders, and to identify the pathology of aspiration.

Several new methods are being developed for testing swallowing function. For example, in Patent Document 1, audio waveform data including swallowing sounds is acquired, and swallowing sounds such as epiglottic closure sounds, esophageal passage sounds, and epiglottic opening sounds are analyzed in detail. In addition, in Patent Document 2, a three-dimensional shape measurement device is used to measure and analyze the movement of the skin and superficial muscles and the distance between the corners of the mouth to evaluate swallowing function.

JP 2016-185209 A International Publication No. 2018/193955

The measurement methods described in Patent Documents 1 and 2 are relatively new techniques, and clinically, videofluoroscopic examination of swallowing (VF) and videoendoscopic examination of swallowing (VE) using X-rays have been established as methods for evaluating swallowing disorders (swallowing function evaluation tests). A videofluoroscopic examination of swallowing is an examination in which a subject swallows a contrast agent to obtain radiological images of the pharynx, larynx, and esophagus during swallowing, while a swallowing endoscopy is an examination in which an endoscope is inserted transnasally into the body to obtain endoscopic images of the pharynx and larynx, particularly the epiglottis area, during swallowing. In order to perform accurate and simple diagnosis, and to obtain information on the detailed pathology of swallowing disorders, a method for detailed analysis of the images obtained by these examinations is required.

The present invention aims to provide an endoscope system and an operating method thereof that perform detailed analysis of images obtained during a swallowing endoscopy examination.

The endoscope system of the present invention illuminates a subject, captures reflected light from the subject, and is equipped with a control processor. The control processor acquires an examination image and determines that swallowing is occurring when the amount of change over time is equal to or greater than a first threshold, when an image feature value related to the examination image is equal to or greater than a second threshold, when the number of image feature points included in the examination image is equal to or less than a third threshold, or when swallowing-related sounds are acquired.

It is preferable that the amount of change is a value calculated using the absolute value of the simple pixel value difference between successive inspection images in a time series.

It is preferable that the swallowing-related sounds are audio signals that are determined to be sounds produced by the patient by analyzing the audio signals acquired by the control processor.

It is preferable that the control processor determines the audio signal as a swallowing-related sound or a non-swallowing-related sound, which is either a cough, choking, breathing sound or vocalization.

When the control processor determines from the test image that swallowing is occurring and non-swallowing related sounds are acquired, it is preferable to determine that the test image is not swallowing.

It is preferable that the control processor inputs the test image into a classifier, calculates the probability that swallowing is occurring, and determines that swallowing is occurring if the probability is equal to or greater than a fourth threshold.

The control processor preferably inputs the test image and the audio signal to a classifier, calculates the probability of swallowing, and determines that swallowing is occurring when the probability is equal to or greater than a fourth threshold. The classifier is preferably trained on images that have been determined to be swallowing or not swallowing.

When a series of test images determined to be during swallowing is treated as a swallowing video and a series of test images determined to be during non-swallowing is treated as a non-swallowing video, and when a continuous non-swallowing period during which a non-swallowing video was acquired exceeds a certain period after a continuous swallowing period during which a swallowing video was acquired, the control processor preferably determines that the swallowing video acquired during the continuous swallowing period immediately prior to the continuous non-swallowing period that exceeded the certain period is one swallowing action and counts the number of swallows.

It is preferable that the control processor determines a swallowing video acquired during a continuous swallowing period before a continuous non-swallowing period that exceeds a certain period and a non-swallowing video acquired during a continuous non-swallowing period not exceeding a certain period as one swallowing action, and counts the number of swallows. It is preferable that the certain period can be set arbitrarily.

It is preferable that the control processor displays a display screen that enables recognition of a portion of the test images arranged in chronological order that has been determined to represent a single swallowing action.

The method of operating an endoscope system of the present invention is a method of operating an endoscope system that includes a control processor, illuminates a subject, and captures reflected light from the subject, and includes the steps of acquiring test images, and determining that swallowing is occurring when the amount of change between chronologically successive test images is equal to or greater than a first threshold, when an image feature amount related to the test image is equal to or greater than a second threshold, when the number of image feature points included in the test image is equal to or less than a third threshold, or when swallowing-related sounds are acquired.

The endoscope system and operating method of the present invention can provide an endoscope system and operating method that perform detailed analysis of images and the like obtained during a swallowing endoscopy examination.

FIG. 1 is a configuration diagram of an endoscope system. 1 is a graph showing the spectrum of normal light. 13 is a block diagram showing functions of a swallowing determination unit. FIG. FIG. 1 is an explanatory diagram showing swallowing. FIG. 1 is an explanatory diagram showing aspiration. 1A and 1B are explanatory diagrams and image diagrams showing a method for capturing an inspection image. FIG. 11 is an explanatory diagram showing a method for determining swallowing in the first determination unit. FIG. 11 is an explanatory diagram showing a method for determining swallowing in the second determination unit. FIG. 13 is an explanatory diagram showing a method for determining swallowing in the third determination unit. 13 is a flowchart showing a swallowing determination method in a fourth determination unit. 13 is a flowchart showing a swallowing determination method using a test image and a voice signal in the swallowing determination unit. FIG. 13 is an explanatory diagram showing a method for determining swallowing in the fifth determination unit. 13 is an explanatory diagram showing a test image transmitted to a swallowing counting unit; FIG. FIG. 13 is an explanatory diagram showing a method of counting the number of swallows in a swallowing number counter. FIG. 2 is an image diagram showing a display screen. 1 is a flowchart illustrating an embodiment of the present invention.

As shown in FIG. 1, the endoscope system 10 includes an endoscope 12, a light source device 14, a processor device 15, a computer 16, a recording device 17, a display 18, and a user interface 19. The endoscope 12 is optically connected to the light source device 14 and electrically connected to the processor device 15. The endoscope 12 has an insertion section 12a that is inserted into the body of an observation subject, an operation section 12b provided at the base end of the insertion section 12a, and a bending section 12c and a tip section 12d provided at the tip side of the insertion section 12a. The bending section 12c is bent by operating the angle knob 12e of the operation section 12b. The tip section 12d is directed in a desired direction by the bending operation of the bending section 12c. The endoscope 12 may be a fiberscope or may be located at the operation section side end of the insertion section 12a. The endoscope 12 is an endoscope used for swallowing endoscopy.

Inside the endoscope 12, an optical system for forming an image of a subject and an optical system for irradiating the subject with illumination light are provided. The subject is a structure in a living body related to the swallowing movement. Specifically, it is the pharynx and larynx. In addition to the angle knob 12e, the operation unit 12b is provided with a still image acquisition command switch 12h used to command acquisition of a still image of the observation subject, and a zoom operation unit 12i used to operate the zoom lens.

The light source device 14 generates illumination light. The processor device 15 controls the endoscope system 10 and performs image processing on image signals transmitted from the endoscope 12. The display 18 is a display unit that displays images captured by the endoscope 12. The user interface 19 is an input device that inputs settings to the processor device 15, etc.

The light source device 14 includes a light source unit 20 that emits illumination light and a light source control unit 22 that controls the operation of the light source unit 20.

The light source unit 20 emits illumination light that illuminates the subject. The light source unit 20 includes a light source such as a laser diode, an LED (Light Emitting Diode), a xenon lamp, or a halogen lamp, and emits illumination light (normal light) with at least the spectrum shown in FIG. 2. The white light includes so-called pseudo-white light that is a mixture of purple light V, blue light B, green light G, or red light R as shown in FIG. 2, which is substantially equivalent to white light when imaging a subject using the endoscope 12. The light source unit 20 also includes an optical filter or the like that adjusts the wavelength band, spectrum, or light amount of the illumination light as necessary.

The light source control unit 22 controls the on/off of each light source constituting the light source unit 20, and the amount of light emitted. An illumination optical system and an imaging optical system are provided at the tip 12d of the endoscope 12. Illumination light emitted by the light source unit 20 enters the endoscope 12 via a light guide, and is emitted from the tip 12d toward the subject via an illumination lens of the illumination optical system. The imaging optical system has an objective lens and an imaging sensor. Light from the observation object as a result of irradiation with illumination light enters the imaging sensor via the objective lens and zoom lens. As a result, an image of the observation object is formed on the imaging sensor. The zoom lens is a lens for enlarging the observation object, and is moved between the telephoto end and the wide end by operating the zoom operation unit 12i.

The imaging sensor may be a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge-Coupled Device) sensor, etc. An inspection image is generated based on the image signal detected by the imaging sensor.

The image sensor may include a color image sensor equipped with a color filter (such as a Bayer filter) that converts the sensed light into a color image signal, as well as a monochrome image sensor that does not have a color filter that converts the sensed light into a monochrome image signal. Note that the color image sensor may convert the sensed light into a CMY signal instead of an RGB signal.

When acquiring a color image, the image signal includes a B image signal output from the B pixel, a G image signal output from the G pixel, and an R image signal output from the R pixel. The image signal is transmitted to the image acquisition unit 31 of the processor device 15, and acquired as a test image which is a monochrome image or a color image. The test image acquired by the image acquisition unit 31 is transmitted to the image input unit 33 of the computer 16. The test image transmitted to the image input unit 33 is transmitted to the swallowing assessment unit 40. The test image is a series of chronologically continuous moving images captured during an endoscopic examination.

The processor device 15 includes a control unit 30, an image acquisition unit 31, and a display control unit 32. In the processor device 15, the functions of the image acquisition unit 31 and the display control unit 32 are realized by the control unit 30, which is composed of a control processor, running a program in the program memory.

The computer 16 includes an image input unit 33, a swallowing assessment unit 40, and a result recording unit 34. In the computer 16, a central control unit (not shown) consisting of a control processor runs a program in a program memory to realize the functions of the image input unit 33, the swallowing assessment unit 40, and the result recording unit 34. Note that the computer 16 and/or the light source control unit 22 may be included in the processor device 15.

The function of the swallowing determination unit 40 will be described below with reference to FIG. 3. The swallowing determination unit 40 includes a first determination unit 41, a second determination unit 42, a third determination unit 43, a fourth determination unit 44, a fifth determination unit 45, and a swallowing count unit 47. The swallowing determination unit 40 determines whether the acquired test image or sound is during swallowing, and further determines the timing of one swallowing action for a video determined to be during swallowing or not during swallowing, and counts the number of swallowing actions. The result recording unit 34 records the timing and number of swallowing actions, and generates images to be displayed on the display 18 and images to be output to the recording device 17, and edits the videos.

Swallowing refers to a series of actions that involves putting food or drink into the mouth, chewing, swallowing, and sending it down the esophagus. Figure 4 is an explanatory diagram of normal swallowing, and Figure 5 is an explanatory diagram of abnormal swallowing (aspiration). As shown in Figure 4, the swallowing movement is divided into the "oral stage" in which food F is transported from the oral cavity to the pharynx mainly by the movement of the tongue To, the "pharyngeal stage" in which food F is transported from the pharynx to the esophagus Es by the swallowing reflex, and the "esophageal stage" in which food F is transported from the esophagus Es to the stomach by the peristaltic movement of the esophagus. When swallowing, the epiglottis Eg, which serves to cover the trachea Tr, closes the entrance (glottis) of the trachea Tr by a reflex movement to direct food F towards the esophagus Es and prevent it from flowing into the trachea Tr. In addition, the soft palate Sp, which is the roof of the oral cavity, also moves backwards to close the passage between the oral cavity and the nasal cavity, preventing the esophagus F from entering the nasal cavity. If some kind of functional disorder occurs during the oral, pharyngeal, or esophageal phase, food F that would normally be transported to the esophagus Es will flow into the trachea Tr, as shown in Figure 5. This is called aspiration.

Example 1 of aspiration in FIG. 5 is an example of aspiration in which food F flows into the trachea Tr before the swallowing reflex occurs from the oral to pharyngeal phase. Example 2 of aspiration in FIG. 5 is an example of aspiration in which food F flows into the trachea Tr due to incomplete closure of the glottis (the entrance to the trachea Tr) by the epiglottis Eg halfway through the swallowing reflex from the pharyngeal to esophageal phase. Example 3 of aspiration in FIG. 5 is an example of aspiration in which food F remaining in the epiglottic vallecula Ev or the pyriform sinuses, which are depressions on the left and right sides of the entrance to the esophagus (see example 100 of the examination image in FIG. 6), flows into the trachea Tr after the swallowing reflex.

The examination image acquired in this embodiment is captured by inserting the insertion section 12a of the endoscope 12 from the nasal cavity into the pharynx so that the tip 12d of the endoscope is located near the position R of the oropharynx shown in FIG. 6. As shown in the example examination image 100 in FIG. 6, the examination image preferably includes anatomical structures such as the epiglottis Eg, glottic cleft Rg, and left and right pyriform sinuses Ps. The glottic cleft Rg is the space between the left and right folds that make up the vocal cords. The following describes the case where the tip of the endoscope is placed in the oropharynx, but swallowing may also be judged by placing it in the nasopharynx, nasopharynx, hypopharynx, or larynx.

The swallowing judgment is performed by one or more of the first judgment unit 41, the second judgment unit 42, the third judgment unit 43, the fourth judgment unit 44, and the fifth judgment unit 45. In order to improve the accuracy of the judgment, the swallowing judgment may be performed by combining the judgment results of the first to fifth judgment units. The analysis performed by the swallowing judgment unit 40 is described below.

The first determination unit 41 calculates the amount of change related to images of two frames that are adjacent in time series as the first swallowing detection algorithm. The amount of change between the test images is preferably a value obtained by using the absolute value of the simple pixel value difference between the test images that are adjacent in time series. An area of at least 10 pixels or more in the vertical and horizontal directions from the center of the test image for each frame (image processing target area 41g) is the target of image processing. As illustrated in FIG. 7, the amount of change is taken as the difference in pixel values in the image processing target area 41g between the test image of the previous frame that is adjacent in time series (test image 41a in the upper part of FIG. 7 and test image 41d in the lower part of FIG. 7) and the test image of the subsequent frame (test image 41b in the upper part of FIG. 7 and test image 41e in the lower part of FIG. 7). If this amount of change is equal to or greater than the first threshold, the test image 41e of the frame after the amount of change is calculated is determined to be swallowing. The first swallowing detection algorithm utilizes the fact that the subject (especially the area around the epiglottis Eg) moves vigorously during swallowing, resulting in a larger amount of change between frames compared to when not swallowing. The pixel value, which is the first threshold value, is preferably between 1 and 255, and can be set arbitrarily.

A specific example will be described in detail. The upper part of FIG. 7 is an example of the calculation of the change amount of the test images of two frames (test image 41a of the previous frame and test image 41b of the following frame) before and after the swallowing movement does not occur. In the state where the swallowing movement does not occur, the epiglottis Eg is open, and the glottis Rg can be easily observed. In addition, the soft palate (see soft palate Sp in FIG. 4), which is the roof of the oral cavity, and the epiglottis Eg hardly move, and there is only a slight movement due to breathing. Therefore, the movement of the tip 12d of the endoscope 12 is small, and as shown in the example of the change amount 41c in the upper part of FIG. 7, there is almost no movement or blurring in the entire image. Therefore, the change amount between the test image 41a of the previous frame and the test image 41b of the following frame in the image processing target region 41g is less than the first threshold, and the test image 41b of the following frame is determined to be not swallowing. In the example of the change amount 41c, the part calculated as the change amount is illustrated by a line.

The lower part of FIG. 7 shows an example of the calculated change in the test images (test image 41d of the previous frame and test image 41e of the following frame) before and after the swallowing movement. During the swallowing movement, the soft palate (see soft palate Sp in FIG. 4) moves and the epiglottis Eg closes the glottis Rg, causing the tip 12d of the endoscope 12 to move violently, resulting in a large blur in the entire image, as shown in the example of change 41f in the lower part of FIG. 7. In this case, the change between the test image 41d of the previous frame and the test image 41e of the following frame in the image processing target region 41g is equal to or greater than the first threshold, and the test image 41e of the following frame is determined to be swallowing. As in the upper part, in the example of change 41f, the calculated change is illustrated by a line. Also, because there is a large amount of blurring between frames during swallowing, the example of change in the lower row 41f, which shows the amount of change during swallowing, has more lines indicating the amount of change than the example of change during non-swallowing 41c.

The second determination unit 42 calculates the image feature of the test image as the second swallowing detection algorithm. The image feature is preferably an edge amount calculated from the test image. The image processing target is an area (image processing target area 42g) of at least 10 pixels in the vertical and horizontal directions from the image center of at least one frame of the test image. As illustrated in FIG. 8, the variance of Laplacian, which is a value calculated using an edge amount detection filter, is obtained for the test image (test image 42a in the upper part of FIG. 8 and test image 42c in the lower part of FIG. 8), and the image feature (example of image feature 42b in the upper part of FIG. 8 and example of image feature 42d in the lower part of FIG. 8) is calculated from the variance of Laplacian. If the image feature is equal to or greater than the second threshold, the test image of the frame in which the image feature was calculated is determined to be in the process of swallowing. The second swallowing detection algorithm utilizes the fact that the tip of the endoscope 12 moves vigorously during swallowing, and as the blur increases, the edge amount also increases. The pixel value, which is the second threshold value, is preferably between 1 and 255 and can be set arbitrarily. Although the method of calculating the Variance Of Laplacian has been exemplified as a method of calculating image features, other methods that use edge detection filters such as the Sobel Filter, Prewitt Filter, and Canny method may also be used.

A specific example will be described in detail. The top part of Figure 8 is an example of calculation of image features of test image 42a when no swallowing movement is occurring. When no swallowing movement is occurring, the soft palate and epiglottis Eg hardly move, and the tip 12d of the endoscope 12 also hardly moves, and as shown in the example image feature 42b in the top part of Figure 8, there is almost no movement or blurring in the entire image. In this case, the image feature in the image processing target region 42g is less than the second threshold, and test image 42a is determined to be not swallowing. Note that in the example image feature 42b, the areas that were the subject of image feature calculation are illustrated with lines.

The lower part of FIG. 8 shows an example of the calculation of image feature amounts of the test image 42c in which a swallowing movement is occurring. When a swallowing movement is occurring, the tip 12d of the endoscope 12 also moves significantly due to the movement of the soft palate and epiglottis Eg, and blurring occurs throughout the image, so the image feature amount becomes large, as shown in the example image feature amount 41d in the lower part of FIG. 8. In this case, the image feature amount in the image processing target region 42g is equal to or greater than the second threshold, and the test image 42c is determined to be in the process of swallowing. As in the upper part, in the example image feature amount 42d, the areas that are the subject of the calculation of the image feature amount are illustrated by lines. Also, because the amount of blurring is large during swallowing, the example image feature amount 42d in the lower part, which shows the swallowing movement, has more lines indicating the image feature amount than the example image feature amount 42b in the non-swallowing state.

As a third swallowing detection algorithm, the third determination unit 43 calculates the number of image feature points extracted from the edges extracted by expressing the test image as lines, using as key points the parts with a high degree of corners with a large edge amount. The image processing target is an area of at least 10 pixels vertically and horizontally from the image center of at least one frame of the test image (image processing target area 43g). As illustrated in FIG. 9, image feature points 43c are extracted from the test image (test image 43a in the upper part of FIG. 9 and test image 43b in the lower part of FIG. 9) and the number is calculated. If the number of image feature points 43c is equal to or less than the third threshold, the test image of the frame in which the number of image feature points was calculated is determined to be in the process of swallowing. The third swallowing detection algorithm utilizes the fact that the tip of the endoscope 12 moves violently during swallowing, resulting in large blur. The value of the third threshold can be set arbitrarily to 0 or more. Also, when the number of image feature points is equal to or less than a third threshold, a swallowing determination value may be calculated by multiplying the pixel value by -1, and when the swallowing determination value is less than the threshold, it may be determined that swallowing is occurring. In this case, when the number of image feature points exceeds the third threshold, it is determined that not swallowing is occurring.

Note that, for feature point extraction, it is preferable to use AKAZE (Accelarated KAZE), which is a type of calculation algorithm. For feature point extraction in this embodiment, it is preferable to recognize parts in the image with a high amount of edges (parts recognized as "angles" or "corners"). For feature point extraction, other feature point extraction methods such as the Scale Invariant Feature Transform (SIFT) method and the Speeded-Up Robust Features (SURF) method may also be used.

A specific example will be described in detail. The top part of Figure 9 shows an example of calculating the number of image feature points 43c in an examination image 43a in which no swallowing movement is occurring. In a state in which no swallowing movement is occurring, the tip 12d of the endoscope 12 barely moves, and there is almost no movement or blurring in the entire image, so that the number of detected image feature points 43c is large, as shown in the examination image 43a in the top part of Figure 9. In the example of Figure 9, if the third threshold is set to 5, the number of image feature points 43c in the image processing target region 43g is 30, which exceeds the third threshold, and therefore the examination image 43a is determined to be not swallowing.

The bottom row of Figure 9 shows an example of calculating the number of image feature points 43c in test image 43b in which swallowing is occurring. When swallowing is occurring, the tip 12d of the endoscope 12 also moves significantly, causing blurring of the entire image, making it difficult to detect image feature points 43c, as shown in test image 43b in the bottom row of Figure 9. If the third threshold is set to 5 in the example of Figure 9, the number of image feature points 43c in image processing target region 43g is 0, which is less than the third threshold, and therefore test image 43b is determined to be in the process of swallowing.

The fourth determination unit 44 performs voice determination as a fourth swallowing detection algorithm. The user interface 19 connected to the processor device 15 includes a microphone (not shown) that acquires voice, and the voice waveform acquired from the microphone is input to the computer 16 each time. The voice waveform is transmitted as an audio signal to the fourth determination unit 44 of the swallowing determination unit 40. The voice is usually acquired together with the test image.

The method of determining the voice will be described with reference to the flowchart of FIG. 10. First, a voice signal is input to the fourth determination unit 44 (step S101). The fourth determination unit 44 analyzes the input voice signal and first determines whether the voice is from the patient or from someone other than the patient (steps S102 and S103). If the voice signal is from someone other than the patient, it is recorded together with the examination time (examination time). If the voice signal is from the patient, it is determined whether the voice signal is a swallowing-related sound or a non-swallowing-related sound (step S104). Swallowing-related sounds include swallowing sounds and epiglottis opening and closing sounds associated with swallowing. Non-swallowing-related sounds include sounds such as coughing, choking, vocalization, and breathing sounds. If the voice signal is a swallowing-related sound, it is determined that swallowing is in progress (step S105). If the voice signal is not a swallowing-related sound, it is determined that non-swallowing is in progress (step S106).

Even if the first judgment unit 41, the second judgment unit 42, the third judgment unit 43, and the fifth judgment unit 45 judge the test image to be during swallowing, if the fourth judgment unit 44 judges the test image to be during non-swallowing based on the audio signal acquired at the time the test image was acquired, the test image is judged to be during non-swallowing. In other words, if the larynx or vocal cords move significantly as in swallowing due to coughing, choking, speaking, etc., the first judgment unit 41, the second judgment unit 42, the third judgment unit 43, and the fifth judgment unit 45 described below judge the test image to be during swallowing, but even if the test image judges the test image to be during swallowing, if the fourth judgment unit 44 acquires non-swallowing-related sounds, it is judged to be during non-swallowing.

A method of determination using a test image determined to be swallowing and an audio signal will be described with reference to the flowchart of FIG. 11. First, the first determination unit 41, the second determination unit 42, the third determination unit 43, or the fifth determination unit 45 described below determines that the test image is being swallowed (step S201). At the same time, an audio signal is input to the fourth determination unit 44 (step S202). Next, the fourth determination unit 44 determines whether the audio signal is a swallowing-related sound or a non-swallowing-related sound (step S203). If the audio signal is a swallowing-related sound, it is determined that swallowing is occurring (step S204). If the audio signal is not a swallowing-related sound, it is determined that non-swallowing is occurring (step S205). With the above configuration, when it is not possible to distinguish between a swallowing response and a non-swallowing response based on images alone, for example when there is significant movement of the glottis or epiglottis that is not swallowing, such as the closing of the glottis or the opening and closing of the epiglottis associated with coughing, the accuracy of determining whether swallowing or not is improved by using voice data to exclude these non-swallowing responses.

The fourth determination unit 44 may also associate the swallowing-related sounds and non-swallowing-related sounds with the test image at the timing when the swallowing-related sounds and non-swallowing-related sounds are produced. Furthermore, the timing when the swallowing-related sounds and non-swallowing-related sounds are produced may be differentiated as normal or abnormal (dysphagia), and associated with the test image. Specifically, the normal or abnormal differentiation related to the swallowing-related sounds and non-swallowing-related sounds is performed after analyzing the number of swallowing-related sounds and non-swallowing-related sounds, the intervals at which swallowing-related sounds and non-swallowing sounds are produced if they are produced multiple times, whether the epiglottis opening and closing sounds associated with swallowing are related to dysphagia, etc.

In addition, if the audio signal is a voice emitted by someone other than the patient, and it is determined to be a specific voice emitted by a doctor or the like (for example, a call such as "start the examination"), the specific voice may be associated with the examination image at the time the specific voice was emitted.

The fifth determination unit 45 is preferably equipped with a classifier 45a that, when an examination image is input, calculates the probability that swallowing is in progress and determines whether or not swallowing is in progress. The classifier 45a is a classifier generated using machine learning. It is preferable to use deep learning for the machine learning, and for example, it is preferable to use a multi-layer convolutional neural network. In addition to deep learning, machine learning includes decision trees, support vector machines, random forests, regression analysis, supervised learning, semi-unsupervised learning, unsupervised learning, reinforcement learning, deep reinforcement learning, learning using a neural network, generative adversarial networks, and the like.

It is preferable that the classifier 45a is a machine learning type that has previously learned images determined to be swallowing and non-swallowing. The classifier 45a may be a machine learning type that uses unsupervised learning or semi-unsupervised learning to automatically cluster images of swallowing and non-swallowing. Furthermore, in addition to images of swallowing and non-swallowing, the classifier 45a may be previously trained on swallowing-related sounds and non-swallowing-related sounds, and an audio signal may be input in addition to the test image during the test, and the probability of swallowing or non-swallowing may be output.

The test images input to the classifier 45a (test image 45b in the upper part of FIG. 12 and test image 45d in the lower part of FIG. 12) are processed by processing an area of at least 224 pixels vertically and horizontally from the center of the test image for at least one frame (image processing target area 45g). As shown in the example of FIG. 12, the classifier 45a outputs the probability of swallowing. If this probability is equal to or greater than a fourth threshold, it determines that the test image of the frame for which the probability was calculated is swallowing. The value of the fourth threshold can be set arbitrarily in the range of 0.001 to 0.999.

A specific example will be explained in detail. The top part of Fig. 12 is an example using test image 45b in which no swallowing movement is occurring. When test image 45b is input to classifier 45a, the corresponding swallowing probability 45c is output. In the example of Fig. 12, if the fourth threshold is set to 0.98, probability 45c is less than the fourth threshold, and test image 45b is determined to be not swallowing.

The bottom part of Figure 12 shows an example using test image 45d, in which a swallowing movement is occurring. When test image 45d is input to classifier 45a, the corresponding swallowing probability 45e is output. In the example of Figure 12, if the fourth threshold is set to 0.98, probability 45c is equal to or greater than the fourth threshold, and test image 45b is determined to be in the process of swallowing.

The test image input to the classifier 45a may be an image signal input from the imaging sensor, or may be a test image determined as being swallowed or not by the first, second, or third swallowing detection algorithm. The test image determined as being swallowed or not by the classifier 45a may be corrected by using the first, second, third, or fourth swallowing detection algorithm to determine whether or not swallowing is occurring, as output by the classifier 45a.

The timing of swallowing assessment is assumed to be performed in real time during the examination and the results are displayed on the examination screen, but it may be performed after the examination is completed. It may be performed automatically after the examination is completed and the results are recorded. It may be performed only when a user such as a doctor is instructed to assess only the necessary videos. Swallowing assessment may be performed when images are called up and displayed from a recording device 17 such as a PACS (Picture Archiving and Communication Systems), electronic medical record, or server. The computer 16 may be independent of the processor device 15, and may read and assess videos recorded on an external recording device such as a USB (Universal Serial Bus) memory. With the above configuration, swallowing assessment can be performed automatically from the examination images to assist the user in diagnosis.

The swallow count unit 47 determines how many times a swallowing action has occurred in a series of chronologically consecutive examination images (videos) that have been determined to be "swallowing" or "not swallowing" by one or more of the first determination unit 41, the second determination unit 42, the third determination unit 43, the fourth determination unit 44, or the fifth determination unit 45. Counting the number of swallows is a function for reducing the burden on users such as doctors when reviewing the examination images. By using the image analysis method of this embodiment, swallowing judgment is made on a huge number of images during or after a swallowing endoscopy examination. However, when a doctor makes a diagnosis, he or she observes one swallowing action acquired as a series of videos rather than the image during swallowing itself, and then diagnoses whether the swallowing action is normal or abnormal. For this reason, it is necessary to know at what timing one swallowing action occurred in the acquired video. A specific counting method will be described below.

A series of test images associated with a state of "swallowing" or "not swallowing" in any one or more of the first judgment unit 41, the second judgment unit 42, the third judgment unit 43, or the fifth judgment unit 45 are transmitted to the swallowing count unit 47. FIG. 13 shows a series of test images 47a that are determined to be "swallowing" or "not swallowing" and are transmitted to the swallowing count unit 47. The series of test images 47a are images acquired in chronological order, and are determined to be "swallowing" (indicated by "A" in FIG. 13) or "not swallowing" (indicated by "B" in FIG. 13), respectively. Hereinafter, a series of test images "swallowing" will be referred to as a "swallowing video," and a series of test images "not swallowing" will be referred to as a "not swallowing video."

The swallowing count unit 47 counts the number of swallows when the duration of the non-swallowing video exceeds a certain period from the time when the non-swallowing video appears after the swallowing video. In other words, when the continuous non-swallowing period, which is the time when the non-swallowing video was acquired, exceeds a certain period after the continuous swallowing period, which is the time when the swallowing video was acquired, the swallowing video immediately before the continuous non-swallowing period that exceeded the certain period is determined to be one swallowing action.

Or, if a continuous non-swallowing period exceeds a certain period after a continuous swallowing period, a combined video of swallowing during a continuous swallowing period before the continuous non-swallowing period that exceeded the certain period and a non-swallowing video not exceeding the certain period are determined to be a single swallowing action. Note that by not including videos that were once determined to be a single swallowing action in "videos of swallowing during a continuous swallowing period before the continuous non-swallowing period that exceeded the certain period and a non-swallowing video not exceeding the certain period," duplicate swallow counts are prevented.

A specific example of the swallow count method in the swallow count count unit 47 is shown in FIG. 14. The video during swallowing is designated as video A, and the video during non-swallowing is designated as video B. In addition, "swallowing" refers to a continuous period of swallowing, and "non-swallowing" refers to a continuous period of non-swallowing. The points of time lapse indicated by T from the start of the examination to the end of the examination shown in FIG. 14 will be described below in chronological order. T0 (start of examination) to T1 is the continuous period of non-swallowing during which video B (video during non-swallowing) was captured, T1 to T2 is the continuous period of swallowing during which video A (video during swallowing) was captured, and T2 to T3 is the continuous period of non-swallowing during which video B was captured. Here, as time passes from T2 to T3, a certain period of time Ts has passed since T2. Therefore, at the time point of T2+Ts, the examination image included in the time from T1 to T2 is determined to be one swallowing action.

Furthermore, the period from T3 to T4 is the continuous swallowing period during which video A was captured, the period from T4 to T5 is the continuous non-swallowing period during which video B was captured, the period from T5 to T6 is the continuous swallowing period during which video A was captured, and the period from T6 to T7 is the continuous non-swallowing period during which video B was captured. Here, while time passes from T4 to T5, the certain period Ts from T4 is not exceeded. Meanwhile, while time passes from T6 to T7, the certain period Ts from T6 is exceeded. In this case, at the time point of T6+Ts, the test image included in the video from T3 to T6 is determined to be one swallowing action.

In the specific example shown in FIG. 14, there are two swallowing actions. In this example, the timing of each swallowing action, for example, start times T1 and T3 and end times T2 and T6, are recorded in the result recording unit 34. The length of the fixed period Ts may be set by the user or automatically. With the above configuration, one swallowing action can be determined from the test images. Furthermore, when a "not swallowing" image is present in a "swallowing" image within an extremely short period of time from a series of test images, counting the number of swallowing actions excessively can be prevented, and a video display screen that can be generated that is easy to check efficiently after the test.

The test image in which the number of swallowing actions has been counted by the swallow count unit 47 is transmitted to the display control unit 32. The display control unit 32 generates a display screen 50 as shown in FIG. 15. The display screen 50 is displayed on the display 18.

As shown in FIG. 15, the display screen 50 displays the current test image 50a, and the seek bar 51 indicates the total playback time, while the position of the seek button 52 indicates the current playback time. The current playback time can be adjusted by operating the seek button 52 on the seek bar 51, the rewind playback button 53, and the fast forward playback button 54. The video being played can be paused by the pause button 55, and can be played by the play button 56. It is also preferable that the portion 57 of the test image determined to be one swallowing movement is calculated from the start and end times of the swallowing movement and displayed in a recognizable format. For example, as shown in FIG. 15, the portion 57 determined to be one swallowing movement is displayed on the seek bar 51. With the above configuration, the user can skip the video to the test image during swallowing, eliminating the need to search for the portion during swallowing in the entire video.

A series of steps from acquiring the test images to displaying the display screen will be described with reference to the flowchart shown in FIG. 16. The image acquisition unit 31 acquires the test images and transmits them to the swallowing determination unit 40 (step S301). Next, at least one of the first determination unit 41, the second determination unit 42, the third determination unit 43, the fourth determination unit 44, and the fifth determination unit 45 determines whether or not swallowing is occurring (step S302). The series of test images that have been determined to be "swallowing" or "not swallowing" are judged to have one swallowing movement by the swallowing count unit 47 (step S303). The test images that have been determined to have one swallowing movement are generated as a display screen by the display control unit 32 and displayed on the display 18 (step S304).

In this embodiment, the processor device 15 and the computer 16 are provided in the endoscope system 10, but the present invention is not limited to this, and other medical devices may be used. In addition, the endoscope 12 may be a rigid or flexible endoscope. In addition, part or all of the image acquisition unit 31 and/or the control unit 30 of the endoscope system 10 may be provided in a medical image processing device that communicates with the processor device 15 and cooperates with the endoscope system 10. For example, they may be provided in a diagnosis support device that acquires images captured by the endoscope 12 directly from the endoscope system 10 or indirectly from a PACS. In addition, part or all of the image acquisition unit 31 and/or the control unit 30 of the endoscope system 10 may be provided in a medical work support device that is connected via a network to various examination devices, such as a first examination device, a second examination device, ..., an Nth examination device, including the endoscope system 10.

In this embodiment, the hardware structure of the processing units that execute various processes, such as the control unit 30, image acquisition unit 31, display control unit 32, image input unit 33, result recording unit 34, and swallowing determination unit 40, is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), which is a processor whose circuit configuration can be changed after manufacture, such as an FPGA (Field Programmable Gate Array), and a dedicated electric circuit, which is a processor with a circuit configuration designed specifically to execute various processes.

A processing unit may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Multiple processing units may also be configured with one processor. As an example of configuring multiple processing units with one processor, first, as represented by a computer such as a client or server, one processor is configured with a combination of one or more CPUs and software, and this processor functions as multiple processing units. Second, as represented by a system on chip (SoC), there is a form in which a processor is used that realizes the functions of the entire system including multiple processing units with a single IC (Integrated Circuit) chip. In this way, the various processing units are configured using one or more of the above various processors as a hardware structure.

More specifically, the hardware structure of these various processors is an electric circuit (circuitry) that combines circuit elements such as semiconductor elements. The hardware structure of the memory unit is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD).

LIST OF SYMBOLS 10 ENDOSCOPIC SYSTEM 12 ENDOSCOPIC 12a INSERTION SECTION 12b OPERATION SECTION 12c BENDING SECTION 12d TIP SECTION 12e ANGLE KNOB 12h STILL IMAGE ACQUISITION INSTRUCTION SWITCH 12i ZOOM OPERATION SECTION 14 LIGHT SOURCE DEVICE 15 PROCESSOR 16 COMPUTER 17 RECORDING DEVICE 18 DISPLAY 19 USER INTERFACE 20 LIGHT SOURCE SECTION 22 LIGHT SOURCE CONTROL 30 CONTROL SECTION 31 IMAGE ACQUISITION SECTION 32 DISPLAY CONTROL SECTION 33 IMAGE INPUT SECTION 34 RESULT RECORDING SECTION 40 SWALLOWING DECISION SECTION 41 FIRST DETERMINATION SECTION 41a, 41b, 41d, 41e, 42a, 42c, 43a, 43b, 45b, 45d, 100 TEST IMAGE 41c, 41f EXAMPLE OF VARIATION 41g, 42g, 43g, 45g IMAGE PROCESSING SUBJECT REGION 42 SECOND DETERMINATION SECTION 42b, 42d Examples of image features in the upper part of Fig. 8 43 Third determination unit 43c Image feature points 44 Fourth determination unit 45 Fifth determination unit 45a Classifier 45c, 45e Probability 47 Swallowing count unit 47a Series of test images 50 Display screen 50a Current test image 51 Seek bar 52 Seek button 53 Rewind video playback button 54 Fast forward video playback button 55 Pause button 56 Play button 57 Locations determined as one swallowing action Es Esophagus Eg Epiglottis Ev Vallecula F Food Rg Glottic cleft Ps Piriform sinus Sp Soft palate To Tongue Tr Trachea

Claims (12)

1. An endoscope system that illuminates an object and captures an image of reflected light from the object,
A control processor is provided,
The control processor includes:
Obtaining an inspection image;
determining that swallowing is occurring when an amount of change between the test images successive in time series is equal to or greater than a first threshold, when an image feature amount relating to the test image is equal to or greater than a second threshold, or when swallowing-related sounds are acquired;
The control processor includes:
The test image is input to a classifier to calculate a probability that swallowing is occurring, and when the probability is equal to or greater than a fourth threshold, it is determined that swallowing is occurring;
An endoscope system, wherein the image feature related to the inspection image is an edge amount calculated from the inspection image. The endoscope system according to claim 1, wherein the amount of change is a value calculated using the absolute value of the simple pixel value difference between the inspection images that come before and after in a time series. The endoscope system according to claim 1 or 2, wherein the swallowing-related sound is an audio signal that is determined to be a sound emitted by a patient by analyzing an audio signal acquired by the control processor. The endoscope system according to claim 3, wherein the control processor determines the audio signal to be a swallowing-related sound or a non-swallowing-related sound, which is a cough, a chomp, a breathing sound, or a vocalization. The endoscope system according to claim 4, wherein the control processor determines that the test image indicates non-swallowing when the test image indicates swallowing and the non-swallowing-related sound is acquired. The control processor includes:
The endoscope system according to any one of claims 3 to 5, wherein the inspection image and the audio signal are input to a classifier, a probability that swallowing is in progress is calculated, and swallowing is determined to be in progress if the probability is equal to or greater than a fourth threshold value. The endoscope system according to claim 6 , wherein the classifier is trained using images determined to indicate swallowing or non-swallowing. The control processor includes:
A series of the test images determined to be during swallowing is a swallowing video, and a series of the test images determined to be during non-swallowing is a non-swallowing video,
An endoscopic system as described in any one of claims 1 to 7, wherein, when a continuous non-swallowing period, which is the time when the swallowing video was acquired, exceeds a certain period after a continuous swallowing period, which is the time when the non-swallowing video was acquired, the swallowing video acquired during the continuous swallowing period immediately before the continuous non-swallowing period that exceeds the certain period is determined to be one swallowing action, and the number of swallows is counted. The control processor includes:
The endoscopic system of claim 8, wherein a combined video of the swallowing motion acquired during the continuous non-swallowing period prior to the continuous non-swallowing period that exceeds the certain period and a video of the non-swallowing motion acquired during the continuous non-swallowing period not exceeding the certain period are determined to be one swallowing action, and the number of swallows is counted. The endoscope system according to claim 8 or 9 , wherein the certain period can be arbitrarily set. The control processor includes:
The endoscope system according to claim 8 or 9 , further comprising a display screen that enables recognition of a portion determined to be one swallowing movement among the examination images arranged in chronological order. 1. A method for operating an endoscope system including a control processor, illuminating an object, and capturing an image of reflected light from the object, comprising:
The control processor includes:
acquiring an inspection image;
determining that swallowing is occurring when an amount of change between the test images successive in time series is equal to or greater than a first threshold, when an image feature amount relating to the test image is equal to or greater than a second threshold, or when swallowing-related sounds are acquired;
The control processor includes:
The test image is input to a classifier to calculate a probability that swallowing is occurring, and when the probability is equal to or greater than a fourth threshold, it is determined that swallowing is occurring;
A method for operating an endoscope system, wherein the image feature related to the inspection image is an edge amount calculated from the inspection image.

Images