JP7812751B2 - Image processing method, image processing device, program, and recording medium - Google Patents

Description

This disclosure relates to a method, apparatus, program, and recording medium for processing optical coherence tomography (OCT) images.

In recent years, OCT has been attracting attention for generating images that represent the surface and internal morphology of a measured object (sample) using a light beam from a laser light source or other source. Because OCT is not invasive to living organisms like X-ray CT (Computed Tomography), it is expected to be particularly useful in the medical and biological fields. For example, in the field of ophthalmology, devices for generating images of the fundus and cornea have been put to practical use.

Various artifacts occur in OCT images. One of these is artifacts resulting from the birefringence of the sample. Birefringence is an optical property of materials, with a refractive index that depends on the polarization and propagation direction of light. A light beam passing through a birefringent sample is split into two beams depending on its polarization state. Artifacts resulting from birefringence (birefringence-related artifacts) occur not only in OCT images generated by OCT modalities capable of detecting polarized light, but also in standard OCT intensity images. Examples of OCT modalities capable of detecting polarized light (polarization-sensitive OCT) include those described in Patent Documents 1 to 4.

A known technique for removing birefringence artifacts is to separate the OCT signal into two polarization components, detect them, and then generate an OCT intensity signal by combining the intensities of the detected two polarization component signals. This technique is realized, for example, using a detection module (polarization separation detection function) that combines a polarizing beam splitter and two photodetectors, as described in Non-Patent Document 1.

Patent No. 4344829 specification Patent No. 6256879 specification Patent No. 6346410 specification Patent No. 6579718 specification

Shuichi Makita, Toshihiro Mino, Tastuo Yamaguchi, Mashiro Miura, Shinnosuke Azuma, and Yoshiaki Yasuno, “Clinical "prototype of pigment and flow imaging optical coherence tomography for posterior eye investigation", Biomedical Optics Express, Vol. 9, Issue 9, pp. 4372-4389 (2018)

One of the objectives of this disclosure is to provide a new technology for generating images with reduced birefringence artifacts from images acquired using an OCT modality that does not have polarization separation detection capabilities.

One aspect of the embodiment is a method for processing optical coherence tomography (OCT) images, comprising: preparing a machine learning model constructed by machine learning using training data including a set of pairs of a first image containing birefringence information of an object and a second image not containing birefringence information of the object; the machine learning model constructed to receive an OCT image as input and output a denoised image in which birefringence-induced artifacts in the OCT image are reduced; acquiring an OCT image of a sample; inputting the OCT image of the sample to the machine learning model; and storing or providing the denoised image output from the machine learning model based on the input OCT image of the sample.

Another aspect of the embodiment is an apparatus for processing optical coherence tomography (OCT) images, comprising: an image acquisition unit that acquires OCT images of a sample; and a processing unit that performs processing using a machine learning model constructed by machine learning using training data including a set of pairs of a first image containing birefringence information of an object and a second image not containing birefringence information of the object, the machine learning model being constructed to receive an OCT image as input and output a denoised image in which birefringence-induced artifacts in the OCT image are reduced. The processing unit inputs the OCT image of the sample acquired by the image acquisition unit to the machine learning model, and stores or provides the denoised image output from the machine learning model based on the input OCT image of the sample.

Yet another aspect of the embodiment is a program for processing optical coherence tomography (OCT) images, which causes a computer to execute the following steps: acquiring an OCT image of a sample; inputting the OCT image of the sample into a machine learning model constructed by machine learning using training data including a set of pairs of a first image containing birefringence information of an object and a second image not containing birefringence information of the object, the machine learning model being constructed to receive an OCT image as input and output a denoised image in which birefringence-related artifacts in the OCT image are reduced; and storing or providing the denoised image output from the machine learning model based on the input of the OCT image of the sample.

Yet another aspect of the embodiment is a computer-readable non-transitory recording medium having a program recorded thereon for processing optical coherence tomography (OCT) images, the program comprising:
The program causes a computer to perform the following steps: acquiring an OCT image of a sample; inputting the OCT image of the sample into a machine learning model constructed by machine learning using training data including a set of pairs of a first image containing birefringence information of an object and a second image not containing birefringence information of the object, the machine learning model being constructed to receive an OCT image as input and output a denoised image in which birefringence-induced artifacts in the OCT image are reduced; and storing or providing the denoised image output from the machine learning model based on the input of the OCT image of the sample.

According to the embodiment, it is possible to provide a new technology for generating images with reduced birefringence artifacts from images acquired using an OCT modality that does not have polarization separation detection functionality.

1 is a schematic diagram illustrating an example of a configuration of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 1 is a schematic diagram illustrating an example of a configuration of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 1 is a schematic diagram illustrating an example of a configuration of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 1 is a schematic diagram illustrating an example of a configuration of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 1 is a schematic diagram illustrating an example of a configuration of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 10 is a flowchart illustrating an example of an operation of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 10 is a flowchart illustrating an example of an operation of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 10 is a flowchart illustrating an example of an operation of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment. 10 is a flowchart illustrating an example of an operation of an image processing apparatus (ophthalmologic apparatus) according to an exemplary aspect of an embodiment.

Image processing methods, image processing devices, programs, and recording media according to some exemplary aspects of the embodiments will be described in detail with reference to the drawings. While the present disclosure describes some application examples, particularly in the field of ophthalmology, the embodiments are not limited thereto and can be applied to any field in which OCT can be used and in which birefringence-related artifacts occur or are likely to occur.

Any matter described in the documents cited in this disclosure or any matter relating to other publicly known technologies may be combined with the embodiments. Furthermore, in this disclosure, unless otherwise specified, no distinction is made between "image data" and "images," which are visualized information based on such data.

The image processing device according to the embodiment has a function of acquiring an OCT image of a sample. Any configuration may be employed to achieve this function. In some exemplary aspects, the image processing device is configured to acquire an OCT image of the sample by applying an OCT scan to the sample to collect data and then process the collected data to construct the OCT image. The type of OCT (OCT modality) used for this OCT scan may be any, and may be, for example, Fourier-domain OCT (swept-source OCT or spectral-domain OCT) or time-domain OCT.

In some exemplary embodiments, the image processing device is configured to externally acquire OCT images of the sample. Such an image processing device may include, for example, a communications device for receiving OCT images stored in a storage device via a communications line, a communications device for receiving OCT images acquired by the OCT device via a communications line, or a data reader for reading OCT images recorded on a recording medium.

Furthermore, the image processing device according to the embodiment has the function of generating an image in which birefringence artifacts have been reduced from the acquired OCT image. An image in which birefringence artifacts have been reduced by the processing according to the present disclosure is called a denoised image.

The image processing function for generating a denoised image is achieved using a pre-created machine learning model. The machine learning system according to the embodiment is constructed by machine learning using training data (teacher data, learning data) including a set of pairs of a first image including birefringence information of an object and a second image not including birefringence information of the object, and functions to receive an OCT image as input and output a denoised image in which birefringence-related artifacts in the OCT image have been reduced. In some aspects of the embodiment, the first image including birefringence information of the object is an image including information derived from the birefringence of the object (e.g., birefringence-related artifacts), and the second image not including birefringence information of the object is an image not including information derived from the birefringence of the object.

The denoised image generated by the image processing device according to the embodiment is stored and/or provided. In some exemplary aspects, the denoised image is saved in a storage device located inside or outside the image processing device. The storage device inside the image processing device may be, for example, a hard disk drive, a solid-state drive, etc. The storage device outside the image processing device may be, for example, a database system, a data management system, etc. An example of such a system is a Picture Archiving and Communication System (PACS), a typical image management system in the medical field.

In some exemplary embodiments, the denoised image generated by the image processing device is provided to a computer located inside or outside the image processing device. This computer has a function of processing the denoised image to generate new information. Examples of the functions of this computer include a function of analyzing the denoised image to generate analytical information, a function of processing the denoised image to generate visualized information, a function of combining the denoised image with other information to generate new information, a function of generating training data for machine learning based on the denoised image, a function of performing machine learning using training data including the denoised image and/or training data including information based on the denoised image, a function of performing inference using the denoised image and/or information based thereon, and a function that at least partially combines two or more of these functions.

At least a portion of the functionality of the elements disclosed herein is implemented using circuitry or processing circuitry. The circuitry or processing circuitry may be a general-purpose processor, a special-purpose processor, an integrated circuit, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), an Application Specific Integrated Circuit (ASIC), a programmable logic device (e.g., a Simple Programmable Logic Device (SPLD), a Complex Programmable Logic Device (CPLD), a Field Programmable Gate Array (FPGA)), or a combination of these devices configured and/or programmed to perform at least some of the disclosed functions. Array), conventional circuitry, and any combination thereof. A processor is considered to be processing circuitry or circuitry including transistors and/or other circuitry. In this disclosure, circuitry, unit, means, or similar terms refers to hardware that performs at least some of the disclosed functions or that is programmed to perform at least some of the disclosed functions. The hardware may be the hardware described in this disclosure or may be known hardware that is programmed and/or configured to perform at least some of the functions described in this disclosure. In the case of a processor, where the hardware can be considered to be a type of circuitry, circuitry, unit, means, or similar terms refers to a combination of hardware and software, and the software is used to configure the hardware and/or processor.

<Configuration of image processing device (ophthalmic device)>
1 is an exemplary embodiment of an image processing device according to the present invention, and is configured to acquire an OCT image of a sample by applying an OCT scan to the sample to collect data, and to construct the OCT image by processing the collected data. In this embodiment, the sample is a living human eye (examined eye).

The ophthalmic device 1 is a multifunction device that combines an OCT device and a fundus camera, and has the function of applying OCT scanning to the subject's eye E (fundus Ef and anterior segment Ea) and the function of taking digital photographs of the subject's eye E (fundus Ef and anterior segment Ea).

The ophthalmic device 1 includes a fundus camera unit 2, an OCT unit 100, and an arithmetic and control unit 200. The fundus camera unit 2 is provided with various elements (optical systems, mechanisms, etc.) for acquiring a front image of the subject's eye. The OCT unit 100 is provided with some of the various elements (optical systems, mechanisms, etc.) for OCT scanning. Some other elements for OCT scanning are provided in the fundus camera unit 2. The arithmetic and control unit 200 includes one or more processors and one or more storage devices configured to perform various processes (arithmetic operations, control, etc.). The ophthalmic device 1 further includes a chin rest, a forehead rest, etc.

The attachment 400 includes a lens group for switching the area where the OCT scan is applied between the posterior segment (fundus Ef) and the anterior segment Ea. The attachment 400 may be, for example, the optical unit disclosed in Japanese Patent Application Laid-Open No. 2015-160103. The attachment 400 can be placed between the objective lens 22 and the subject's eye E. The attachment 400 is retracted from the optical path to apply an OCT scan to the fundus Ef, and is placed in the optical path to apply an OCT scan to the anterior segment Ea. Conversely, the attachment 400 may be retracted from the optical path to apply an OCT scan to the anterior segment Ea, and is placed in the optical path to apply an OCT scan to the fundus Ef. The attachment 400 is moved manually or automatically. The element for switching the area where the OCT scan is applied is not limited to such an attachment, and may, for example, be configured to include one or more lenses that are movable along the optical path.

<Fundus camera unit 2>
The fundus camera unit 2 is provided with elements (optical system, mechanism, etc.) for taking digital photographs of the subject's eye E. The acquired digital photographs are front images such as observation images and photographed images. The observation images are acquired by video capture using near-infrared light, for example, and are used for alignment, focusing, tracking, etc. The photographed images are still images captured using flash light in the visible or infrared range, for example, and are used for diagnosis, analysis, etc.

The fundus camera unit 2 includes an illumination optical system 10 and an imaging optical system 30. The illumination optical system 10 irradiates the subject's eye E with illumination light. The imaging optical system 30 detects return light of the illumination light from the subject's eye E. The measurement light from the OCT unit 100 is guided to the subject's eye E via an optical path within the fundus camera unit 2, and the return light is guided to the OCT unit 100 via the same optical path.

Light (observation illumination light) output from the observation light source 11 of the illumination optical system 10 is reflected by the concave mirror 12, passes through the condenser lens 13, and passes through the visible light cut filter 14 to become near-infrared light. The observation illumination light is then focused near the imaging light source 15, reflected by the mirror 16, and passes through the relay lens system 17, relay lens 18, aperture 19, and relay lens system 20. The observation illumination light is then reflected by the peripheral portion of the aperture mirror 21 (the area surrounding the aperture), passes through the dichroic mirror 46, and is refracted by the objective lens 22 (and the optical elements within the attachment 400) to illuminate the subject's eye E. Return light of the observation illumination light from the subject's eye E is refracted by (the optical elements within the attachment 400 and) the objective lens 22, transmitted through the dichroic mirror 46, passed through the hole formed in the central region of the aperture mirror 21, transmitted through the dichroic mirror 55, passed through the photographing focusing lens 31, and reflected by the mirror 32. This return light then passes through the half mirror 33A, is reflected by the dichroic mirror 33, and is imaged by the imaging lens 34 on the light-receiving surface of the image sensor 35. The image sensor 35 detects the return light at a predetermined frame rate. The focus (focal position) of the photographing optical system 30 is typically adjusted to match the fundus Ef or the anterior segment Ea.

Light output from the imaging light source 15 (imaging illumination light) travels the same path as the observation illumination light and is irradiated onto the subject's eye E. Return light of the imaging illumination light from the subject's eye E travels the same path as the return light of the observation illumination light and is guided to the dichroic mirror 33, passes through the dichroic mirror 33, is reflected by the mirror 36, and is imaged by the imaging lens 37 on the light-receiving surface of the image sensor 38.

The liquid crystal display (LCD) 39 displays a fixation target (fixation target image). A portion of the light beam output from the LCD 39 is reflected by the half mirror 33A, then reflected by the mirror 32, passes through the photographing focusing lens 31 and dichroic mirror 55, and passes through the hole in the aperture mirror 21. The light beam that passes through the hole in the aperture mirror 21 passes through the dichroic mirror 46, is refracted by the objective lens 22, and is projected onto the fundus Ef. By changing the display position of the fixation target image, the fixation position of the subject's eye E, determined by the fixation target, can be changed. This makes it possible to guide the subject's eye E's gaze in the desired direction.

The alignment optical system 50 generates an alignment index used to align the optical system with the subject's eye E. Alignment light output from a light-emitting diode (LED) 51 passes through an aperture 52, an aperture 53, and a relay lens 54, is reflected by a dichroic mirror 55, passes through the hole in the aperture mirror 21, transmits through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. The return light of the alignment light from the subject's eye E is guided to the image sensor 35 along the same path as the return light of the observation illumination light. Manual alignment or automatic alignment can be performed based on the received light image (alignment index image).

The alignment method is not limited to the method using alignment indicators. For example, as described in JP 2013-248376 A, some exemplary embodiments of ophthalmic devices may be configured to photograph the anterior segment of the eye from different directions to obtain two or more anterior segment images, analyze these anterior segment images to determine the three-dimensional position of the subject's eye, and move the optical system based on this three-dimensional position (stereo alignment).

The focusing optical system 60 generates a split index used for focus adjustment of the subject's eye E. The focusing optical system 60 moves along the optical path (illumination optical path) of the illumination optical system 10 in conjunction with the movement of the imaging focusing lens 31 along the optical path (imaging optical path) of the imaging optical system 30. To perform focus adjustment, a reflecting rod 67 is inserted into the illumination optical path and positioned at an angle. The focusing light output from the LED 61 passes through the relay lens 62, is split into two beams by the split index plate 63, passes through the two-hole diaphragm 64, is reflected by the mirror 65, and is first imaged and reflected on the reflective surface of the reflecting rod 67 by the condenser lens 66. The focusing light then passes through the relay lens 20, is reflected by the aperture mirror 21, passes through the dichroic mirror 46, and is projected onto the subject's eye E via the objective lens 22. The returning light of the focusing light from the subject's eye E is guided to the image sensor 35 along the same path as the returning light of the alignment light. Manual and autofocusing can be performed based on the received light image (split index image).

Diopter correction lenses 70 and 71 are selectively inserted in the photographic optical path between the aperture mirror 21 and the dichroic mirror 55. Diopter correction lens 70 is a plus lens (convex lens) for correcting severe hyperopia. Diopter correction lens 71 is a minus lens (concave lens) for correcting severe myopia.

The dichroic mirror 46 couples the OCT optical path (measurement arm) to the digital photography optical path (illumination optical path and imaging optical path). The dichroic mirror 46 reflects light in the wavelength band for OCT scanning and transmits light in the wavelength band for digital photography. The measurement arm is equipped with, in order from the OCT unit 100 side, a collimator lens unit 40, a retroreflector 41, a dispersion compensation member 42, an OCT focusing lens 43, an optical scanner 44, and a relay lens 45.

The retroreflector 41 is movable in the directions indicated by the arrows in Figure 1 (the incident and exit directions of the measurement light LS). This changes the length of the measurement arm. Changing the measurement arm length is used, for example, to correct the optical path length according to the axial length of the eye, the shape of the cornea or fundus, or to adjust the interference state.

The dispersion compensation element 42, together with the dispersion compensation element 113 (described below) arranged in the reference arm, acts to match the dispersion characteristics of the measurement light LS with the dispersion characteristics of the reference light LR.

The OCT focusing lens 43 is movable in the direction indicated by the arrow in Figure 1 (the optical axis of the measurement arm) to adjust the focus of the measurement arm. This changes the focus state (focal position, focal length) of the measurement arm. The ophthalmic device 1 may be capable of coordinated control of the movement of the imaging focusing lens 31, the movement of the focus optical system 60, and the movement of the OCT focusing lens 43.

The optical scanner 44 is positioned substantially optically conjugate with the pupil of the subject's eye E. The optical scanner 44 is a deflector for changing the direction of travel (propagation direction) of the measurement light LS guided by the measurement arm. The optical scanner 44 is, for example, a two-dimensional deflector including a deflector for scanning in the x direction (x-scanner) and a deflector for scanning in the y direction (y-scanner). The type of deflector may be any type, such as a galvanometer scanner.

When the attachment 400 is retracted from the measurement arm for posterior segment OCT, the optical scanner 44 is positioned substantially optically conjugate with the pupil of the subject's eye E, and when the attachment 400 is inserted into the measurement arm for anterior segment OCT, the optical scanner 44 is positioned substantially optically conjugate with a position near the anterior segment Ea (e.g., a position between the anterior segment Ea and the attachment 400).

<OCT unit 100>
The exemplary OCT unit 100 shown in FIG. 2 is provided with optical systems and mechanisms for applying swept-source OCT. The optical system includes an interference optical system. The interference optical system splits light from a wavelength-tunable light source (swept-wavelength light source) into measurement light and reference light, superimposes the measurement light returned from the subject's eye E with the reference light that has passed through the reference light path, and generates interference light, which is then detected. An electrical signal (detection signal) generated by interference light detection includes a signal (interference signal) representing the spectrum of the interference light, and is sent to the arithmetic and control unit 200 (image construction unit 220).

The light source unit 101 includes, for example, a near-infrared tunable laser that changes the wavelength of the emitted light at high speed. Light L0 output from the light source unit 101 is guided by optical fiber 102 to polarization controller 103, where its polarization state is adjusted, and then guided by optical fiber 104 to fiber coupler 105, where it is split into measurement light LS and reference light LR. The optical path of the measurement light LS is called the measurement arm, and the optical path of the reference light LR is called the reference arm.

The reference light LR is guided by optical fiber 110 to collimator 111, where it is converted into a parallel beam, and then guided to retroreflector 114 via optical path length correction element 112 and dispersion compensation element 113. The optical path length correction element 112 is an optical element that matches the optical path length of the reference light LR with that of the measurement light LS. The dispersion compensation element 113, together with the dispersion compensation element 42 arranged in the measurement arm, acts to match the dispersion characteristics between the reference light LR and the measurement light LS. The retroreflector 114 is movable along the optical path of the reference light LR incident thereon, thereby changing the length of the reference arm. Changing the reference arm length is used, for example, to correct the optical path length according to the axial length of the eye, the shape of the cornea or fundus, and adjust the interference state.

The reference light LR that passes through the retroreflector 114 passes through the dispersion compensation element 113 and the optical path length correction element 112, is converted from a parallel beam to a focused beam by the collimator 116, and enters the optical fiber 117. The reference light LR that enters the optical fiber 117 is guided to the polarization controller 118, where its polarization state is adjusted. The polarization controller 118 is used, for example, to optimize the interference intensity between the measurement light LS and the reference light LR. The reference light LR that passes through the polarization controller 118 is guided through the optical fiber 119 to the attenuator 120, where its light amount is adjusted, and is then guided through the optical fiber 121 to the fiber coupler 122.

Meanwhile, the measurement light LS generated by the fiber coupler 105 is guided by the optical fiber 127 to the collimator lens unit 40, where it is converted into a parallel beam, and then passes through the retroreflector 41, dispersion compensation member 42, OCT focusing lens 43, optical scanner 44, relay lens 45, dichroic mirror 46, and objective lens 22 (and attachment 400) before being projected onto the subject's eye E. The measurement light LS incident on the subject's eye E is scattered and reflected at various depth positions in the subject's eye E. Return light (backscattered light, reflected light, etc.) of the measurement light LS from the subject's eye E travels the same path as the outward path in the opposite direction and is guided to the fiber coupler 105, and then guided to the fiber coupler 122 via the optical fiber 128.

The fiber coupler 122 generates interference light by superimposing the measurement light LS (return light from the test eye E) from the optical fiber 128 and the reference light LR from the optical fiber 121. The fiber coupler 122 splits the generated interference light into two beams at a predetermined split ratio (e.g., 1:1) to generate a pair of interference light LC. The pair of interference light LC is guided to the detector 125 via optical fibers 123 and 124, respectively.

The detector 125 includes, for example, a balanced photodiode. The balanced photodiode includes a pair of photodetectors that respectively detect a pair of interference light LC, and outputs the difference between a pair of electrical signals generated by the pair of photodetectors. The output differential signal (detection signal) is sent to the data acquisition system (DAQ, DAS) 130.

The data collection system 130 is supplied with a clock KC from the light source unit 101. The clock KC is generated in the light source unit 101 in synchronization with the output timing of each wavelength swept within a predetermined wavelength range by the tunable light source. The light source unit 101, for example, branches light L0 of each output wavelength to generate two branched lights, applies an optical delay to one of the branched lights, combines the delayed branched light with the other branched light, detects the resulting combined light, and generates a clock KC based on the detection result. The data collection system 130 samples the detection signal input from the detector 125 based on the clock KC input from the light source unit 101. The sampling result is sent to the arithmetic and control unit 200.

In this embodiment, both an element for changing the measurement arm length (such as a retroreflector 41) and an element for changing the reference arm length (such as a retroreflector 114 or a reference mirror) are provided, but in some exemplary embodiments, only one of these two elements may be provided.

In this embodiment, an element (polarization controller 118) is provided for changing the polarization state of the reference light LR. In some exemplary embodiments, an element (polarization controller) for changing the polarization state of the measurement light LS may be provided instead of an element for changing the polarization state of the reference light LR. In some exemplary embodiments, both an element for changing the polarization state of the reference light LR and an element for changing the polarization state of the measurement light LS may be provided.

Swept-source OCT, as used in the OCT unit 100 of Figure 2, is a technique that splits light from a tunable light source into measurement light and reference light, superimposes the measurement light returned from the test object with the reference light to generate interference light, detects this interference light with a photodetector, and constructs an image by applying Fourier transform or other techniques to the detection data collected in response to wavelength sweeps and measurement light scans. In contrast, spectral-domain OCT is a technique that splits light from a low-coherence light source (broadband light source) into measurement light and reference light, superimposes the measurement light returned from the test object with the reference light to generate interference light, detects the spectral distribution of this interference light with a spectroscope, and constructs an image by applying Fourier transform or other techniques to the detected spectral distribution. Simply put, swept-source OCT is an OCT technique that acquires the spectral distribution of interference light in a time-division manner, while spectral-domain OCT is an OCT technique that acquires the spectral distribution of interference light in a space-division manner. It will be apparent to those skilled in the art that the OCT techniques applicable to the embodiments are not limited to swept-source OCT.

<Control system/processing system>
3 shows an example of the configuration of the control system and processing system of the ophthalmic apparatus 1. The control unit 210, the image construction unit 220, and the data processing unit 230 are provided in, for example, the arithmetic control unit 200. Although not shown, the ophthalmic apparatus 1 may also include a communication device and a drive device (reader/writer).

<Control Unit 210>
The control unit 210 performs various types of control. The control unit 210 includes a main control unit 211 and a storage unit 212. The main control unit 211 includes a processor and controls the elements of the ophthalmologic apparatus 1 (the elements shown in FIGS. 1 to 5). The main control unit 211 is realized by cooperation between hardware including the processor and control software. The storage unit 212 includes a storage device such as a hard disk drive or a solid state drive, and stores data.

The imaging focusing driver 31A moves the imaging focusing lens 31 provided in the imaging optical path and the focus optical system 60 provided in the illumination optical path under the control of the main controller 211. The retroreflector (RR) driver 41A moves the retroreflector 41 provided in the measurement arm under the control of the main controller 211. The OCT focusing driver 43A moves the OCT focusing lens 43 provided in the measurement arm under the control of the main controller 211. The retroreflector (RR) driver 114A moves the retroreflector 114 provided in the reference arm under the control of the main controller 211. The movement mechanism 150 moves the optical system of the ophthalmic device 1 three-dimensionally (in the x, y, and z directions). The insertion/removal mechanism 400A inserts and removes the attachment 400 into and from the optical path.

<Image Construction Unit 220>
The image construction unit 220 constructs OCT image data of the subject's eye E based on signals (sampling data) input from the data collection system 130. The constructed OCT image data is one or more A-scan image data, for example, B-scan image data (two-dimensional cross-sectional image data) made up of a plurality of A-scan image data. The image construction unit 220 is realized by cooperation between hardware including a processor and image construction software.

To construct OCT image data from the sampling data, the image forming unit 220 performs signal processing on the spectral distribution based on the sampling data for each A-line to generate a reflection intensity profile for each A-line (A-line profile), similar to conventional swept-source OCT, and then images each A-line profile to generate multiple A-scan image data sets, which are then arranged according to the scan pattern (arrangement of multiple scan points). The signal processing for generating the A-line profile includes noise reduction, filtering, fast Fourier transform (FFT), and the like. When using other OCT techniques, known OCT image data construction processes appropriate for that technique are performed.

The image construction unit 220 may be configured to construct three-dimensional image data that represents a three-dimensional region (volume) of the subject's eye E. Three-dimensional image data is image data in which the positions of pixels are defined by a three-dimensional coordinate system, and examples of such data include stack data and volume data. Stack data is image data obtained by arranging multiple cross-sectional images obtained along multiple scan lines in accordance with the positional relationships of these scan lines. Volume data is image data in which pixels are three-dimensionally arranged voxels, constructed, for example, by applying interpolation processing or voxelization processing to stack data, and is also called voxel data.

The image constructor 220 can create new OCT image data from the OCT image data constructed in this manner. In some exemplary embodiments, the image constructor 220 can apply rendering to the three-dimensional image data. Examples of rendering include volume rendering, surface rendering, maximum intensity projection (MIP), minimum intensity projection (MinIP), and multiplanar reconstruction (MPR).

In some exemplary embodiments, the image constructor 220 may be configured to construct an OCT en face image from the three-dimensional image data. For example, the image constructor 220 may construct projection data by projecting the three-dimensional image data in the z direction (A-line direction, depth direction). The image constructor 220 may also construct projection data from partial data (e.g., slabs) of the three-dimensional image data. This partial data may be specified automatically using, for example, image segmentation (also simply referred to as segmentation), or may be manually specified by the user. This segmentation method may be any method and may include, for example, image processing such as edge detection and/or segmentation using machine learning. This segmentation may be performed, for example, by the image constructor 220 or the data processor 230.

The ophthalmologic device 1 may be capable of performing OCT motion contrast imaging. OCT motion contrast imaging is an imaging technique that extracts the movement of fluids and other substances present within the eye (see, for example, JP 2015-515894 A). OCT motion contrast imaging is used, for example, in OCT angiography (OCTA) to visualize blood vessels.

<Data Processing Unit 230>
The data processing unit 230 is configured to apply specific data processing to the image of the subject's eye E. The data processing unit 230 is realized, for example, by cooperation between hardware including a processor and data processing software.

The data processing unit 230 in this embodiment has the function of executing processing using models constructed using machine learning (machine learning models, mathematical models, inference models).

The training data used in this machine learning includes a set of pairs of a first image containing birefringence information of an object and a second image containing no birefringence information of the object. In other words, the training data used in this machine learning includes multiple pairs of OCT images, one containing birefringence information and the other not containing birefringence information, acquired of the same object.

In this disclosure, a pair of a first image and a second image acquired of a single object is referred to as an image pair. Furthermore, a set of image pairs included in the training data is referred to as an image pair set.

In this embodiment, the object to which OCT scanning is applied to collect images included in the training data may be a living human eye, or more specifically, a predetermined structure of a living human eye (e.g., an ocular region, an ocular tissue, an intraocular prosthesis, etc.). The ocular region or ocular tissue may be any region or tissue of the eye, such as the fundus (posterior segment), anterior segment, eyelid, optic nerve head, lamina cribrosa, macula, retina, retinal sub-tissues (e.g., internal limiting membrane, nerve fiber layer, ganglion cell layer, inner plexiform layer, inner nuclear layer, outer plexiform layer, outer nuclear layer, external limiting membrane, photoreceptor layer, retinal pigment epithelium layer), Bruch's membrane, choroid, choroid-scleral boundary (CSI), sclera, vitreous body, cornea, corneal sub-tissues (e.g., corneal epithelium, Bowman's membrane, lamina propria, Dua's layer, Descemet's membrane, corneal endothelium), conjunctiva, Schlemm's canal, trabecular meshwork, iris, lens, Zonule of Zinn, and ciliary body. An intraocular prosthesis can be any object implanted in the eye, examples of which include an intraocular lens (IOL), an intraocular contact lens (ICL), and a minimally invasive glaucoma surgery (MIGS) device.

The image pair sets included in the training data may be acquired from a single object, but in some exemplary embodiments, image pair sets acquired from multiple objects may be used. For example, the image pair sets may include one or more OCT image pairs acquired from a first object, one or more OCT image pairs acquired from a second object, ..., one or more OCT image pairs acquired from a Kth object (K is an integer greater than or equal to 2).

In some exemplary embodiments, the image pair set may be a set consisting only of images (unprocessed images) acquired from an object by OCT scanning. In addition, in some exemplary embodiments, the image pair set may also include images (processed images) obtained by processing the unprocessed images.

Some examples of how to create training data will be provided below.

The machine learning model of this embodiment is configured to receive an OCT image obtained by applying an OCT scan to a sample (subject's eye E) as input, and to output a denoised image in which birefringence-induced artifacts in the OCT image are reduced.

Some examples of the configuration of the data processing unit 230 in this embodiment and some examples of the processing performed by the data processing unit 230 will be described with reference to Figures 4 and 5.

The data processing unit 230A shown in FIG. 4 is one example configuration of the data processing unit 230 in FIG. 3. The data processing unit 230A includes a denoised image generation unit 231. The denoised image generation unit 231 is configured to generate a denoised image based on an OCT image of the subject's eye E, in which birefringence-induced artifacts in the OCT image have been reduced.

The denoised image generation unit 231 generates a denoised image using an inference model 2311. The inference model 2311 includes a neural network 2312 trained by the aforementioned machine learning to receive an OCT image of the subject's eye E as input and output a denoised image.

The inference model 2311 may be located inside the denoised image generation unit 231 (data processing unit 230) as shown in FIG. 4, or may be located in a location in the ophthalmic apparatus 1 other than the data processing unit 230 (for example, in the memory unit 212), or may be located outside the ophthalmic apparatus 1. As an example of the latter, the inference model 2311 may be located inside a computer or storage device accessible by the ophthalmic apparatus 1.

The device that constructs the inference model 2311 (inference model construction device) may be provided in the ophthalmic device 1, in a peripheral device (such as a computer) of the ophthalmic device 1, or may be another computer.

The model construction unit 300 shown in Figure 5 is an example of an inference model construction device and includes a learning processing unit 310 and a neural network 320.

Neural network 320 typically includes a convolutional neural network (CNN). Reference numeral 330 in Figure 5 shows an example of the structure of this convolutional neural network.

An image is input to the input layer of the convolutional neural network 330. Behind the input layer are multiple pairs of convolutional layers and pooling layers. In the example shown in Figure 5, there are three pairs of convolutional layers and pooling layers, but the number of pairs can be any number.

The convolutional layer performs convolution operations to extract features (such as contours) from an image. A convolution operation is a multiplication and accumulation operation of a filter function (weighting coefficients, filter kernel) of the same dimension as the input image on the input image. The convolutional layer applies the convolution operation to multiple portions of the input image. More specifically, the convolutional layer multiplies the value of each pixel in the partial image to which the filter function has been applied by the value (weight) of the filter function corresponding to that pixel to calculate the product, and then calculates the sum of the products across the multiple pixels in this partial image. This resulting sum-of-products value is assigned to the corresponding pixel in the output image. By performing the multiplication and accumulation operation while shifting the location (partial image) to which the filter function is applied, the convolution operation result for the entire input image is obtained. This convolution operation uses multiple weighting coefficients to generate multiple images in which various features have been extracted. In other words, it generates multiple filtered images, such as smoothed images and edge images. The multiple images generated by the convolutional layer are called feature maps.

The pooling layer compresses (e.g., thins out) the feature map generated by the previous convolutional layer. More specifically, the pooling layer calculates statistical values for a specified number of neighboring pixels of a pixel of interest in the feature map at specified pixel intervals, and outputs an image with smaller dimensions than the input feature map. The statistical value applied to the pooling operation is, for example, the maximum value (max pooling) or the average value (average pooling). The pixel interval applied to the pooling operation is called the stride.

Convolutional neural networks can extract many features from input images by processing them using multiple pairs of convolutional and pooling layers.

A fully connected layer is provided after the final pair of convolutional and pooling layers. In the example shown in Figure 5, two fully connected layers are provided, but any number of fully connected layers may be used. In the fully connected layer, features compressed by a combination of convolution and pooling are used to perform processes such as image classification, image segmentation, and regression. An output layer is provided after the final fully connected layer to provide output results.

In some exemplary embodiments, the convolutional neural network may not include a fully connected layer (e.g., a fully convolutional network (FCN)), and/or may include a support vector machine, a recurrent neural network (RNN), or the like. Furthermore, the machine learning performed on the neural network 320 may include transfer learning. That is, the neural network 320 may include a neural network that has already been trained using other training data (training images) and whose parameters have been adjusted. Furthermore, the model construction unit 300 (learning processing unit 310) may be configured to be able to apply fine tuning to the trained neural network (neural network 320). The neural network 320 may be constructed using a known open-source neural network architecture.

The learning processing unit 310 applies machine learning using training data to the neural network 320. If the neural network 320 includes a convolutional neural network, the parameters adjusted by the learning processing unit 310 include, for example, the filter coefficients of the convolutional layer and the connection weights and offsets of the fully connected layer.

The types of images included in the training data are not limited to OCT images. For example, the training data may include images acquired by ophthalmic modalities other than OCT (such as fundus cameras, slit lamp microscopes, SLOs, and surgical microscopes), images acquired by imaging diagnostic modalities of any medical specialty (such as ultrasound diagnostic devices, X-ray diagnostic devices, X-ray CT devices, and magnetic resonance imaging (MRI) devices), images created by processing actual eye images (processed image data), computer-generated images, simulated images, and any other images. Furthermore, the number of data items included in the training data may be increased using techniques such as data augmentation.

The training method (machine learning method) for constructing the neural network 2312 is, for example, supervised learning, but is not limited to this. In some exemplary embodiments, in the machine learning for constructing the neural network 2312, any known method such as unsupervised learning, reinforcement learning, semi-supervised learning, transduction, or multi-task learning can be used in addition to or instead of supervised learning.

In some exemplary embodiments, supervised learning is performed using training data generated by annotation, which assigns metadata (labels, tags) to input images. For example, this annotation assigns labels indicating birefringence artifacts to images included in the training data. Identification of birefringence artifacts in an image may be performed, for example, by at least one of a doctor, an engineer, a computer, and other inference models. The image to be annotated in this example, i.e., the image to which annotation information indicating birefringence artifacts is attached, is an image containing birefringence information of an object (the first image described above).

The learning processing unit 310 can construct the neural network 2312 by applying supervised learning using such training data to the neural network 320. This supervised learning is performed, for example, for each image pair included in the image pair set (a pair of a first image containing birefringence information of an object and a second image not containing birefringence information of the object) so that a second image is obtained as an output corresponding to the input of the first image. The inference model 2311 including the neural network 2312 constructed in this example functions to receive an input OCT image of the subject's eye E and output a denoised image in which birefringence-related artifacts in the OCT image have been reduced. In some exemplary embodiments, the OCT image input to the inference model 2311 is an image acquired with an OCT modality without polarization separation detection functionality (e.g., a conventional OCT intensity image), and the corresponding image output from the inference model 2311 is a denoised image in which birefringence-related artifacts in the input image have been reduced (a denoised image of a conventional OCT intensity image).

When supervised learning is performed using training data including a set of image pairs in which annotation information indicating birefringence artifacts is attached to a first image, this supervised learning is performed, for example, for each image pair included in the set of image pairs (a pair consisting of a first image containing birefringence information of an object and annotated with annotation information, and a second image not containing birefringence information of the object) so that the birefringence artifacts indicated by the annotation information attached to the first image are reduced and a second image is obtained as an output corresponding to the input of the first image. The inference model 2311, including the neural network 2312 constructed in this example, functions to receive an input OCT image of the subject's eye E (e.g., a normal OCT intensity image) and output a denoised image in which birefringence artifacts in the OCT image have been reduced. According to this example, because the annotation information indicates the location (area, range) of the birefringence artifacts, the efficiency and quality (accuracy, precision, reproducibility, etc.) of machine learning can be improved.

To avoid concentrating processing on specific units of the neural network 2312, the learning processing unit 310 may randomly select and disable some units of the neural network 320 and perform learning using the remaining units (dropout).

The techniques used to build the inference model are not limited to the examples shown here. In some exemplary embodiments, any known technique can be used to build the inference model, such as a support vector machine, a Bayesian classifier, boosting, k-means, kernel density estimation, principal component analysis, independent component analysis, self-organizing maps, random forests, or generative adversarial networks (GANs).

The information that the data processing unit 230 can generate from an OCT image is not limited to a denoised image. For example, the data processing unit 230 may have any of the following functions: a function to detect birefringence-induced artifacts from an OCT image; a function to generate attribute information for birefringence-induced artifacts; a function to generate a discrimination result (discrimination information) between an image of the true structure of the sample (examined eye E) and birefringence-induced artifacts; and a function to generate image segmentation information related to birefringence-induced artifacts. Each function is realized using a machine learning algorithm and/or a non-machine learning algorithm. That is, each function may be realized at least in part using a machine learning model, or at least in part without using a machine learning model.

The attribute information of birefringence artifacts is information that represents any property (characteristic) of birefringence artifacts in OCT images, and may be information about any parameter related to birefringence artifacts, such as the position, size, shape, strength, and influence of the birefringence artifacts. When a machine learning model is used, the training data used to construct this machine learning model includes annotation information generated by annotations related to the attributes of birefringence artifacts in a first image containing birefringence information of an object (living human eye), and the machine learning model trained using this training data is configured to receive input of an OCT image and output attribute information.

Discrimination information is information obtained by processing to discriminate (identify, distinguish) between images derived from the true structure of a sample (structure-derived images; images of ocular tissues, ocular regions, images of intraocular prostheses, etc.) and birefringence-derived artifacts. Discrimination information is, for example, information about an arbitrary position (an arbitrary pixel, an arbitrary group of pixels, an arbitrary image region, etc.) in an OCT image, and includes information indicating whether the position corresponds to a structure-derived image or a birefringence-derived artifact. In some exemplary embodiments, discrimination information may be information (discrimination list information, discrimination table information, discrimination map information, etc.) in which an identifier indicating that each pixel in an OCT image of the eye is a pixel of a structure-derived image or a pixel of an image of a birefringence-derived artifact is assigned to each pixel. The identifier indicating that the pixel is a structure-derived image may include an identifier indicating that the pixel is an image of ocular tissue (region) and an identifier indicating that the pixel is an image of an artificial object implanted in the eye. Furthermore, the identifier indicating that the image is of an ocular tissue (region) may include an identifier for each tissue (region). Furthermore, the identifier indicating that the image is of an artifact may include an identifier for each type of artifact. Furthermore, an identifier different from both the identifier indicating that the pixel is a structure-related image and the identifier indicating that the pixel is an image of a birefringence-related artifact may be used. Examples include an identifier indicating a pixel for which discrimination failed, and an identifier indicating the quality (certainty, reliability, accuracy, etc.) of the discrimination result. When a machine learning model is used, the training data used to build this machine learning model includes annotation information generated by annotations related to discrimination between structure-related images and birefringence-related artifacts in a first image containing birefringence information of an object (living human eye), and the machine learning model trained using this training data is configured to receive an OCT image as input and output discrimination information.

The image segmentation information includes information obtained by image segmentation of the OCT image. Image segmentation is a process of dividing an image into multiple segments (multiple regions, multiple pixel groups). Any technique may be used for image segmentation in this embodiment, such as semantic segmentation, instance segmentation, or panoptic segmentation. Any known segmentation method may also be used or combined, such as thresholding, clustering, dual clustering, histograms, edge detection, region growing, partial differential equations, calculus of variations, graph partitioning, or the watershed algorithm. When a machine learning model is used, the training data used to build this machine learning model includes annotation information generated by annotations related to image segmentation of a first image containing birefringence information of the object (human living eye). The machine learning model trained using this training data is configured to receive an OCT image as input and output image segmentation information.

The above describes various exemplary aspects of the data processing unit 230, but the aspects of the data processing unit 230 are not limited to these exemplary aspects, and the aspects of the ophthalmologic apparatus 1 including the data processing unit 230 are not limited to aspects that use these exemplary aspects.

<User Interface 240>
The user interface 240 includes a display unit 241 and an operation unit 242. The display unit 241 includes the display device 3. The operation unit 242 includes various operation devices and input devices. The user interface 240 may include a device that integrates a display function and an operation function, such as a touch panel. It is also possible to construct an embodiment that does not include at least a part of the user interface 240. For example, the display device may be an external device connected to the ophthalmologic apparatus 1.

<Operation of the ophthalmic device>
Several examples of the operation of the ophthalmologic apparatus 1 will be described.

<First Operation Example>
A first example of the operation of the ophthalmologic apparatus 1 will be described with reference to Fig. 6. This example provides an example of the operation of denoising image generation (birefringence-derived artifact reduction) using a machine learning algorithm.

(S1: Obtaining data of a living human eye using polarization-sensitive OCT)
First, training data is prepared by acquiring data from living human eyes using polarization-sensitive OCT. In a typical embodiment, data from a large number of living human eyes is collected and stored for building a machine learning model.

Polarization-sensitive OCT is an OCT modality capable of detecting polarized light and is implemented using an OCT device equipped with a polarization diversity receiver. The polarization diversity receiver may be, for example, one of those described in Patent Documents 1 to 4 and Non-Patent Document 1, but is not limited to these. In this example, data on the living human eye is acquired using polarization-sensitive OCT, such as the devices described in Patent Documents 1 to 3. However, embodiments are not limited to this. For example, data on the living human eye may also be acquired using an OCT modality that simply incorporates a polarization diversity receiver into a regular OCT device, such as the device described in Non-Patent Document 1.

(S2: Generate polarization-sensitive and polarization-insensitive images)
Next, a polarization-sensitive image and a polarization-insensitive image are generated from the OCT data of the living human eye acquired in step S1. For example, a pair of a polarization-sensitive image and a polarization-insensitive image is generated from one OCT data. That is, one image pair is generated from one OCT data. In a typical embodiment, a large number of image pairs are generated from a large number of OCT data to form an image pair set.

For a method for generating a polarization-sensitive image, see, for example, (Equation 21) in Patent Document 3. For a method for creating a polarization-insensitive image, see, for example, Equation (5) in Non-Patent Document 1. A polarization-sensitive image is an example of a first image that includes birefringence information of the living human eye. A polarization-insensitive image is an example of a second image that does not include birefringence information of the living human eye.

(S3: Annotate)
Next, the polarization-sensitive image generated in step S2 is annotated by identifying birefringence artifacts in the polarization-sensitive image, generating annotation information indicating the identified birefringence artifacts, and attaching the generated annotation information to the polarization-sensitive image.

(S4: Create training data)
Next, training data is created that includes the image pairs generated in step S2 and the annotation information attached to the polarization-sensitive images in step S3. In other words, training data is created that includes a set of pairs (image pair set) of polarization-sensitive images and polarization-insensitive images with annotation information attached in step S3.

Annotation information may be added to the polarization-sensitive images in all image pairs, or annotation information may be added to only the polarization-sensitive images in some image pairs.

The types of images included in the training data are not limited to polarization-sensitive and polarization-insensitive images. For example, as described above, in addition to image pair sets, the training data may also include eye images acquired using modalities other than OCT, images created by processing eye images, images generated using computer graphics, images generated using data augmentation, and simulated images.

(S5: Build a machine learning model)
Next, a machine learning model is constructed by applying machine learning using the training data acquired in step S4 to a neural network. This machine learning model receives an OCT image of the eye as input and outputs a denoised image in which birefringence-induced artifacts in the OCT image have been reduced.

(S6: Provide the machine learning model to the ophthalmic device)
Next, the machine learning model acquired in step S5 is provided to the ophthalmologic apparatus 1. This machine learning model is used as the inference model 2311 in FIG.

The above steps S1 to S6 are an example of preparations for causing the ophthalmic apparatus 1 to perform denoising image generation. An example of denoising image generation by the ophthalmic apparatus 1 is described below. Note that it is assumed that scan preparation operations such as patient ID input, alignment, and focus adjustment have already been performed.

Furthermore, an operating mode of the ophthalmic apparatus 1 for generating a denoised image may be specified. In this operating mode, the ophthalmic apparatus 1 applies, for example, an OCT scan to the subject's eye E to generate a normal OCT intensity image.

(S7: Acquire an OCT image of the subject's eye)
The ophthalmologic apparatus 1 applies OCT scanning to the subject's eye E to construct an OCT image (a normal OCT intensity image).

(S8: Input the OCT image into the machine learning model)
The ophthalmologic apparatus 1 inputs the OCT image of the subject's eye E acquired in step S7 into the inference model 2311.

(S9: Generate a denoised image)
The inference model 2311 generates a denoised image from the OCT image input in step S8.

(S10: Store/Provide Denoised Image)
The ophthalmologic apparatus 1 stores or provides the denoised image acquired in step S9.

Storing the denoised image may involve, for example, storing the denoised image in a storage device located inside the ophthalmic device 1, and/or storing the denoised image in a storage device located outside the ophthalmic device 1.

Providing the denoised image may be, for example, one or more of providing the denoised image to a computer provided inside the ophthalmic device 1, providing the denoised image to a computer provided outside the ophthalmic device 1, providing the denoised image to a display device provided inside the ophthalmic device 1, and providing the denoised image to a display device provided outside the ophthalmic device 1.

This concludes the first operation example (end).

<Second Operation Example>
A second example of the operation of the ophthalmologic apparatus 1 will be described with reference to Fig. 7. Like the first example of operation, this example of operation provides an example of denoising image generation (birefringence-induced artifact reduction) using a machine learning algorithm, but differs from the first example of operation in that annotation is not performed. Unless otherwise specified, detailed descriptions of steps and processes similar to those in the first example of operation will be omitted.

First, data on a living human eye is acquired using polarization-sensitive OCT (S11), and polarization-sensitive and polarization-insensitive images (image pairs) are generated from the acquired OCT data of the living human eye (S12). Next, in this operational example, training data is created (S13) without annotation, as in step S3 of the first operational example. The training data in this operational example includes multiple image pairs (image pair sets) acquired for multiple living human eyes in step S12. As with the first operational example, the types of images included in the training data in this operational example are not limited to polarization-sensitive and polarization-insensitive images.

Next, a machine learning model is constructed by applying machine learning using the training data acquired in step S13 to a neural network (S14), and this machine learning model is provided to the ophthalmic device 1 (S15). This machine learning model receives an input OCT image of the eye and functions to output a denoised image in which birefringence-related artifacts in the OCT image are reduced, and is used as the inference model 2311 in Figure 4.

The above steps S11 to S15 are an example of preparations for causing the ophthalmic apparatus 1 to perform denoising image generation. An example of denoising image generation by the ophthalmic apparatus 1 is described below. It is assumed that scan preparation operations such as patient ID input, alignment, and focus adjustment have already been performed. In addition, the operating mode of the ophthalmic apparatus 1 for performing denoising image generation is specified.

The ophthalmic apparatus 1 applies an OCT scan to the subject's eye E to construct an OCT image (S16) and inputs the acquired OCT image into the inference model 2311 (S17). The inference model 2311 generates a denoised image from the input OCT image (S18). The ophthalmic apparatus 1 then stores or provides the generated denoised image (S19).

<Third Operation Example>
A third example of the operation of the ophthalmologic apparatus 1 will be described with reference to Figure 8. Like the first and second operation examples, this operation example provides an example of denoising image generation (birefringence-induced artifact reduction) using a machine learning algorithm, but differs from the first operation example in that annotation is not performed and in the method of acquiring images included in the training data. Unless otherwise specified, detailed descriptions of steps and processes similar to those in the first operation example will be omitted.

First, data of a living human eye is acquired using polarization-sensitive OCT (S21), and a polarization-sensitive image is generated from the acquired OCT data of the living human eye (S22). In the first and second operation examples, both polarization-sensitive and polarization-insensitive images are generated from the OCT data acquired using polarization-sensitive OCT, but in this embodiment, only a polarization-sensitive image is generated from the OCT data acquired using polarization-sensitive OCT.

Furthermore, data of a living human eye is acquired using polarization-insensitive OCT (S23), and an image (polarization-insensitive image) is generated from this OCT data (S24). Polarization-insensitive OCT is an OCT modality performed by an OCT device that does not have a polarization diversity receiver.

The order of steps S21 to S24 is not limited to this. For example, steps S23 and S24 may be performed before steps S21 and S22. Alternatively, steps S22 and S24 may be performed after steps S21 and S23.

The polarization-sensitive OCT of step S21 and the polarization-insensitive OCT of step S23 are applied to the same living human eye. This results in a pair of polarization-sensitive and polarization-insensitive images (image pairs) of the living human eye. In a typical embodiment, the polarization-sensitive OCT of step S21 and the polarization-insensitive OCT of step S23 are applied to a large number of living human eyes, and a large number of image pairs are collected.

Next, training data is created (S25). The training data in this operational example includes multiple image pairs (a set of image pairs) acquired for multiple living human eyes in steps S21 to S24. As with the first operational example, the types of images included in the training data in this operational example are not limited to polarization-sensitive images and polarization-insensitive images.

Next, a machine learning model is constructed by applying machine learning using the training data acquired in step S25 to a neural network (S26), and this machine learning model is provided to the ophthalmic device 1 (S27). This machine learning model receives an input OCT image of the eye and functions to output a denoised image in which birefringence-related artifacts in the OCT image are reduced, and is used as the inference model 2311 in Figure 4.

The above steps S21 to S27 are an example of preparations for causing the ophthalmic apparatus 1 to perform denoising image generation. An example of denoising image generation by the ophthalmic apparatus 1 is described below. It is assumed that scan preparation operations such as patient ID input, alignment, and focus adjustment have already been performed. In addition, the operating mode of the ophthalmic apparatus 1 for performing denoising image generation is specified.

The ophthalmic apparatus 1 applies an OCT scan to the subject's eye E to construct an OCT image (S28), and inputs the acquired OCT image into the inference model 2311 (S29). The inference model 2311 generates a denoised image from the input OCT image (S30). The ophthalmic apparatus 1 then stores or provides the generated denoised image (S31).

<Fourth Operation Example>
A fourth example of the operation of the ophthalmologic apparatus 1 will be described with reference to FIG. 9 . Like the first to third examples, this example provides an example of denoising image generation (birefringence artifact reduction) using a machine learning algorithm, but differs from the first example in the method of acquiring images included in the training data. Furthermore, the difference from the third example is that annotation is performed. Unless otherwise specified, detailed descriptions of steps and processes similar to those in the first example and/or the third example will be omitted.

First, data of a living human eye is acquired using polarization-sensitive OCT (S41), a polarization-sensitive image is generated from the acquired OCT data of the living human eye (S42), and annotation is performed on the generated polarization-sensitive image (S43).

Furthermore, data of the living human eye is acquired using polarization-insensitive OCT (S44), and an image (polarization-insensitive image) is generated from this OCT data (S45).

The order of steps S41 to S45 is not limited to this. For example, steps S44 and S45 may be performed before steps S21 to S23. Alternatively, steps S41 and S44 may be performed first, followed by steps S42 and S45, and then step S43.

The polarization-sensitive OCT of step S41 and the polarization-insensitive OCT of step S44 are applied to the same living human eye. This results in a pair of polarization-sensitive and polarization-insensitive images (image pairs) of the living human eye. In a typical embodiment, the polarization-sensitive OCT of step S41 and the polarization-insensitive OCT of step S44 are applied to multiple living human eyes, and multiple image pairs are collected.

Next, training data is created (S46). The training data in this operational example includes multiple image pairs (image pair set) acquired for multiple living human eyes in steps S41, S42, S44, and S45, and annotation information attached to the polarization-sensitive images in step S43. In other words, the training data in this operational example includes a set of pairs (image pair set) of polarization-sensitive images and polarization-insensitive images attached with annotation information in step S43. As with the first operational example, the types of images included in the training data in this operational example are not limited to polarization-sensitive images and polarization-insensitive images.

Next, a machine learning model is constructed by applying machine learning using the training data acquired in step S46 to a neural network (S47), and this machine learning model is provided to the ophthalmic device 1 (S48). This machine learning model receives an input of an OCT image of the eye and functions to output a denoised image in which birefringence-related artifacts in the OCT image are reduced, and is used as the inference model 2311 in Figure 4.

The above steps S41 to S48 are an example of preparations for causing the ophthalmic apparatus 1 to perform denoising image generation. An example of denoising image generation by the ophthalmic apparatus 1 is described below. It is assumed that scan preparation operations such as patient ID input, alignment, and focus adjustment have already been performed. In addition, the operating mode of the ophthalmic apparatus 1 for performing denoising image generation is specified.

The ophthalmic apparatus 1 applies an OCT scan to the subject's eye E to construct an OCT image (S49) and inputs the acquired OCT image into the inference model 2311 (S50). The inference model 2311 generates a denoised image from the input OCT image (S51). The ophthalmic apparatus 1 then stores or provides the generated denoised image (S52).

<Effects>
Some advantages of this embodiment will be described below.

According to this aspect, training data including a set of pairs of a first image containing birefringence information of an object (eye) and a second image not containing birefringence information of the object is first prepared, and a machine learning model (inference model 2311) can be constructed by machine learning using this training data. This machine learning model functions to receive an OCT image as input and output a denoised image in which birefringence-related artifacts in the OCT image have been reduced. Furthermore, according to this aspect, an OCT image of a sample (test eye E) is acquired, and this OCT image is input to the machine learning model (inference model 2311), and the denoised image output from the machine learning model can be stored or provided.

In this way, this aspect makes it possible to reduce birefringence-related artifacts in images acquired with an OCT modality (ophthalmic device 1) that does not have polarization separation detection functionality by using a novel machine learning model trained to denoise images from OCT images of a sample, in other words, by using a novel machine learning model trained to perform birefringence-related artifact reduction on OCT images of a sample.

In other words, this aspect employs a novel feature of employing a machine learning model generated by machine learning using training data including a set of pairs of a first image containing birefringence information of an object and a second image not containing birefringence information of the object, making it possible to generate denoised images with reduced birefringence-related artifacts from images acquired with an OCT modality (ophthalmic device 1) that does not have polarization separation detection functionality.

The various points described in this embodiment provide various examples for improving the birefringence artifact reduction function.

<Program and recording medium>
A program for causing a computer to execute the OCT image processing method realized by the image processing device (ophthalmologic apparatus 1) according to the above aspect can be configured. Any of the features described in the above aspect can be combined with this program.

It is also possible to create a computer-readable non-transitory recording medium on which such a program is recorded. Any of the features described in the above aspects can be combined with this program. The non-transitory recording medium may be in any form, including, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

The embodiments and aspects thereof described in this disclosure are merely examples. Anyone wishing to implement the inventions disclosed herein may make any modifications (omissions, substitutions, additions, etc.) within the scope of the spirit of the invention.

1 Ophthalmic apparatus 230 Data processing unit 231 Denoised image generation unit 2311 Inference model

Claims (8)

1. A method for processing an optical coherence tomography (OCT) image, comprising:
a machine learning model constructed by machine learning using training data including a set of pairs of a first image including birefringence information of an object and a second image not including birefringence information of the object, the machine learning model being constructed to receive an input of an OCT image and output a denoised image in which birefringence-induced artifacts in the OCT image have been reduced;
Obtaining an OCT image of the sample;
inputting the OCT image of the sample into the machine learning model;
storing or providing a denoised image output from the machine learning model based on an input of the OCT image of the sample;
method. the first image and the second image are generated based on the same data acquired from the object by an OCT device equipped with a polarization diversity receiver;
The first image is accompanied by annotation information indicating birefringence-induced artifacts.
10. The method of claim 1. the first image and the second image are generated based on the same data acquired from the object by an OCT device equipped with a polarization diversity receiver;
10. The method of claim 1. the first image is generated based on first data acquired from the object by a first OCT device having a polarization diversity receiver;
the second image is generated based on second data acquired from the object by a second OCT device that does not include a polarization diversity receiver;
10. The method of claim 1. The first image is accompanied by annotation information indicating birefringence-induced artifacts.
10. The method of claim 1. 1. An apparatus for processing optical coherence tomography (OCT) images, comprising:
an image acquisition unit that acquires an OCT image of the sample;
a processing unit that executes processing using a machine learning model constructed by machine learning using training data including a set of pairs of a first image including birefringence information of an object and a second image not including birefringence information of the object, the machine learning model being constructed so as to receive an input of an OCT image and output a denoised image in which birefringence-induced artifacts in the OCT image have been reduced;
The processing unit inputs the OCT image of the sample acquired by the image acquisition unit into the machine learning model, and stores or provides a denoised image output from the machine learning model based on the input of the OCT image of the sample.
Device. 1. A program for processing optical coherence tomography (OCT) images, comprising:
On the computer,
acquiring an OCT image of the sample;
inputting an OCT image of the sample into a machine learning model constructed by machine learning using training data including a set of pairs of a first image including birefringence information of an object and a second image not including birefringence information of the object, the machine learning model being constructed to receive an OCT image and output a denoised image in which birefringence-induced artifacts in the OCT image are reduced;
and storing or providing a denoised image output from the machine learning model based on an input of the OCT image of the sample. A computer-readable non-transitory recording medium having a program for processing optical coherence tomography (OCT) images recorded thereon,
The program is configured to:
acquiring an OCT image of the sample;
inputting an OCT image of the sample into a machine learning model constructed by machine learning using training data including a set of pairs of a first image including birefringence information of an object and a second image not including birefringence information of the object, the machine learning model being constructed to receive an OCT image and output a denoised image in which birefringence-induced artifacts in the OCT image are reduced;
and storing or providing a denoised image output from the machine learning model based on an input of the OCT image of the sample.